According to data disclosed by iiMedia Research, there have been over 18 million enterprises that resorted to online telecommuting during China's work resumption in 2020, involving a total of more than 300 million telecommuting application users. Meanwhile, around 200 million students took online classes in remote education. And telecommuting, video conference and online education system all require the support of the underlying real-time audio/video technology.
Real-time audio/video technology covers real-time voice call, real-time video call, and mic-to-mic interaction and "live broadcast" may be the most familiar scenario of real-time audio/video application to the public. As COVID-19 swept across the world, remote contactless services based on the audio/video technology were booming in other industries and scenarios.
ZEGO hitting 2 billion minutes in daily average duration of calls by giving "X+audio/video" aPaaS access to all industries
In response to remote service needs that exploded in various industries during the pandemic, audio/video cloud service provider ZEGO instantly unveiled two SaaS platforms including TalkLine video conference and Xiaoyibang online help in preparation for entrance examination for art majors and many remote aPaaS solutions. Taking advantage of strengths of download-and-use and quick development, ZEGO rapidly broadened the applied range of the audio/video technology from merely live broadcast, social networking and
pan-entertainment to many other industries, such as education, office conferencing, entrance examination for art majors, government & public service, healthcare and finance, covering all government & public service areas.
As the collaborative online remote service apps were exploding, ZEGO as their underlying technical service provider kept building up its platform database as well. It is said that ZEGO's total platform stream increased nearly 4 times and daily average duration of audio/video interaction rose to 2 billion minutes earlier in 2020.
ZEGO's partners cover a mass of top pan-entertainment and online education platforms, e.g., Inke, Ximalaya, Kugou, Lizhi, TAL, Zuoyebang, 17zuoye, or Zhangmen 1V1, etc. The two industries were exactly the fastest-growing industries regarding data traffic earlier in 2020. The user count of many platforms was mounting by multiples, among which, some top national online education platform in particular fulfilled a
hundredfold jump in the user count.
The sharp increase in data traffic indicated a mass of users to platforms but caused greater pressure on platform stability and may lead to platform collapse and service unavailability. In those "stay-at-home" days, we all experienced the crash of many Apps. So how did ZEGO assure service stability given a sharp increase in data traffic--2-billion-min-long interaction per day on average and concurrence of thousands of millions of data requests?
Ten-million-level high concurrency architecture + backend service mechanism assuring service stability
Ten-million-level high concurrency requires an underlying architecture available for ten-million-level concurrency. ZEGO's founding members were Tencent's former employees, most of whom got involved in fulfilling a leapfrog increase in the QQ product user population from millions of persons to billions of persons. Therefore, they have foresight in high concurrency design on underlying platform architecture. Based on experience of mass user operation and nearly two decades of experience of working on the audio/video technology, the required concurrency attained the ten-million-level in underlying architecture design as early as 5 years ago at the inception of ZEGO.
In subsequent years, ZEGO relentlessly improved the technology and optimized single point load capacity of servers, intelligent dispatching algorithm of the trunking system and multi-cloud and -net virtual network building. As a result, ZEGO has successfully put into practice flexibility. Now its users can experience seamless and smooth capacity expansion/reduction in a damage-free and unwitting way. In June 2019, ZEGO's daily average audio/video interaction totaled 600 million minutes in backend statistics.
Apart from the ten-million-level concurrency infrastructure, such backend service mechanisms as warning, data traffic limiting, and emergency capacity expansion are needed to sustain service smoothness and stability upon waves of traffic peaks.
In this respect, Jack heading ZEGO's backend media service team elaborated on the operation of ZEGO's package of backend service mechanisms:
For every enterprise customer, first of all, ZEGO provides real-time data traffic monitoring and warning services. Backend service technicians monitor the current capacity level. When the customer's server load exceeds a certain threshold, the automatic trigger of warning will be enabled to remind those concerned of capacity expansion;
Secondly, in the event of instant sudden sharp increase, ZEGO's backend service staff will assure the availability of the customer service through the data traffic limiting mechanism. What needs to be noted is that ZEGO's data traffic limiting mechanism is premised by no impact on the user experience. Take one of ZEGO's customers, a certain education platform, for example. Unexpectedly, the customer found out a large jump in the platform's user count. Then ZEGO's data traffic limiting mechanism was triggered. After data traffic limiting was enabled, the platform's existing teachers and students sustained no impact on service experience; and for new users, ZEGO sacrificed part of user experience in exchange for service availability by lowering bitrate and resolution in stream publishing/playing. Subsequently, ZEGO immediately enabled the emergency capacity expansion mechanism. Ten minutes later, capacity expansion was completed and new-coming teachers and students experienced a recovery in the bitrate and resolution.
Thirdly, ZEGO's emergency capacity expansion mechanism is available for increasing capacity of hundreds of servers in 10 minutes. And such server resources can support an online education platform to add programs involving 100,000 teachers offering courses or nearly 1 million attending students.
Well prepared in the architecture and backend service mechanisms, ZEGO also renders one-to-one technical support to help customers avoid beforehand and timely address problems that may arise out of data traffic increase.
Pre-services + one-to-one support helping all ZEGO users gain the best experience
"Technical support is a meticulous and thorough job and a sharp increase in the platform's user count is a daunting challenge to it. We all are 24 hours on call to support increasing user count and expanding capacity for customers throughout the entire process," quoted Tina, ZEGO's technical support director.
Different from the normal image of a technical supporter that "solves problems that come up", ZEGO advocates mapping customer success orientation to technical support work or providing pre-services to help customers evade from possible risks in advance.
Earlier in February, the Ministry of Education proposed an initiative on "home-based distance education". Predicting a subsequent probable sharp increase in the online education platform user count, ZEGO's support staff paid a return visit to all of its education platform customers. Through an inquiry on each customer's estimate on the platform's user count, the staff determined whether the customer's present server resources could suffice. And the staff worked out a one-to-one service program for each platform, detailing applied scenarios, e.g., large class, one-to-one small class, one-to-many small class, AI classroom or transnational education.
After working out the program, the technical support staff immediately communicated with the backend service staff to confirm server resources and make advance deployment. Based on the customer's estimate, ZEGO added a certain amount of redundant resources for assuring stability.
Throughout the entire process from user count increasing, peaking to declining, ZEGO's technical staff need back up customers. And industries vary in peak hours. For an online education platform, the user count starts to increase in the morning and peaks in the afternoon or evening; for a transnational education or cross-border platform, the user count peaks after midnight; for one of ZEGO's customers, some top financial platform, the user count started to rise at 7:00 a.m. and maximized in the morning upon a remote internal training on 30,000 salespersons through ZEGO's SDK. It means that the technical support staff need work in turns for 24 hours a day.
In addition to supporting increase in the user count, technical supporters also need to do routine checks to guarantee optimal customer experience. "Frequently, we do checks till 1:00-2:00 a.m. but our effort does pay off. Much to our relief, customers leave a message in the communication group upon a day of hard work: Today everything works well in the platform. Your support is appreciated," said Tina.
Behind 2 billion minutes of interaction per day is a hefty number of customers well recognizing ZEGO and its services. But ZEGO did encounter some intractable problems. As online users were mounting at the fastest pace, the entire network was constrained in server resources. On one hand, customers were badly thirsty for capacity expansion to support service; on the other hand, vendors' resources did not suffice. After taking possession of all available resources from the existing vendors, ZEGO purchased from other cloud service providers one after another.
"Our architecture is already in favor of multi-cloud e-commerce solutions, compatible to and supportive of physical machines and cloud hosts made by different manufacturers. Therefore, our server resources are not confined by single vendors and we can call as many resources as possible in no time. Lack of server resources did not beset us for a long time," said Jack, ZEGO's backend media service director.
ZEGO highlights values of altruism and common growth and aspires for giving full support to customer success and growing with customers. In line with the principle of "perfecting technology and not wasting every penny spent by customers", ZEGO optimizes each customer's cloud service cost fundamentally through relentless optimization of server load capacity.
In addition, for different customers at different periods, ZEGO introduces different preferential measures. During the outbreak of the pandemic, users had a mass of requests for telecommuting and office automation (OA). In response, ZEGO's video conference platform TalkLine endowed every newly registered user with 5000 minutes of service free of charge. So far, approximately 1 billion minutes of service has been presented for free in accumulation.
When the wave arrives, ZEGO gives the greatest support to customers; after the wave ebbs, ZEGO makes a thorough arrangement for customers.