Top 6 Increased WFH UC Challenges & Solutions
As the world struggles with the unprecedented COVID-19 crisis, businesses of all sizes are asking employees to work remotely. According to Telecommuting Advantage Group, 92% of all employees already have the technology and flexibility to work remotely. Until recently, almost half of all global companies required employees to work from an office without remote work option. These policies are becoming increasingly antiquated while remote is becoming more popular increasingly challenging IT. Most enterprise IT organizations are under-prepared, especially for this sudden and voluminous increase in remote workers due to the crisis.
To assist the organizations hit hardest, Unify Square created a special FREE offer in early March 2020. This free offer is available to educational, healthcare and state & local government institutions. Since mid-January we’ve been helping our customers around the world adjust their Skype for Business, Zoom and Microsoft Teams unified communications (UC) systems to respond to the Work-From-Home onslaught. Not surprisingly, we’ve started noticing some obvious patterns emerge. We understand that IT teams are under significant strain as they support the unexpectedly high number of remote employees. To that end, we’ve compiled this checklist of top challenges and solutions/adjustments to help best scale your UC systems.
Maintaining UC System Health for Remote Workers @ Scale
Given the pandemic COVID-19, how do IT teams ensure critical UC systems remain available and performant for remote work capability? Managing a remote workforce is incredibly complex. This is because IT teams do not control the employee’s environment, but are still responsible for their employee’s experience. Sitting between the UC systems and end-users is a complex set of different software and hardware. These include firewalls, edge servers, last-mile ISPs, VPN gateways Wi-Fi networks, and end user endpoint devices. These are always parts of the system architecture that IT considers when deploying a UC system. However, the recent rapid increase in remote workers means that key adjustments to parts of the system are necessary to keep it healthy.
Interestingly, IT’s challenge is made even harder by Shadow IT usage. Our data (below) show increases in not just a single UC platform, but ALL UC platforms. We’re not just seeing increases in one mode or type of calling, but ALL modes. For example, even internal calling greatly spikes at times. This is simply because the “sneaker net” option of just walking into someone’s office for a quick question is gone. Instead a phone call (or chat session) is required. IT may follow our guidance below to help fix the approved UC systems, the Shadow UC system issues will persist. Employees will still appeal to IT for help, even if these unofficial UC systems are not working for them.
If this scenario applies to your business (especially related to Zoom usage), you may be interested in a new Unify Square offering that is in BETA today. Unify Square’s Shadow IT Scout can help determine what UC systems are in use and allow you to get a jump on troubleshooting them.
Over the course of just the last 2 months (Jan 2020 to March 2020), the aggregate data from our global customer base shows a 30%+ rise in external based calling. The increase in external calling is leading to a disproportionate increase in Poor Call Percentages for customers, brought on by a number of different factors:
- 28% increase in Anonymous users joining meetings with an increase in poor call percentage from 4.8% to over 8.1%
- 16% increase in Federated calling with an increase in poor call percentage from 2.78% to 4.67%
- 31% increase in VPN calling with poor calls spiking from 6.45% to as high as 9%
- 31% increase in call counts for external users with an increase in Poor Call Percentage from 3.39% to 9.1%
- Total Call Volume spikes up to 28% with a 68% increase in Poor Calls
- Calls traversing mediation servers increased at 5%, but with an 11% increase in poor call percentages from .67% to .75%
The above extremes, while aggressive, can still be quite manageable, especially if IT has a team to focus on the issues. PowerSuite Could Managed service subscribers get 24/7 UC expert hands on support. When we filter for these subscribers, we’re seeing only very minor spikes and issues.
Top 6 UC System Challenges and Solutions
- Challenge #1: Home networks
For a full days’ worth of video conferencing, screen sharing, and phone calls a Wi-Fi only solution will be the call quality death knell for many users.
Solution: IT should request/suggest the following of employees: Implement a policy that users should, whenever possible either a) Plug into a dedicated ethernet wire; b) Set up their remote work station as close as possible to the Wi-Fi access point; or c) Upgrade their home router to handle not just the increased WFH demands. For example, increased demands that come with stuck-at-home kids gaming or streaming Netflix on screens all over the house. Not to mention adults or students sucking up bandwidth for their own teleconferences while working, or schooling from home. Finally, suggest that gaming and Netflix streaming activities be curtailed and/or limited to non-working hours.
- Challenge #2: Laptop mic & speaker
The occasional phone call using generic ear buds, or the laptop mic and speaker may be OK. However, over the long haul such unoptimized devices really eat into call quality for everyone on the conference.
Solution: This is a tough one to fix, unless your organization can purchase/subsidize quality headsets.At best, IT can communicate with all employees regarding the importance of a quality headset (e.g. the microphone/ear bud combo that comes with most cellphone purchases) and even suggest a variety of different brands and styles that are considered more robust. In many cases even using a gaming headset will be better than the PC mic and speakers to help prevent poor quality and echo.
- Challenge #3: Scale of the Edge Servers
During many Skype for Business or Cisco on-premises deployments customers scaled their edge environment to support only a portion of their remote workforce (the Microsoft User Model used for estimating usage by default only includes 30% remote users). Given that this number has likely drastically shifted edge servers have probably become a bottle neck or have performance issues (same for reverse proxy or load balancers external VIPs). Even if you built for N+1 redundancy based on Microsoft sizing numbers, you’re likely to hit scale issues on your edge servers when those loads shift out.
Solution: The fix here might involve several different changes as follows:
- Monitor edge server scale by checking system resources like available memory, processor and network utilization
- If a spike in TCP connections for media like Audio and Video calls is detected IT may also need to verify that the firewall isn’t exhausting its UDP port range (or licensed connection limit)
- Add additional Edge Servers or add resources to existing edges if they are resource constrained. Especially if they are virtual scaling up to support more load should be easy.
- Remember that edge sizing numbers have varied by release version, so the safest option is to monitor resources and scale up if needed — https://docs.microsoft.com/en-us/skypeforbusiness/plan-your-deployment/capacity/user-models. Edge documents have been updated to 18k users concurrent, but that contradicts the user model and is specific to 2019 Edge – sizing recommendation were different in 2010 – 2015.
- Challenge #4: VPN usage & bandwidth issues
A virtual private network is one of your most critical technologies, providing a secure communication channel through public Internet connections. However, Skype for Business and Microsoft Teams workloads are secured from the start. Forcing these workloads through VPN only furthers exponential increase in load to your VPN. This in turn requires adjustment or scaling of VPN concentrators.
Solution: Again here, several different changes will likely be appropriate as follows:
- Allow your Skype for Business and Microsoft Teams media to bypass the VPN
- Reconfigure your VPN to help stabilize issue introduced by VPN tunneling — https://techcommunity.microsoft.com/t5/skype-for-business-blog/enabling-lync-media-to-bypass-a-vpn-tunnel/ba-p/620210. Most organizations do not separate their real time collaboration services from the rest of their internet facing services such as VPN. Once remote access blooms, the internet egress becomes overwhelmed. Real time workloads like Skype are early indicators of this (canary in the coal mine).
- Challenge #5: Firewall connections and server sizing
Many Firewalls have connection limits based on either physical hardware or licensing. Going beyond those limits causes unexpected behaviors such as failed connections, packet drops, workloads using less optimal protocols etc.
Solution: Once again, several different changes will likely be appropriate as follows:
- Pay close attention to firewall connectivity (especially if you start seeing lots of TCP) and VPN utilization. Monitor the number of connections that are coming in and watch for legitimate traffic being blocked or port exhaustion issues.
- If you don’t have split-tunneling setup for your media (or if it isn’t working as expected or allowing hair pinning), then strongly consider enabling.
- If you have “Tunnel All” style policies, you need to exclude your own Skype for Business IP address, or Microsoft Teams or Zoom data center ranges from your tunnels to not have your concentrators jamming up the works – see the article above about how to make sure your VPN will work for Skype for Business.
- As you plan for bandwidth changes (and possibly bursting existing connections), it’s important to also plan for the increase you’ll likely see in Federated and Anonymous traffic if you have Skype for Business on Premises.
- Challenge #6: Mediation Server Load
Increased mediation load is very likely. Calls that were previously able to take advantage of media bypass to internal users are no longer. This forces a transcode at your mediation servers which impacts process, memory and network load on your Mediation servers. Additionally, your newly home-based users may have quality issues with their home connections. This could initiate a decision to dial in or have the system call them back, creating increased PSTN usage.
Solution: It’s very important to monitor System resources and watch for spikes in load for mediation servers, SBCs and PSTN connections. Depending on your sizing it may be important to add additional memory/processing to servers or additional servers (a very easy task if you’re running a virtual configuration).
Challenges #1 and #2 are ubiquitous in nature regardless of whether your system is pure cloud or on-premises based. However, challenges 3-6 focus largely on on-premises or hybrid systems. Therefore, if you company executes a rapid cloud transformation to Zoom or Microsoft Teams, these issues may simply disappear. If this option appeals to you, there are multiple types of FREE offers on the market, from both the UC providers as well as from Unify Square (to help manage the services), which IT should explore.
The Importance of an IT Department That ‘Thinks Remote’
If you require more assistance than this article was able to provide or have follow-up questions, our UC experts are standing by (at home) ready to assist. You might also consider a free trial of our PowerSuite software. By running PowerSuite in your UC environment we can help you pro-actively acquire insights regarding how to fine-tune your system for remote work. Some of the issues that we have spotted to date include large increases in Poor Call Percentage because of the heavy shift t remote work and reports on which devices are creating them the worst quality and echo issues.
It used to be the mantra for IT to “think mobile.” The new mantra should be “think remote”. Especially when it comes to UC systems IT needs to build and architect with remote work in mind. The future of work is all about efficiency and making the workplace a satisfying experience. That workplace is now, at least for the next couple of months, more often than not, entirely a remote location — at home.