Why 60% of organizations leveraging cloud-services will be benchmarking by 2020
Benchmarking has many uses across a variety of verticals in business. It’s best for measuring performance achievements and then comparing that performance both internally and externally to the organization. Collective intelligence benchmarking (CIB), the official Gartner term, is a baseline of performance for a given service or application that is typically based on anonymized data aggregated from end users.
Benchmarking is not a new concept, but CIB elevates the concept to focus on cloud-sourced data, meaning that the metric is still very much in its infancy. In 2016, Gartner estimated that just 5 percent of organizations used collective intelligence benchmarking as their primary method of benchmarking the availability and performance of SaaS-based applications. Yet, the expectation is that the concept is catching on quickly – by 2020, Gartner predicts this number to be as high as 60 percent.
Collective intelligence benchmarking creates a solution for IT teams facing challenges such as device proliferation or shadow IT groups who may be introducing un-vetted apps and services into the organization. In the past, companies have had to rely on traditional monitoring solutions, but these toolsets on their own have become insufficient. Traditional monitoring tools struggle to correlate large datasets from every client interaction and experience with an exhaustive bottoms-up approach.
CIB enters the picture to offer a modern way to create a baseline for IT environments using objective benchmarks to determine optimum or at least normal performance. It can also be used as a method to detect deviations or suboptimal service delivery based on cross-company sharing and comparison of IT operation’s KPIs and infrastructure metrics – all of which require pristine performance to support IT initiatives like unified communications or workstream collaboration.
Internal vs External Benchmarking
Internal benchmarking is when organizations use their own aggregated collective intelligence benchmarking data. Internal benchmarking removes the constraints of anonymized data and also opens up the opportunity for things like automated polling to provide qualitative feedback.
On the flip side, externally generated CIB utilizes shared datasets from multiple third-party end users and allows companies to compare third-party results against their own experience of a specified service or application. In this instance, the data is anonymized but the number of data points is typically significantly larger than if a company relied on internal data alone.
Cloud-Sourced Benchmarking: Four and 1/2 Secrets from a Gartner Cool UC Vendor
At Unify Square, cloud-sourced CIB is baked into PowerSuite™ to help organizations better understand and monitor their UC and WSC service availability and performance. In fact, our benchmarking leadership is so cutting-edge that Gartner actually named us as a Cool UC vendor in 2018.
You can learn more about how combined internal and external CIB can super-size UC performance monitoring by attending our webinar where we will cover key topics such as:
- How to build a world-class offering by benchmarking your service
- How benchmarking for quality can save your job
- Can geo-benchmarking save your users?
- How to drive continuous improvement with time-benchmarking
Real World Use Cases
Some examples of real-world use cases for collective intelligence benchmarking include:
- Unified Communications: UC monitoring tools are typically instrumentation and device-based quantitative measurements, but tools that utilize CIB look to survey end users to provide consistent measurements of user satisfaction and sentiment toward the adoption of applications like Skype for Business. With PowerSuite we take these benchmarking basics a step further and also allow IT to compare poor call percentages, service availability, and user satisfaction based on both internal and external collective intelligence benchmarking capabilities.
- Workstream Collaboration (WSC): Like UC, workstream collaboration applications like Microsoft Teams requires an understanding of performance, user satisfaction, and adoption metrics. Because WSC applications can put a different strain on networks, understanding performance capabilities will be crucial for a successful deployment.
- Network monitoring: Tools that look at CIB analytics for wireless network monitoring are based on generating anonymized customer data and correlating this with comparable networks. Cloud-sourced monitoring provides continuous monitoring of infrastructure for hundreds of thousands of client devices to help identify incidents and proactively notify end-users.
Key Considerations and Recommendations for Collective Intelligence Benchmarking
CIB helps reduce the visibility gap of cloud-based environments using alternative perspectives for enterprises that face limitations from their existing monitoring approaches. UC and WSC monitoring scenarios present the strongest case for internally generated CIB due to growth within large enterprises, which provides a large set of end users to sample from –end users who are typically intolerant of high latency issues with real-time type enterprise apps and services.
While there are many advantages to CIB, there are a few risks or considerations companies should take into account before integrating CIB too deeply into an IT measurement system:
- Dataset Size – Large, well understood external datasets that align to your own organization should be utilized. Be cautious of tools with insufficient datasets, as the level of meaningful analysis will likely be seriously impacted resulting in spurious interpretations due to outliers that may skew the results.
- Vendor Accountability – Externally generated CIB can be used for more than just comparing organizational KPIs to other outside datasets – it can be used to verify that your cloud and SaaS providers are providing a quality service, and if not they can be held accountable for below-par service delivery.
- Specialty Tools – There are many uses for CIB, and therefore potentially many monitoring tools which may eventually offer CIB-like technology expertise. Be sure to confirm that the collective intelligence benchmarking data points are actually valuable and relevant to your organization.