Within the face of rising IT complexity, Cisco IT unified observability throughout its world atmosphere. The outcomes: 25% fewer main incidents, 45% quicker decision, and scalable automation. Right here’s how we did it—and suggestions to your digital resilience journey.
Our problem: fragmented visibility
Many organizations wrestle with fragmented monitoring and extended incident decision. We confronted the identical challenges and wanted to interrupt down knowledge silos to guard our fast-changing atmosphere.
In Cisco IT, managing our world, dispersed IT panorama is more and more advanced. Fragmented knowledge and visibility gaps had been making it more and more troublesome to take care of digital resilience amid our fast-paced innovation and frequent environmental modifications. We would have liked a option to flip our knowledge into actionable insights however lacked a unified observability platform to centralize and make sense of all of it.
When a significant database outage in 2024 revealed our fragmented knowledge — uncorrelated alerts throughout associated units delaying trigger identification — we knew we needed to rethink our observability strategy.
“After the outage, we knew we needed to rethink all the things. The transformation wasn’t nearly expertise, however empowering our engineers to behave quicker and smarter.”
– Chuck Churchill, Snr. Director, IT Observability, Cisco IT
This inflection level sparked a broader shift in how Cisco IT approached observability. Within the video under, Cisco IT leaders share what modified — and the way it led to a 25% discount in main incidents.
Getting began
Like several IT subject, understanding the foundation trigger was important earlier than remediation planning might start. Constructing stronger digital resilience isn’t any completely different.
In working with our groups, we pinpointed the next as high problem areas that had been hindering our efforts:
- Fragmented knowledge and visibility gaps: With over 100,000 endpoints, we generated huge volumes of telemetry knowledge. Siloed monitoring instruments led to alert fatigue and visibility gaps, slowing response occasions and limiting our skill to foretell and forestall points.
- Dangers from frequent modifications: Our tradition of innovation and early adoption means fixed modifications to our IT environments. We noticed a direct hyperlink between speedy change and elevated incidents, so minimizing disruption with out slowing innovation turned important.
- Useful resource optimization: As the environment and knowledge complexity grew, it turned important to enhance how we leverage AI and knowledge extra successfully. We would have liked to show our knowledge into actionable insights that empower engineers, not overwhelm them, to make sure productiveness saved tempo with development.
“Bringing all our knowledge collectively was step one—turning it into actual, actionable insights is what actually permits us to remain resilient as the environment evolves.”
– Jon Heaton, Director, Community Engineering and Operations, Cisco IT
Deciding on our three-pronged observability strategy
Digital resilience requires greater than merely deploying new instruments. We would have liked a holistic strategy that might span our whole IT panorama. We obtaind this by structuring our IT observability apply throughout three interconnected pillars:
- The community: Safe, dependable community efficiency is crucial for retaining the enterprise up and working. This pillar focuses on complete community visibility, together with third–occasion supplier networks, to make sure the community is working securely and optimally for a easy person expertise.
- Platforms and knowledge: Right here, we concentrate on observability throughout our knowledge facilities, cloud, and underlying infrastructure, centralizing our observability knowledge to be accessible to your entire group, together with our DevOps and SRE groups, via our platform consolidation and knowledge technique.
- Service Operations: Our Service Operations and Enterprise Operations Heart engineers monitor, analyze, and resolve points utilizing wealthy knowledge and insights delivered from our community, infrastructure, purposes and providers — which feed AI automation to enhance effectivity.
Including within the important expertise
Every of those observability pillars is powered by a mix of knowledge, expertise, and processes that allow us to make use of the complete potential of our knowledge and drive better effectivity via AI automation. These core components are important to our observability strategy and success:
- Splunk: Splunk acts because the spine of our observability technique, centralizing knowledge from throughout our community, infrastructure, and purposes to ship a single supply of reality for our IT groups.
- ThousandEyes: ThousandEyes delivers end-to-end community visibility and person expertise monitoring throughout inner and exterior environments, enabling speedy identification and determination of connectivity points.
- Configuration Administration Database (CMDB): Our CMDB supplies a single supply of reality for all IT belongings, enriching alerts and incidents with important context and powering environment friendly, proactive operations.
- AI Operations: Our AI programs leverage the centralized observability knowledge in Splunk to automate occasion evaluation, scale back alert fatigue, and speed up incident response — empowering engineers to concentrate on higher-value work. For instance, our Service Operations makes use of numerous AI assistants for AI-driven incident administration to foretell incident assignments and recommend decision steps.
By integrating Splunk with ThousandEyes, our CMDB, and different instruments, we will guarantee a seamless, scalable strategy to observability that grows with our enterprise.
Tangible outcomes
This unified observability strategy has helped us sort out our most urgent challenges and enhance outcomes that collectively strengthen our digital resilience. Previously 18 months, we’ve seen:
- Important discount in main incidents: We lowered main incidents by 25% yr over yr and had zero main community incidents, down from 3-4 per quarter beforehand.
- Sooner, more practical restoration: We lowered our Imply Time to Detect and Resolve by 45% yr over yr, enabling quicker restoration and minimized disruption.
- Improved change administration: We decreased incidents brought on by change by 20%, enabled by unified knowledge insights and end-to-end visibility that helps smarter change administration processes.
- Strengthened visibility and knowledge utilization: We now monitor 10x extra community telemetry knowledge that yields deeper insights and 4x better visibility — enabling earlier detection and proactive decision of potential points earlier than they escalate.
- Scalable automation and effectivity: Centralizing knowledge in Splunk has established a basis for continued developments in AIOps, enabling us to develop AI-automations that now deal with 99.998% of ~4 million each day alert — considerably bettering operational effectivity.
“This journey is as a lot about altering mindsets as deploying expertise. We’re empowering each engineer to behave on insights, not simply alerts.”
– Mark Hutchins, Director, IT Service Administration, Cisco IT
Sensible takeaways
Our digital resilience journey is ongoing, however we study extra every day that we share with prospects enhancing their very own journey. We suggest:
- Acquire telemetry from in all places: Centralize telemetry to optimize probably the most related metrics, occasions, logs and traces from throughout your community, infrastructure, cloud, purposes, and providers.
- Prioritize knowledge – the bedrock of success: concentrate on knowledge high quality and hygiene. Correct, clear knowledge is crucial for producing reliable insights and enabling efficient automation.
- Use what’s already sensible: First, leverage the programs that you’ve: product capabilities which have AI and observability in-built. Second, empower groups to experiment with customized AI options to maximise worth the place your present programs have gaps.
Keep tuned for updates and learnings as we proceed to innovate and optimize.
“Digital resilience is a shifting goal. We’re at all times studying, adapting, and refining our strategy as the environment evolves.”
– Jon Heaton, Director, Community Engineering and Operations, Cisco IT
Extra assets:
Discover extra case research to see how Cisco strengthens digital resilience
