CX Analytics Solution for Fault and Performance Management

Introduction

This document describes the CX analytics solution for fault and performance management.

Objective

This document primarily aims to showcase the capabilities of Cisco CX Network Wide Visibility Solution (Matrix) in handling Fault and Performance management concerning Packet Core. It provides valuable insights to internal and external stakeholders who are in search of an optimised, customisable Analytical Network Monitoring Solution and who are currently using Cisco Prime and looking for an alternative to monitor their Mobile Packet Core Network.

Overview

Every business relies on networks as a fundamental support system. Networks are becoming more complex due to the continuous evolution of interdependent technology layers, various domains, a multi-vendor environment, varying data volumes and formats, virtualization, dynamic workload-based scaling, and microservice architecture. This complexity poses challenges in managing SP/Enterprise networks.

A network outage causing a loss of productivity can lead to significant damage. The consequential downtime may impact critical business operations, disrupt services, hamper customer satisfaction, and potentially damage the reputation of the organization. Hence, maintaining a robust and reliable network infrastructure is crucial for ensuring uninterrupted productivity and mitigating potential adverse effects on the business.

Network visibility enhances security, performance and planning while reducing the workload on Network Operations professionals. A holistic, centralized view lets organizations map their entire networks, detect threats early, prioritize alerts, identify anomalies and plan for future network investments. Cisco Prime Performance manager and Prime Network have played their role widely in these areas by proactively monitoring SP Network.

According to Sirkin Research, network operation professionals encounter several challenges when striving to meet strategic initiatives, with the most significant being a shortage of time, as indicated by nearly 43% of respondents in the 2019 Top Network Challenges Survey. Approximately 42% face difficulties troubleshooting issues across the entire network due to disparate legacy architectures. Network professionals also grapple with identifying network performance issues spanning multiple network domains (38%) and are hindered by inadequate performance visibility across various network fabrics (35%). These blind spots in network performance contribute to the inability to enhance overall network performance quality, ultimately impeding the successful execution of network transformation initiatives.

“Source: 2019 Top Network Performance Challenges Survey - https://www.liveaction.com/2019-top-network-performance-challenges/”

In conclusion, the identified challenges in network operations emphasize the significance of implementing thorough network monitoring solutions. With the complexities involved in troubleshooting, time constraints, and performance visibility gaps, a robust network monitoring approach becomes indispensable. It not only addresses these issues but also plays a pivotal role in ensuring the success of network transformation initiatives by providing comprehensive insights and proactive management.

Cisco Prime Performance Manager and Prime Network have played a vital role in proactively monitoring SP Network. It has a presence all over the world. Like any other product, Prime has reached the end of life for several reasons, market demands, technology innovation and development-driven changes, or product maturity and replacement with functionally richer technology. So, What’s next? Which product to transition to? Cisco Prime BU recommends migrating performance monitoring of Packet Core devices to Matrix. Hence, this document is compiled to elaborate on the role of Matrix in facilitating the performance and fault monitoring use case for the packet core.

Prime EOL Dates

“Source: 

• PN: https://www.cisco.com/c/en/us/products/collateral/cloud-systems-management/prime-network/eos-eol-notice-c51-744070.html
• PPM: https://www.cisco.com/c/en/us/products/collateral/cloud-systems-management/prime-performance-manager/eos-eol-notice-c51-744071.html “

What is PN, PPM, and Matrix?

Cisco Prime

Prime Layered Architecture

Cisco Prime Network is a network management system that allows you to operate, administrate and manage Network Element. It provides automated configuration and change management, fault monitoring and correlation to enable proactive service assurance for outstanding service availability.

Cisco Prime Performance Manager is a performance management solution that provides immediate, actionable information for complex carrier networks.

The integrated Cisco Prime Network and Prime Performance Manager provide a complete network assurance management solution. This combination offers fault management and trend information to avoid future service disruptions proactively. 

Cisco CX Matrix

Cisco CX Matrix is an advanced analytics solution that provides analytics capabilities for managing and operating Networks, Services, Infrastructure, and Applications across domains. As an analytics and assurance solution, Matrix supports multiple cross-domain architectures and is flexible and scalable to meet customer requirements. This solution promotes several out-of-the-box capabilities, including out-of-the-box data adapters, KPIs, machine learning algorithms, automated inventory discovery, alarms, and event-driven automation.

The solution is highly extensible, empowering users to seamlessly onboard network data, establish analytics logic for in-depth data analysis, and create customized dashboards through the integrated business intelligence (BI) layer. Illustrated in the diagram below, the data pipeline and dashboards are configurable by users to precisely align with customer requirements. Once the data is ingested, users can leverage Matrix’s larger feature set, such as the audit and reporting framework and the machine learning pipelines.

Matrix Layered Architecture

The platform comes equipped with out-of-the-box functionality, offering pre-built use cases that comprehensively address critical architecture domains in service provider and enterprise environments. These function pack extensions provide end users with out-of-the-box use cases that can be enabled during application setup.

The solution extends its support across mobile core (4G/5G), transport, optical, and data centre domains within the service provider space. In the enterprise segment, the platform caters to diverse domains such as Campus Wireless, WAN/SDWAN, and Datacenter. It provides cross-domain visibility, ensuring a versatile and holistic approach to meet varied organizational needs.

Matrix capabilities:

• Single pane of glass for monitoring network, service, and subscribers.
• Correlated multi-domain visibility.
• Service and subscriber experience analytics.
• Network capacity utilization analysis.
• Intelligent alerting using AI/ML.
• Multi-domain incident management.
• Real-time proactive alerting rather than manual reports.
• On-demand network audits.
• Consolidated inventory management.

Use Case - Packet Core Network Monitoring

The Mobility Packet Core (MPC) is a critical component in a service provider network, particularly in the context of mobile communication systems. It plays a crucial role in enabling seamless mobility and providing various essential functionalities - seamless mobility, packet switching, quality of service, policy enforcement, security, charging and billing, and network management. It forms the backbone of modern mobile communication systems, facilitating efficient and reliable connectivity for millions of subscribers.

Over the years MPC has evolved a lot to bring in flexibility and intelligent network architecture with the help of Software Defined Networks (SDN) and Network Functions Virtualization (NFV) features. To achieve optimal resource utilization and mitigate potential bottlenecks and other network issues, modern networks must integrate monitoring tools.

Performance Monitoring

Cisco CX Matrix offers a robust suite of features, including an extensive knowledge base, counters, schema types, default dashboards, and a library of over 6,000 Key Performance Indicators (KPIs) to effectively oversee packet core network operations. It actively monitors the performance of the entire packet core, encompassing 3G, 4G, and 5G, as well as the infrastructure layer, promptly detecting any real-time performance degradation. Matrix provides flexibility to customers, allowing the creation of new dashboards, KPIs, and various aggregations at different levels to suit specific requirements.

Performance monitoring feature list:

Bulkstats Collection and Processing

Along with advanced data collection techniques, Matrix also supports the collection and processing of bulkstats. Performance monitoring using bulkstats is an efficient and comprehensive technique for monitoring system performance. It involves collecting and analyzing performance data in bulk, rather than retrieving individual data points separately. By aggregating and processing performance metrics in batches, bulkstats reduces the overhead associated with data retrieval, processing, and transmission. This leads to improved monitoring efficiency and reduced network congestion.

Using bulkstats, real-time analysis of performance trends becomes possible. It allows for the identification of bottlenecks and enables proactive optimization of system resources. By analyzing performance data in bulk, Matrix empowers users to make informed decisions and take prompt actions to enhance performance and deliver a better user experience.

Matrix value adds and differentiators:

1. Matrix also supports multiple different data collection mechanisms listed below:

• SNMP
• CLI
• API
• DB Connection
• Webhook
• Netflow
• gNMI/MDT

2. A flexible user interface to set different sampling intervals for data collection.

Raw Counter Reports

This is one of the required use cases widely seen across SPs as their existing ecosystem has OSS which are dependent on custom raw information for further processing. With KPI report Matrix also supports the generation of raw data, to support customers’ existing operations in the environment.

Matrix value adds and differentiators:

1. Users are relieved from the manual configuration burden when it comes to introducing new counters and schema updates. Matrix effortlessly detects and incorporates any changes in the SSD automatically.
2. There are no limitations on rows, columns, or size; all counter details can be accessed in a single file unless there are limitations imposed by Excel. This eliminates the overhead at the OSS or any manual processes involving the examination of multiple files for analysis.

KPI Reports

Matrix processes and calculate the defined KPI during the processing of the bulkstats files based on the sampling interval set. These calculated values get stored in the Database for historical analysis. Matrix provides a flexible user interface to configure any KPI of interest in the case provided KPIs are not serving the purpose. It also allows the user to add in threshold to the defined KPI for alert notification and choice to choose the following aggregation methods.

• Average - averaging all samples for a node or nodes in the network
• Peak - considering only the peak (max.) of all samples for a node or nodes in the network
• Sum - adding up all samples for a node or nodes in the network

Matrix value adds and differentiators:

• Flexible user interface to add any counter, schema and KPI.
• Network-wide, node-wise aggregations.
• User interface to configure thresholds and forward threshold crossing alerts to NBI.
• Report framework to generate any custom reports.
• Email notification/reports.
• Performance KPI alerts can be generated on crossing user-definable thresholds or on ML-based learnings.
• Alerts can be forwarded to northbound systems (for ex. BPA, Netcool, Prometheus, ServiceNow etc) via REST API or SNMP trap (v3) and can be pushed to Kafka topic from which consumers can consume it for further processing.

Visualization and Dashboard

Visualization and dashboards play a crucial role in KPI monitoring. Matrix offers various charts and graphs, to convert raw KPI data into visually engaging and easily comprehensible representations. Historical trends over the period give insights into long-term performance to make informed decisions, and plan network enhancements or optimizations accordingly.

Matrix value adds and differentiators:

• A flexible user interface to create a custom graphical view of KPI data.

Matrix Custom Dashboard Panel

• N number of dashboards with different views can be created and role-based views can be granted to any of the dashboards. For example; the Executive dashboard is only made available to the executive team.

Matrix RBAC Panel

• Filter options in any dashboard will help you do a comparative analysis of KPIs across different network elements, regions, or time periods. By comparing KPIs side-by-side, network operators can identify underperforming areas, pinpoint bottlenecks, and allocate resources effectively.
• KPI Work Bench: Out of the box Matrix feature often offers drill-down capabilities, allowing users to explore KPI data at different levels of granularity and aggregation. This feature enables network operators to dive deeper into specific KPIs or segments of the network, identify root causes of performance issues, and take appropriate action.
• With the default KPI plot feature on the dashboard, users can easily plot and visualize the Top N/Worst N node view.

	Matrix KPI Workbench

• The dashboard allows users to export data in a tabular representation.

Data Aggregation

KPI data aggregation enables businesses to gain a holistic view of performance, identify areas for improvement, and make data-driven decisions. It provides a comprehensive understanding of how key metrics evolve and helps track progress towards organizational goals. Cisco CX Matrix provides different levels of data aggregation.

Matrix value adds and differentiators:

• Supports different aggregation logic such as average (avg), minimum (min), and maximum (max).
• Enables aggregations at the node, network, and region levels.
• Provides flexibility for aggregations based on various time windows, including hourly, daily, and custom time intervals.

South Bound Integration

Matrix offers a wide variety of data collection mechanisms, including bulkstats and SSD files. In the case of MPC, the bulkstats and SSD files play a crucial role. Southbound devices periodically transmit both SSD and bulkstats files to Matrix. However, the SSD file is pushed less often than bulkstats, as it only changes when there are modifications to the network device configuration. As a result, the SSD file is sent to Matrix either when alterations occur or at scheduled intervals, such as monthly or weekly. Device integration or onboarding is achieved through successful parsing of this SSD file.

Matrix value adds and differentiators:

• Provides a wide variety of data adapters.
• Supports both legacy and new technologies.
• Includes adapters for webhook integration.
• Offers adapters for telemetry data integration.

Devices Support list based on current implementation:

• ASR5500
• vePDG
• MME
• SGSN
• SGW
• PGW
• UPF
• CP
• SAEGW
• AMF
• SMF
• PCF
• NRF
• CPS (PCRF)
• UCS
• ESC
• ACI
• APIC
• NXOS
• CVIM

Northbound Integration

Matrix is designed to seamlessly integrate with existing OSS applications in the environment. It provides the necessary interfaces, protocols, and APIs to enable smooth communication and data exchange between Matrix and other OSS systems. It offers a user-friendly interface for adding Northbound applications. The communication between Matrix and the Northbound application (NBA) is established through standard REST API and SNMP calls. By integrating Matrix with other OSS applications, organizations can leverage the capabilities of multiple systems to improve efficiency and productivity.

Matrix value adds and differentiators:

• Flexible user interface for NBI configuration.
• Option to exclude the nodes for which alert forwarding is not required.
• ITSM/ticketing tool support.
• Alarm enrichment, correlation.
• Provides multiple interfaces to integrate with northbound systems (REST, SFTP, SNMP v3).
• Alarms can be pushed to Kafka so that consumers can consume further.

List of NBIs supported:

Fault Monitoring

Fault monitoring through SNMP traps is a method used to detect and respond to network and system faults proactively. SNMP traps are asynchronous notifications sent by network devices or systems when predefined events or conditions occur, such as hardware failures, network outages, or configuration issues. By configuring SNMP trap receivers, administrators can capture and analyze these traps in real time, allowing for immediate alerting and troubleshooting. Fault monitoring through SNMP traps enables quick identification and resolution of issues, minimizing downtime, and ensuring the reliability and availability of the network infrastructure.

Matrix value adds and differentiators:

• It also supports event-driven alarms via webhook.
• Filtering enrichment and correlation options with multiple data sources.
• Option to exclude the nodes for which alert forwarding is not required.
• ITSM/ticketing tool support.

Fault monitoring feature list:

Architecture

Cisco Prime consists of unit servers hosting the VNEs, a gateway server, an embedded database or external Oracle database, and Windows-based clients. Depending upon the scale of the network, Cisco Prime unit and gateway server software can be deployed on a single server or across multiple servers for a distributed architecture. Cisco Prime deployment is supported on bare metal or virtual servers. Systems can also be configured in various standby/local/Geo high-availability modes to help ensure business continuity.

Prime Deployment Architecture

Cisco Matrix follows a microservice architecture composed of small, independent services that communicate with each other to provide end outcomes. Each service focuses on a specific business capability and can be developed, deployed, and scaled independently. This architecture promotes modularity, scalability, and resilience, allowing efficient development, easier maintenance, and the ability to adapt to changing requirements. It also brings flexibility, allowing teams to choose the most suitable technology stack for each service, depending on requirements. Matrix supports Kubernetes, docker-based deployment on virtual servers and the cloud. A wide variety of deployment configurations can be tailored to meet the needs of each customer environment.

Matrix Deployment Architecture

While Cisco CX Matrix exhibits higher virtual machine (VM) and database sizing requirements than Prime, it compensates for these demands with a distinct advantage in features, cutting-edge technology, superior scaling capabilities, exceptional performance, and advanced visualization functionalities.

Source:

Prime: https://www.cisco.com/c/en/us/td/docs/net_mgmt/prime/network/5-2/installation/guide/CiscoPrimeNetwork52InstallationGuide/installation_overview.html

Matrix: https://cisco.sharepoint.com/sites/Cross-DomainAnalytics/SitePages/Matrix-Analytics-Release.aspx

Note: Matrix architecture displayed above is with docker in local HA mode, on cloud and K8 environment this will differ.

Deployment facilitation features list:

The platform features list:

Conclusion

This concludes that the feature set of Cisco CX Matrix provides Network-wide Visibility with a range of innovative capabilities that will significantly enhance user experience, streamline operations, and position our offerings at the forefront of industry standards.

Interested in migrating/need more details about the product, reach out to these contacts,

• akhire@cisco.com
• ask-matrix@cisco.com