Description: Flexera One – IT Visibility – NAM – Data Processing Delays

Timeframe: November 7th, 2024, 2:13 PM to November 10th, 2024, 12:11 PM PDT

Incident Summary

On Tuesday, November 7th, 2024, at 2:13 PM PDT, delays in data processing were observed within the IT Visibility (ITV) platform, affecting customers in the NAM region. While the user interface and dashboards remained accessible, the inventory data updates were delayed, impacting the timeliness of dashboard insights. No other ITV services were affected during the incident.

The issue originated from a bottleneck in a core processing service, caused by an internal database's autoscaling challenges. The database’s primary node experienced slow response times, and secondary nodes were either in recovery mode or inaccessible due to insufficient disk space. The incident was further impacted by high incoming traffic and unexpected data configurations from a specific data source, which compounded the processing delays.

The backlog reduction efforts involved scaling resources, temporarily disabling traffic from specific data sources, and extensive collaboration with our service provider. The backlog was fully cleared, and all data processing returned to normal on November 10th, 2024, at 12:11 PM PDT.

Root Cause

Primary Root Cause:

The delays in data processing were caused by a bottleneck in a core processing service. This bottleneck arose from autoscaling challenges in the internal database, where the primary node experienced slow response times, and secondary nodes became either inaccessible or entered recovery mode due to insufficient disk space.

Contributing Factors:

1. High Incoming Traffic: Elevated levels of incoming traffic during the incident period increased the load on the processing service, exacerbating the delays.
2. Unexpected Data Configurations: Data from a specific source introduced unanticipated complexity, placing additional strain on the processing system and compounding the backlog.
3. Resource Constraints: The system's scaling mechanisms and resource allocation were insufficient to manage the combination of high traffic and complex data configurations, leading to prolonged recovery times.
4. Database Limitations: Disk space constraints in secondary database nodes further hindered the autoscaling and recovery process.

Remediation Actions:

1. Traffic Limitation: Traffic to the affected processing service was temporarily reduced to stabilize the system and prevent further strain on the database.
2. Resource Optimization: Additional resources were allocated to the processing service, significantly increasing its capacity to handle the backlog.
3. Targeted Data Handling Adjustments: Specific data sources contributing to the delays were temporarily disabled to alleviate pressure on the system.
4. Collaboration with Service Provider Support: Service Provider support teams were engaged to address the database autoscaling issues and assist in clearing disk space on affected nodes.
5. Incremental Backlog Reduction: The system processed the backlog in stages, monitoring progress continuously to ensure stability and gradual improvement.
6. Database Recovery: Secondary database nodes were re-synced and brought back online after addressing disk space limitations, restoring full functionality to the database.
7. Extended Monitoring: Real-time monitoring was conducted throughout the incident to track system performance and ensure the success of recovery efforts.

Future Preventative Measures:

This incident was caused by an extraordinary and rare occurrence of high data volume, which exceeded the system’s ability to scale automatically. Detailed discussions have been held with the service provider to understand the contributing factors and explore measures to prevent similar issues in the future.

To ensure operational stability, the following actions are being taken:

1. Continuous Monitoring and Alerts: Enhanced monitoring and alert mechanisms are being implemented to detect early warning signs of scaling challenges or abnormal resource usage.
2. Preventative Collaboration: Ongoing collaboration with the service provider will focus on identifying and addressing potential vulnerabilities in advance, ensuring a proactive approach to system stability.
3. Proactive System Evaluations: Regular evaluations of the system’s capacity and performance during peak scenarios will guide preemptive adjustments to resource allocation strategies.

This incident represents a rare occurrence of circumstances that exceeded the platform’s capacity to scale. However, we remain committed to proactively improving our processes, collaborating with our service provider, and implementing measures to prevent similar issues in the future while ensuring a seamless experience for our customers.