Root Cause Analysis: Workday Integration Performance Issue
Overview: On Jan 23 2023, customers experienced slower response times and intermittent service issues. We take these disruptions very seriously, and we want to share a transparent account of the issue, its root cause, and the steps we are taking to address it both in the short and long term.
Root Cause: The issue was traced back to our Workday integration.
Specifically:
- The integration was fetching and processing an unexpectedly large volume of data. - This data processing occurred on our web servers, which caused high memory usage and led to out-of-memory errors and service outage.
Impact:
- Customers experienced slow application response times or outages.
Fixes:
Short-Term:
We have optimized the processing of Workday data to reduce its memory usage and overall load on our web servers. This optimization has already been implemented and has mitigated the immediate issue.
Medium-Term:
To ensure long-term stability and scalability, we are in the process of moving this data processing workload off our web servers. Instead, it will be handled by dedicated worker nodes that are better equipped to process large datasets without impacting web server performance. This change is currently in progress and will be deployed as soon as possible.
Next Steps:
We will continue to monitor system performance closely to ensure the short-term optimizations are effective.
Once the medium-term fix is deployed, we will communicate this update to all customers.
We are also reviewing all other data integrations to proactively identify and mitigate similar risks.
We sincerely apologize for the inconvenience caused by this issue and appreciate your patience as we work to improve our systems. Ensuring a reliable and high-performing service for our customers remains our top priority.
If you have any further questions or concerns, please do not hesitate to reach out to our support team.
Posted Jan 23, 2025 - 14:28 PST
Monitoring
A fix has been implemented and we are monitoring the results.