Summary
On February 7, 2013, a network device issue impacted CRM Online Organization access worldwide, for a short period of time. The impact was total in North America, with lesser percentage of customers impacted in Europe and Asia Pacific. Microsoft engineers took action to resolve the issue.
Customer Impact
The network device failure impacted 100% of organizations hosted in the Americas datacenters, less than 50% Organizations in Europe and less than 10% Organizations in Asia Pacific. Impacted Customers were unable to access the CRM Online service during the incident.
Incident Start Date and Time
Thursday, 7 February 2013, 21:50 PST
Friday, 8 February 2013, 05:50 UTC
Friday, 8 February 2013, 13:50 Singapore
Date and Time Service was Restored
Thursday, 7 February 2013, 22:40 PST
Friday, 8 February 2013, 06:40 UTC
Friday, 8 February 2013, 14:40 Singapore
Root Cause
The Microsoft Networking team has finalized and published their Post Incident report for this service incident. The root cause was the misconfiguration of a network device that cause traffic saturation and connection failures.
Next Steps
- Audit all devices to ensure traffic redundancy is available and device configurations are resilient - Completed by Microsoft Networking
- Implement additional audits, monitoring and alerting to avoid misconfigurations – Ongoing by Microsoft Networking