Summary
On December 7th, the Microsoft Dynamics Service Engineering team resolved a CRM Online reporting feature outage, where all CRM Online customers were having issues with creating, editing, deleting, or running custom and out of box reports (oob). Although no escalations were received from the APAC or EMEA regions this issue would have impacted those customers as well.
The issue surfaced during a routine maintenance of a North American report server. These maintenances are transparent to end users because of the redundancy in the infrastructure. The upgrades are also tested in pre-production environments but unfortunately were undetected in this pass. During the upgrade, the report server database which is shared by multiple report service servers were updated as well which broke the reporting service server’s connection with the backup Report Server database. In order to resolve the mismatch and to reestablish the connection Service Engineering continued with the upgrade and proactively upgraded the report databases across all GEOs during dark hours.
Customer Impact
All CRM customers may have experienced issues with creating, editing, deleting, or running custom and out of box reports (oob).
Incident Start Date and Time
December 7th, 2012 2:00 PM PST
Date and Time Service was Restored
December 7th, 2012 7:00 PM PST
Root Cause
Database version mismatch which broke the connection with the backup report servers.
Next Steps
Issue | Next Step | Team Owner | Timeline |
Pre-production Testing | Update to test cases to increase effectiveness of catching such bugs | CRM Product and Service Engineering Teams | Ongoing |