Blog Post: Post Incident Report (PIR) | CRM Online Organization’s hosted out of one of the ScaleGroups in the America‘s datacenters were inaccessible

Summary

A small subset of CRM Online orgs served from America's datacenters were inaccessible. The issue was identified during a service health check investigation. It impacted less than 1% of customers in the region.

Customer Impact

Impacted customers could not access their CRM Online organizations 

Incident Start Date and Time

10^th February 2013 12:30 AM PST 

Date and Time Service was Restored

10^th February 2013 2:30 AM PST 

Root Cause

The SQL servers lost connectivity, which led to a loss of mirroring for all CRM Online Organization of the impacted ScaleGroup. Incorrect configuration settings on the SQL servers caused the issue.

Next Steps

1. Confirm that all deployed hardware has this configuration setting properly set - Completed by Dynamics CRM Service Engineering
2. Work with product team to ensure deployment automation and/or QC covers this setting - Ongoing
3. Continue working with the SQL Server Support team to identify the cause of the crashes and apply any provided fixes - Ongoing