Latest Updates
- 8/15 @ 9:56 AM CT – Issue resolved. All servers updated with hotfix to use Compatibility Level 1700. Cube rebuilds have concluded or are underway without further errors.
8/15 @ 8:21 AM CT – Fix identified and tested; cubes now building successfully. Code change bypasses recent Microsoft update, which caused the issue. Applying fix server-by-server over next few hours, then re-running jobs to load yesterday’s data.
Overview
Earlier today, a Microsoft Azure Analysis Services (AAS) update caused widespread cube rebuild failures across several Catalyst servers. Overnight integrations ran successfully, but the cubes did not rebuild, leaving yesterday’s data missing from cubes, reports, and visualizations.
Impact
Impacted:
Excel cubes
Power BI dashboards
Reports tab visualizations
Data in cubes (yesterday’s updates missing)
Not impacted:
Catalyst website access
Planning pages
Manage Actuals
Data integrations until the daily job stage
Root Cause
Microsoft released a new Compatibility Level (1702) for Azure Analysis Services. By default, our cube processing uses the newest version, which in this case caused all cube builds at that level to fail.
The new compatibility level appears to be faulty or incompatible with our environment.
Our Actions
Hotfix:
Updated our systems to lock Compatibility Level at a known stable version (1700) instead of automatically using the newest.
Applied this change to all servers.
Re-ran failed jobs to load yesterday’s data back into cubes.
Long-Term Prevention:
Investigating why Compatibility Level 1702 failed (Microsoft bug, change, or new requirement).
Updating our systems to only use major compatibility levels (e.g., 1600, 1700), which are more thoroughly tested and stable.
This will reduce risk of similar outages caused by untested minor version changes.
What You Should Do
No action is required on your part. If you experienced missing data earlier today, it should now be restored as cube rebuilds complete.
Next Steps
We will:
Continue monitoring cube rebuild performance.
Share final root cause confirmation from Microsoft.
Finalize long-term code changes to prevent this type of incident from happening again.
Submit a Request if you have questions or need assistance.
Comments
0 comments
Article is closed for comments.