Timeline

• The table managing our sleepscore overflowed an int32 signed integer foreign key column.
• An ALTER was started but due to the size of the table we changed strategy and added a new column as BIGINT.
• The Index on this table was then added.
• Service was restored once an “ANALYZE” was performed on the table.
• Backfill process was then started to fill the old values of the new column.

Remediation Actions

• Monitoring of near depleted int32 columns already exists on primary keys, but monitoring needs to be added on foreign keys.

Incident has been resolved

A fix has been implemented, we're now monitoring

Timeline

• 11:55 detection of issue
• 12:05 change DNS to bypass CDN
• 12:15 shop website back to normal for most users
• 13:00 CDN provider back up

What happened

Our CDN provider experienced a widespread downtime, causing all requests to return 503 errors. DNS was then changed to bypass our CDN and received traffic directly, which fixed the issue once DNS was propagated.

incident resolved

A fix has been implemented and we are monitoring the results.

The issue has been identified and a fix is being implemented.