We've reached out to all customers affected by last night's data loss and are working with them to restore content wherever possible. In the mean time we've accelerated the frequency interval of our own backups, and are working towards a more rigorous long term data replication strategy. TLDR: Despite this issue being outside of our control, if it ever happens again we're going to be a lot better prepared.
Posted Nov 28, 2017 - 19:34 GMT
Update
So far we've determined that customer data which was created between precisely 12:52 UTC yesterday afternoon and 01:37 UTC early this morning has been lost. We are still working with our infrastructure provider to determine why this happened and if there is any chance of data recovery, however the outlook is not good. We are now working on reaching out to any affected customers directly by email.
Posted Nov 28, 2017 - 11:04 GMT
Identified
So far we've identified a 12 hour window where some new customer data was initially successfully created and then later went missing. We're working with our upstream provider to try to identify exactly what happened before determining what next steps to take.
Posted Nov 28, 2017 - 09:59 GMT
Investigating
Following the outage a few hours ago as a result of our upstream provider taking one of our database servers offline, we are investigating data loss being reported by a small number of customers in the last few hours. We're looking into this now and will update here as soon as we have more information.