Backup System Outage
Incident Report for CubedHost, LLC
Resolved
Our operations team has fully resolved the issues impacting our backup system. Backups from November 30th through December 2nd are unavailable and backups on December 3rd and 4th are less frequent than normal. Our normal backup schedule will resume on December 5th.

Root cause analysis identified one failed HDD in the system's RAID array. Due to the size of the RAID array and drives in it, recovery took an extended period of time and proved challenging without causing a full outage of the system. At this time, all drives are healthy and array consistency is being checked. Once this check is complete, we can resume normal operations.

Backup restorations are available regardless of these issues, so please feel free to reach out to our team if you need one. Additionally, our operations team is looking into long-term solutions to mitigate this problem and prevent it from reoccurring in the future.
Posted 10 days ago. Dec 04, 2018 - 17:29 UTC
Identified
The CubedHost Operations team was notified of an outage regarding our backup system on November 29th after 8PM CT. A disk failure has been determined to be the cause of the system locking up and corruption on a partition that took several hours to repair overnight.

November 30th's backups at 1AM & 1PM (UTC) were not taken due to the outage, and as such, will not be available for restoration. A backup will be made after the drive has been replaced by the datacenter and the data has synced up to the new drive. No ETA is available at this time.
Posted 14 days ago. Nov 30, 2018 - 19:57 UTC