Update - We've received word that the system is online, but the server is not reachable via ping. We've escalated the issue with the datacenter to resolve the matter. If all goes well, backups from this system will be capable of being reached within the next 24 hours.
Apr 14, 06:07 UTC
Update - We're still awaiting additional information from the datacenter. Unfortunately, the system has yet to come back online.
Restorations may be slow for the next few hours - we are running optimization tools on the system to improve our ability to store a larger number of backups.
Apr 8, 20:34 UTC
Update - We've received word that power has been restored and the datacenter is now working on restoring all of their networks. We've asked for an update and confirmation if the system was affected by water damage. Once we know more, we'll update you here.
Apr 6, 23:58 UTC
Update - There are several reports coming in that the datacenter is back online and individual servers are coming up slowly, one by one. At this present moment, the backup system has not come back online. As soon as the system comes back online, we will copy over backups taken in the past 48 hours to the old system to restore all access to backups.
Apr 6, 13:14 UTC
Update - We've received new information from the datacenter:
- Estimation of 90-95% of their servers are entirely unharmed.
- Inspection has taken place and they currently have electricians on-site to resolve issues discovered during the inspection.
- Current ETR is 3:00PM MDT on March 6th, 2021.
It's unclear if the backup system is affected in the small percentage that has any damage to it, and as a byproduct of the potential of any damaged components, the estimated restoration time may not be accurate.
Apr 5, 22:10 UTC
Update - We are now restoring the functionality of backups being taken on an automatic basis. These will occur at a reduced rate of 3 backups per day with a maximum of up to 15 days (down from 30) until we're able to restore service to the previous backup system or the next iteration of our backup solution. The new interim system has a reduced capacity of 20TB, compared to our previous 36TB.
Apr 5, 19:48 UTC
Monitoring - Backups are completing successfully; it will take several hours for the initial backup process to complete in full. We will resume automatic backups March 5th, 2021 between 12PM - 4PM CST.
Apr 5, 05:56 UTC
Update - We're are now beginning a test of the backup system; if all goes well, this will be the first new backup available via Prisma.
Until the previous system is back online and checked for data consistency, any backups prior to March 5th, 2021 at ~12AM UTC should be considered potentially unusable.
Apr 5, 05:45 UTC
Identified - We have received confirmation that the power outage incident was due to a fire at the datacenter during a load testing of a UPS system. This prompted the power being cut to the core network and triggered the datacenter's fire suppression. The local fire department chose to cut all power as a precaution. The facility is now awaiting an emergency inspection to receive the all clear to resume services as normal.
From what we're being told, it's likely that there has not been any data loss and the system should be recoverable. If the system needs any components replaced due to water damage, it's likely it will take several days to recover the data on the existing system.
Regarding the replacement interim system, we are confirming all security access has been rolled out to all of our nodes prior to beginning a new backup run. We'll update here once we have any more information. ETA for new backups starting: < 1h.
Apr 5, 05:34 UTC
Update - We've identified third-party claims of potential power loss and potential damage to unspecified equipment; it is unclear if this is referencing the electrical/UPS systems or otherwise. We are actively deploying a new system to replace the existing system until further notice.
As we retain backups in an off-site facility, completely unrelated to all production services for service continuity reasons, this does not impact any services other than the inability to restore backups. The new deployment will also be in an off-site facility. Currently, no ETA is available for the completion of the deployment.
Until the existing system becomes available, no historic data will be available for restoration prior to today. We apologize for the inconvenience this may cause and thank you for your patience while we work to resolve this matter.
Apr 5, 02:43 UTC
Investigating - We are looking into an issue affecting our off-site backup solution. Until this incident is resolved, backups will not be available for restoration from within Prisma.
Apr 5, 00:30 UTC