Recent Backups & Availability
Identified - We've received reports of numerous recent backups not being taken. It appears that the interim backup system was unable to keep up with the demand and, in light of that, has very sporadically available restoration points. At this time, we have restored the original backup system, the one capable of handling all of our original backups, allowing access to backups from March 4th to April 4th. We're working to migrate the backups from the interim system, purge incomplete backups from the power outage, restore access to backups between April 20th and May 5th, and restore overall general anticipated functionality to the backup system.

Moving forward, the backup system will be migrated to a decentralized platform so that this matter will not recur in the future. Unfortunately, no ETA is currently available due to the ongoing roll out of the Prisma v1.24.0 update and updates that'll be happening thereafter as we make our way through the alpha and beta phases of additional game servers.
May 11, 07:43 UTC
Uptime over the past 90 days. View historical uptime.
Prisma - Game Hosting Platform Operational
90 days ago
99.99 % uptime
Today
Ashburn, VA, USA Operational
90 days ago
99.99 % uptime
Today
Dallas, TX, USA Operational
90 days ago
100.0 % uptime
Today
Chicago, IL, USA Operational
90 days ago
100.0 % uptime
Today
Miami, FL, USA Operational
90 days ago
100.0 % uptime
Today
Los Angeles, CA, USA Operational
90 days ago
100.0 % uptime
Today
Phoenix, AZ, USA Operational
90 days ago
100.0 % uptime
Today
Seattle, WA, USA Operational
90 days ago
100.0 % uptime
Today
Montreal, QC, Canada Operational
90 days ago
99.98 % uptime
Today
Amsterdam, NL Operational
90 days ago
99.99 % uptime
Today
Frankfurt, DE Operational
90 days ago
100.0 % uptime
Today
London, UK Operational
90 days ago
100.0 % uptime
Today
Roubaix, FR Operational
90 days ago
100.0 % uptime
Today
Warsaw, PL Operational
90 days ago
100.0 % uptime
Today
Singapore, SG Operational
90 days ago
100.0 % uptime
Today
Sydney, AU Operational
90 days ago
100.0 % uptime
Today
MySQL Service Operational
90 days ago
100.0 % uptime
Today
Ashburn, VA, USA Operational
90 days ago
100.0 % uptime
Today
Chicago, IL, USA Operational
90 days ago
100.0 % uptime
Today
Dallas, TX, USA Operational
90 days ago
100.0 % uptime
Today
Miami, FL, USA Operational
90 days ago
100.0 % uptime
Today
Los Angeles, CA, USA Operational
90 days ago
100.0 % uptime
Today
Phoenix, AZ, USA Operational
90 days ago
100.0 % uptime
Today
Seattle, WA, USA Operational
90 days ago
100.0 % uptime
Today
Montreal, QC, Canada Operational
90 days ago
100.0 % uptime
Today
Amsterdam, NL Operational
90 days ago
100.0 % uptime
Today
Roubaix, FR Operational
90 days ago
100.0 % uptime
Today
London, UK Operational
90 days ago
100.0 % uptime
Today
Frankfurt, DE Operational
90 days ago
100.0 % uptime
Today
Singapore, SG Operational
90 days ago
100.0 % uptime
Today
Sydney, AU Operational
90 days ago
100.0 % uptime
Today
Prisma Control Panel Operational
90 days ago
100.0 % uptime
Today
POP - Ashburn, VA Operational
90 days ago
100.0 % uptime
Today
POP - Dublin, Ireland Operational
90 days ago
100.0 % uptime
Today
POP - Sydney, Australia Operational
90 days ago
100.0 % uptime
Today
Main Website Operational
90 days ago
100.0 % uptime
Today
cubedhost.com ? Operational
90 days ago
100.0 % uptime
Today
Billing API Operational
90 days ago
100.0 % uptime
Today
Help Center ? Operational
90 days ago
100.0 % uptime
Today
Credit / Debit Card Processor (Stripe) API Operational
90 days ago
100.0 % uptime
Today
Cloudflare Cloudflare API Operational
Cloudflare Cloudflare Workers Operational
Cloudflare CDN/Cache Operational
Cloudflare CDN Cache Purge Operational
Cloudflare Cloudflare Recursive DNS Operational
Cloudflare Cloudflare Authoritative DNS Operational
Cloudflare DNS Root Servers Operational
Cloudflare DNS Updates Operational
Cloudflare SSL Certificate Provisioning Operational
Internal - AWS Operational
90 days ago
100.0 % uptime
Today
AWS ec2-us-east-1 Operational
90 days ago
100.0 % uptime
Today
AWS ec2-us-east-1 Operational
90 days ago
100.0 % uptime
Today
AWS elasticache-us-east-1 Operational
90 days ago
100.0 % uptime
Today
AWS elb-us-east-1 Operational
90 days ago
100.0 % uptime
Today
AWS lambda-us-east-1 Operational
90 days ago
100.0 % uptime
Today
AWS rds-us-east-1 Operational
90 days ago
100.0 % uptime
Today
AWS ses-us-east-1 Operational
90 days ago
100.0 % uptime
Today
AWS sns-us-east-1 Operational
90 days ago
100.0 % uptime
Today
AWS sqs-us-east-1 Operational
90 days ago
100.0 % uptime
Today
GSDB Operational
90 days ago
100.0 % uptime
Today
GSDB API Operational
90 days ago
100.0 % uptime
Today
Operational
Degraded Performance
Partial Outage
Major Outage
Maintenance
Major outage
Partial outage
No downtime recorded on this day.
No data exists for this day.
had a major outage.
had a partial outage.

No incidents or maintenance related to this downtime.

System Metrics Month Week Day
Global Memory Utilization
Fetching
Past Incidents
May 13, 2021
Software Upgrade & Migration
Completed - A few bugs are present in the v1.24.0 release and an update will be made before the weekend to resolve a vast majority of the troubles brought up by the release. Overall, the billing migration has been successful, and it seems the (obviously!) most important update is being well received: dark mode.

As always, if you experience any troubles, please reach out to us as soon as possible. We're here to help!
May 13, 07:13 UTC
Verifying - The maintenance is now complete. Orders and registration have been fully reopened. Please contact us immediately if you experience any troubles regarding payments or service management. We will continue to monitor on our end.
May 12, 01:38 UTC
Update - We've pushed a potential fix for the last outlying major issue, which would have affected orders from being completed successfully. If all goes well, the maintenance will be complete in 30 minutes or less. We'll update once we know more!
May 11, 22:04 UTC
Update - We are finalizing the updates to the billing system now. Due to a configuration issue, the database behind the Prisma control panel and billing system will be rebooted shortly. A brief interruption may be noticed twice as the database servers switch to their secondary servers, but service will resume quickly both times.
May 11, 19:03 UTC
Update - All of our nodes are now running a version of v1.24.0 - there are some small updates being pulled to fix specific use cases (e.g. non-default Java versions). Any smaller updates will be applied in the next 10 - 15 minutes automatically.

At this time, Prisma v1.24.0 is considered to be fully available and operational. We are continuing our maintenance on the billing system, however, registrations and orders are available.
May 11, 14:14 UTC
Update - An additional update is making it's way out now to resolve the blank/white page in Prisma on page load. Apologies for the interruption!
May 11, 12:33 UTC
Update - Prisma v1.24.0 is now rolling out to all daemons and to the control panel. Services may be unreachable for a period of time while the daemons and panel servers restart.
May 11, 12:15 UTC
Update - The billing update is now complete, further configuration is now taking place post-migration. Additionally, we are validating the transferred data before proceeding with the Prisma v1.24.0 update.
May 11, 08:14 UTC
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
May 11, 07:00 UTC
Scheduled - The CubedHost Operations team will be performing software upgrades and complete a billing migration at 7AM UTC / 2AM CDT on May 11th, 2021.
May 11, 02:18 UTC
May 12, 2021
May 11, 2021

Unresolved incident: Recent Backups & Availability.

May 10, 2021
Prisma Upgrade
Completed - Due to the current time and the Prisma update not having been rolled out yet, we are holding off on this for one more night to resolve some last minute issues that have popped up. We'll be back this evening for the final run. Thanks for your patience & understanding in this matter!
May 10, 13:06 UTC
Update - We apologize for the delay on this maintenance; due to some issues that had presented themselves with the billing maintenance, we were required to hold off of the update for a brief period of time.

We will be beginning the maintenance on the Prisma control panel soon. Several actions may be interrupted. More information will be available as we proceed with the update.
May 10, 11:38 UTC
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
May 10, 07:03 UTC
Scheduled - The CubedHost Operations team has scheduled a maintenance window to upgrade the Prisma control panel and server management daemon. During this time, management of services may not be possible while the upgrade is rolling out.
May 9, 21:55 UTC
Billing System Migration
Completed - Due to the current time and the Prisma update not having been rolled out yet, we are holding off on this for one more night to resolve some last minute issues that have popped up. We'll be back this evening for the final run. Thanks for your patience & understanding in this matter!
May 10, 13:06 UTC
Verifying - Another migration, another trouble that arises to attempt to bring down our efforts. Alas, it was no challenge, rather a minor inconvenience.

The billing system migration has been completed, we are now verifying data and service information, as well as last minute necessary adjustments. Some data will be missing until later this evening, such as account credit amounts and legacy support tickets. We'd like to triple check the data regarding these items, but it requires some additional tools that are not currently available. We will delay crons relating to service termination and suspension until 6PM CDT, as opposed to the standard 2PM CDT, to ensure any services relying on account credit will not experience an interruption due to the temporarily missing data.

The billing system remains in "maintenance mode", and will be until after the Prisma upgrade takes place. We anticipate this to be completed within the hour.
May 10, 11:36 UTC
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
May 10, 06:00 UTC
Scheduled - The CubedHost Operations team has schedule a migration window on Monday, May 10th, 2021 at 6AM UTC (1AM CDT). During this migration window, billing and service changes may not be possible up to the full duration of the maintenance. We do not anticipate the change over to take more than 4 hours, but we are anticipating any post-migration changes or verification that may need to take place.
May 9, 21:52 UTC
May 9, 2021

No incidents reported.

May 8, 2021

No incidents reported.

May 7, 2021
Prisma Upgrade
Completed - Unfortunately, due to some troubles with the billing software migration, we are postponing this maintenance until Monday. Additional details will be available on Sunday morning.
May 7, 10:06 UTC
Update - We are still working on the billing migration, and as a precaution, we have not yet begun the Prisma update rollout. Once we've confirmed all data has been migrated over, we will proceed with the Prisma update.
May 7, 06:24 UTC
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
May 7, 05:00 UTC
Scheduled - We'll be performing updates to the Prisma control panel and daemon software on Thursday, May 7th at 5AM UTC / 12AM CDT. During this maintenance window, it's possible that management of your Minecraft server(s) may not be possible for a short duration due to major software changes.

Services will remain online without interruption, however, scheduled tasks may not run for a short period of time while the daemon reboots and comes back online.
May 6, 14:02 UTC
Billing Software Migration
Completed - Unfortunately, due to an internal issue regarding copying over some specific encrypted data, we're postponing both maintenances until Monday. Additional details will be available on Sunday regarding the next maintenance window. Services to the existing billing software will be restored momentarily.
May 7, 10:04 UTC
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
May 7, 04:00 UTC
Scheduled - We'll be replacing a crucial part of our infrastructure: our billing software. On Friday, May 7th at 4AM UTC / 11PM CDT (May 6th), we will be replacing the billing software in full to resolve years-old outstanding issues with the existing billing solution.

Due to the nature of this task, billing access will be completely disabled within Prisma, the legacy billing area will be retired in full, and parts of the Prisma control panel may be slow or unable to load data and/or may be fully unresponsive. Logins will not be affected once the update to the Prisma control panel is completed. Once the data has been converted and verified, billing access will be restored in full.

Please note: we've scheduled this maintenance for a 12 hour window. We do not anticipate using the full duration of this maintenance window, but as always, it's best to prepare for any potential complications.

Below, you'll find a list of anticipated unreachable aspects of the Prisma control panel:
- Billing, profile, and account management (inclusive of all aspects of billing: invoices, orders, transactions, etc.)
- Service upgrades/downgrades
- Account registration
- New Server functionality / service creation
May 6, 14:03 UTC
May 6, 2021
Billing Software Migration
Completed - Due to a number of technical issues that have come up with our production infrastructure, namely the logistics behind replacing an entire billing system, the upgrades and migration have not yet taken place. Some changes have taken place to the infrastructure behind the scenes, but it will not be possible to migrate this morning.

We'll be rescheduling this for midnight CDT tomorrow; roughly 16 hours from this posting.
May 6, 14:01 UTC
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
May 6, 11:01 UTC
Scheduled - We'll be replacing a crucial part of our infrastructure: our billing software. On Thursday, May 6th at 11AM UTC / 6AM CDT, we will be replacing the billing software in full to resolve years-old outstanding issues with the existing billing solution.

Due to the nature of this task, billing access will be completely disabled within Prisma, the legacy billing area will be retired in full, and parts of the Prisma control panel may be slow or unable to load data and/or may be fully unresponsive. Logins will not be affected once the update to the Prisma control panel is completed. Once the data has been converted and verified, billing access will be restored in full.

Please note: we've scheduled this maintenance for a 12 hour window. We do not anticipate using the full duration of this maintenance window, but as always, it's best to prepare for any potential complications.

Below, you'll find a list of anticipated unreachable aspects of the Prisma control panel:
- Billing, profile, and account management (inclusive of all aspects of billing: invoices, orders, transactions, etc.)
- Service upgrades/downgrades
- Account registration
- New Server functionality / service creation
May 5, 17:42 UTC
Prisma - Major Upgrade
Completed - Due to a number of technical issues that have come up with our production infrastructure, namely the logistics behind replacing an entire billing system, the upgrades and migration have not yet taken place. Some changes have taken place to the infrastructure behind the scenes, but it will not be possible to migrate this morning.

We'll be rescheduling this for midnight CDT tomorrow; roughly 16 hours from this posting.
May 6, 14:01 UTC
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
May 6, 14:00 UTC
Scheduled - Due to a number of technical issues that have come up with our production infrastructure, namely the logistics behind replacing an entire billing system, the upgrades and migration have not yet taken place. Some changes have taken place to the infrastructure behind the scenes, but it will not be possible to migrate this morning.

We'll be rescheduling this for midnight CDT tomorrow; roughly 16 hours from this posting.
May 6, 14:00 UTC
In progress - Scheduled maintenance is currently in progress. We will provide updates as necessary.
May 6, 11:01 UTC
Scheduled - We'll be performing updates to the Prisma control panel and daemon software on Thursday, May 6th at 11AM UTC / 6AM CDT. During this maintenance window, it's possible that management of your Minecraft server(s) may not be possible for a short duration due to major software changes.

Services will remain online without interruption, however, scheduled tasks may not run for a short period of time while the daemon reboots and comes back online.
May 5, 17:28 UTC
May 5, 2021
AMS3 - System Unresponsive
Resolved - This incident has been resolved.
May 5, 01:24 UTC
Monitoring - A fix has been implemented and we are monitoring the results.
May 5, 01:02 UTC
Identified - We've applied some software update to resolve the matter. The system is now rebooting to apply the changes.
May 5, 01:01 UTC
Investigating - We are currently investigating this issue.
May 5, 00:56 UTC
May 4, 2021

No incidents reported.

May 3, 2021

No incidents reported.

May 2, 2021

No incidents reported.

May 1, 2021

No incidents reported.

Apr 30, 2021
Backup System Outage
Resolved - We've confirmed the system to be operational and all data is in tact. We're working to consolidate the data to the original system until our new backup solution is in place. This may lead to slow restoration times on a temporary basis.

If you wish to obtain a copy of the files from on or before April 6th, please reach out to us as soon as possible. In the interest of resuming our 30 day backup window, the recovered backups will be available for up to 14 days; any backups from March 15th or older may be pruned in the next 72 hours.

At this time, we are considering this incident resolved. Once we've completed some various upcoming maintenances, we will schedule a window for the migration of our backups to the new solution.
Apr 30, 22:14 UTC
Update - We've received word that the system is online and operational. We will confirm this during business hours on Friday (today) and evaluate the actions necessary moving forward. Any pending backup restorations or download requests for prior to April 6th will be handled appropriately after our evaluation later today.
Apr 30, 06:59 UTC
Update - Unfortunately, there are still no updates to this incident. We've reached out over social media today to try and prompt at least any sort of response to understand the scope of what's happening with the system.

Regarding the new backup system - backups have now been limited to 10 days due to the amount of data that our customer servers require. We're looking at rolling out yet another new solution in the coming weeks to replace both of these systems in the future, opting for a decentralized backup solution. This solution is on the backlog of our tasks until Prisma v1.24.0 is released, likely to be this coming week. Until then, we may opt for twice daily backups for a longer period of time over thrice daily for 10 days. More information will become available via this status page post once we have it available.

We apologize for the inconvenience caused by these issues prompted by both the old backup system availability and the decreased longevity of backups currently being kept. Thank you for your patience while we continue to work to resolve the matter.
Apr 24, 17:14 UTC
Update - We've received word that the system is online, but the server is not reachable via ping. We've escalated the issue with the datacenter to resolve the matter. If all goes well, backups from this system will be capable of being reached within the next 24 hours.
Apr 14, 06:07 UTC
Update - We're still awaiting additional information from the datacenter. Unfortunately, the system has yet to come back online.

Restorations may be slow for the next few hours - we are running optimization tools on the system to improve our ability to store a larger number of backups.
Apr 8, 20:34 UTC
Update - We've received word that power has been restored and the datacenter is now working on restoring all of their networks. We've asked for an update and confirmation if the system was affected by water damage. Once we know more, we'll update you here.
Apr 6, 23:58 UTC
Update - There are several reports coming in that the datacenter is back online and individual servers are coming up slowly, one by one. At this present moment, the backup system has not come back online. As soon as the system comes back online, we will copy over backups taken in the past 48 hours to the old system to restore all access to backups.
Apr 6, 13:14 UTC
Update - We've received new information from the datacenter:
- Estimation of 90-95% of their servers are entirely unharmed.
- Inspection has taken place and they currently have electricians on-site to resolve issues discovered during the inspection.
- Current ETR is 3:00PM MDT on April 6th, 2021.

It's unclear if the backup system is affected in the small percentage that has any damage to it, and as a byproduct of the potential of any damaged components, the estimated restoration time may not be accurate.
Apr 5, 22:10 UTC
Update - We are now restoring the functionality of backups being taken on an automatic basis. These will occur at a reduced rate of 3 backups per day with a maximum of up to 15 days (down from 30) until we're able to restore service to the previous backup system or the next iteration of our backup solution. The new interim system has a reduced capacity of 20TB, compared to our previous 36TB.
Apr 5, 19:48 UTC
Monitoring - Backups are completing successfully; it will take several hours for the initial backup process to complete in full. We will resume automatic backups April 5th, 2021 between 12PM - 4PM CST.
Apr 5, 05:56 UTC
Update - We're are now beginning a test of the backup system; if all goes well, this will be the first new backup available via Prisma.
Until the previous system is back online and checked for data consistency, any backups prior to April 5th, 2021 at ~12AM UTC should be considered potentially unusable.
Apr 5, 05:45 UTC
Identified - We have received confirmation that the power outage incident was due to a fire at the datacenter during a load testing of a UPS system. This prompted the power being cut to the core network and triggered the datacenter's fire suppression. The local fire department chose to cut all power as a precaution. The facility is now awaiting an emergency inspection to receive the all clear to resume services as normal.

From what we're being told, it's likely that there has not been any data loss and the system should be recoverable. If the system needs any components replaced due to water damage, it's likely it will take several days to recover the data on the existing system.

Regarding the replacement interim system, we are confirming all security access has been rolled out to all of our nodes prior to beginning a new backup run. We'll update here once we have any more information. ETA for new backups starting: < 1h.
Apr 5, 05:34 UTC
Update - We've identified third-party claims of potential power loss and potential damage to unspecified equipment; it is unclear if this is referencing the electrical/UPS systems or otherwise. We are actively deploying a new system to replace the existing system until further notice.
As we retain backups in an off-site facility, completely unrelated to all production services for service continuity reasons, this does not impact any services other than the inability to restore backups. The new deployment will also be in an off-site facility. Currently, no ETA is available for the completion of the deployment.

Until the existing system becomes available, no historic data will be available for restoration prior to today. We apologize for the inconvenience this may cause and thank you for your patience while we work to resolve this matter.
Apr 5, 02:43 UTC
Investigating - We are looking into an issue affecting our off-site backup solution. Until this incident is resolved, backups will not be available for restoration from within Prisma.
Apr 5, 00:30 UTC
Apr 29, 2021

No incidents reported.

← Incident History Powered by Statuspage