A major worldwide outage that affected Google Cloud services on Thursday resulted in massive downtime across major internet platforms such as music streaming service Spotify and messaging app Discord.
Google’s engineering teams busily working to resolve the problem, most services were reactivated for about 60,000 affected users within a matter of hours.
The disruption started at about 1:50 PM ET and impacted a number of services that depend on Google Cloud infrastructure. In addition to Spotify and Discord, Google-owned services like Gmail, Google Drive and Google Meet also saw intermittent problems.
Downdetector, a website that tracks outages. com showed it peaked with more than 46,000 reports for Spotify and almost 11,000 reports for Discord in the United States when the outage was at its worst.
Engineers at Google Cloud were able to pinpoint the cause and confirmed it was an invalid automated quota update pushed globally to their API management systems. This resulted in external API requests being refused throughout a number of Google Cloud products.
Google bypassed the quota check that was causing the issue, and most regions recovered quickly (in about 2 hours). Although some regions, such as us-central1, took a bit longer to recover due to an overwhelmed quota policy database, the overall turnaround was swift and well done!
Google has since issued an apology, claiming: We are deeply sorry about the disruption that this service/service outage has caused to all of our users, Google and their customers”. It will do better.”
Some of the most recognizable companies in the world, as well as thousands of small businesses and startups, rely on Google Cloud for their workloads, Poellet continued.
The company has promised a full incident report within the next few days explaining the root cause, a detailed timeline and effectiveness of response to mitigate potential recurrence.
While the episode was resolved relatively rapidly, it offered another reminder of the deep links between the internet and the many important online services that rely on a small number of leading cloud providers.