In 2012, a storm took down one of AWS datacenters in Virginia. The ELB service stopped working and, as a result, the likes of Netflix, Pinterest and Instagram went offline. In the same year, hurricane Sandy took down several carriers, including Equinix and Level 3. In 2015, four successive lightning strikes hit Google’s datacenter in Belgium causing data loss for a subset of customers.
(don’t miss The Verge report into rescuing Manhattan’s drowned internet)
Force majeure types of events can and do happen more often than most people anticipate. Monitoring real clouds alongside your virtual clouds, therefore can be, time well-spent.
Here is how we do it.
Adding Weather Forecasts to Your Monitoring Workflow
We use Server Density to collect weather forecast metrics from Forecast.io. The goal here is to be able to receive alerts when hurricanes or hazardous storms are approaching our datacenters.
To do that, we’re able to use Foocast, an open source Server Density plugin that retrieves metrics from forecast.io.
To replicate our setup, you will need a Forecast.IO API key and a Server Density account (start a free trial here). Follow the installation steps and add all locations you want to monitor in a YAML file, like this:
File: datacenters.yaml
-----------------
google:
ie:
- location: Dublin, Ireland
nl:
- location: Eemshaven, Netherlands
fi:
- location: Hamina, Finland
be:
- location: St Ghislain, Belgium
aws:
us-east-1:
- location: Ashburn, North Virginia, USA
us-west-1:
- location: North California, USA
us-west-2:
- location: Oregon, USA
All you then need to do is setup your forecast notifications in the same way you would with any other alerts.
Cloud Monitoring Explained
But how is cloud monitoring possible? How does weather forecasting really work? We spoke to our friends at the Numerical Fluid Dynamics Group at the University of Zaragoza, and here’s what they had to say.
Real-cloud monitoring is possible through the use of Numerical Weather Prediction software. This is the same stuff your weatherman uses on TV.
Numerical Weather Prediction works by solving the partial-differential equations that govern how air, heat and moisture move about in the Earth’s atmosphere.
This is a numerically intense exercise, and most professional meteorological services rely on High Performance Computing infrastructure for the task (or on cloud computing).
Alternatively, you can monitor clouds from the comfort of your home. Professional-grade software is in the public domain, and your personal computer will usually suffice for crude, yet realistic, predictions.
The Weather Research and Forecasting model (WRF, pronounced warf) is perhaps the most widely used prediction tool. WRF was developed at the US National Center for Atmospheric Research (NCAR), operated by the University Corporation for Atmospheric Research (UCAR).
WRF is used by national meteorological agencies for day-to-day weather forecasts, and for research groups around the world to research extreme weather phenomena, such as hurricanes and tornados (and even the weather on Mars!)
WRF works by pulling boundary conditions from remote repositories. The aggregate of those will then formulate a weather model that generates worldwide forecasts. It then enhances the predictions for a country or region by successively embedding refined meshes, much like matryoshka dolls, honing in to the zone of interest.
For the home aficionado, a decent four-day forecast for a mid-sized country needs about four to six hours of CPU time on a modern desktop computer. This is an operation that can be run overnight.
Numerical Fluid Dynamics Group at the University of Zaragoza
Summary
Companies like Amazon are now claiming readiness for all weather contingencies. Google has even deployed Stormtroopers to keep their datacenters safe.
Still, we believe that adding weather forecasts into your monitoring workflow is a good time investment. At the very least, when you know of impending adverse weather in advance, you can postpone any major production work and avoid unnecessary risks.
What about you? Have weather conditions ever impacted your infrastructure? How do you stay on top of such contingencies and what tools / processes do you have in place?
The post Real Cloud Monitoring appeared first on Server Density Blog.