Hosting Status Overview September 2011

Total uptime in September 2011 on Ad Delivery Platform 1: 99,78% (94 minutes downtime)

Total uptime in September 2011 on Ad Delivery Platform 2: 99,78% (94 minutes downtime)

Total uptime in September 2011 on Content Delivery Platform: 99,78% (94 minutes downtime)

On September 8, at 16.19 CET, a failure of the core network switches took down the entire network that connects our platforms to the internet. Because technicians were unable to connect to the switches remotely (for the very same reason), engineers had to be dispatched to the data center to correct the issue. The SLA with the upstream provider determines that a priority 1 incident like this must be resolved within 2 hours. In practice, the network was operational again in 94 minutes. In post-incident analysis, it was discovered that a software bug in the switches was the cause of this failure, even though it was initially reported as a power failure.

We have an online status dashboard of our platforms and services, maintained by Pingdom. Unfortunately, this external monitoring service is not always entirely reliable, we see ‘false positives’ from time to time, which means a ‘down’ status is reported at a time when all our systems are operating perfectly fine. The status dashboard is hosted externally, so that it will continue to work even in cases where there are technical issues in our own systems. We also use Twitter to send out quick updates in case of issues.

In addition to this external monitoring, we also use an internal monitoring system using tools like Cacti and Nagios. In case of technical incidents, technicians are alerted by e-mail and text messaging. We have 24/7 staff availability.