Incident report OpenX Source hack Tuesday August 2, 2011

On Tuesday August 2nd, there was a serious incident in several (but not all) OpenX Source ad server systems that are hosted by us for our customers. This blog post is intended to inform our clients and others about what happened and what the consequences were.

First Reports

At 2.15PM Amsterdam time we received the first report of problems while loading websites that carry ads that are served through OpenX Source. Within a few minutes it was clear that several OpenX Source systems sent a piece of unsafe html code along with every banner displayed. Soon after that we found this html code in the underlying database and began cleaning it.

Protocol for response

Despite the fact that we had never been confronted with an incident of this nature, we have a protocol that describes how to act in the event it would happen. This type of hacks in OpenX Source is unfortunately not uncommon, but we had not been affected by it ourselves yet. What we had not expected is that the tables in databases that had just been cleaned were once again infected several minutes afterwards. It was like carrying water to the sea.

Origin of the attack

It was discovered that a small program had been uploaded to one of our servers, which was being called frequently from an IP address in Romania. Each time this happened, a snippet of unsafe html code was inserted into a database. By blocking this IP address, we were able to avoid new infections.

Update August 3: we discovered that the actual hack occurred more than 1 year ago, when someone abused the security leak in an older version of the OpenX Source software to upload a malicious program onto the server.  The hacker then used this to create his own administrator user name and password. Even though the malicious program mentioned earlier was no longer on our servers since we upgraded to the most recent version of the OpenX Source software several months ago, the hacker still had an administrator username, unbeknownst to us. Yesterday, he used this username to connect to the system again to install an altered version of a plugin. By calling this altered plugin, he was able to insert the unsafe code into the banners. The fact that more than a year went by between the actual hack and the abuse of the illegally obtained access demonstrates that this was a carefully planned and executed criminal act.

Cause found in third party software

The malicious program mentioned above was uploaded to the server by means of a security leak in a piece of software that’s part of the OpenX Source ad server software. However, it was not made by OpenX developers but by a third party, who has placed it on the internet as open source. This software, named OFC2, is used to display statistics in the form of pretty graphs instead of lists of dull numbers. The developer of OFC2 had created a feature to upload files onto a server, most likely for his own testing purposes. Unfortunately, this feature has found its way into the final version. Someone with evil intentions discovered this and took advantage of it as described above to attack a number of OpenX systems hosted by us, with the aim of inserting said unsafe HTML code.

Potential effect: malware

When a person visiting a site gets to see a banner that contains this unsafe code, the browser will attempt to connect to a website which in this case is located in Russia. This website could be used by hackers to distribute malicious software (malware) or viruses. At the time of writing, we can no longer establish what this could have been, since the site in question is currently not online any more. In almost all cases, a good virus scanner will detect and block an event like this.

Complex clean up work

It took rather long, much longer than we would have wanted, to check and where needed clean all databases. We have several dozen of these systems under our care, and we wanted to check all of them with great care and attention. We have check each system individually ‘by hand’, even the ones where no problems had been reported.

Damaged trust

Like our clients that have been affected, we are very concerned about this incident, and especially about the impact it has had on a large number of websites and the people that like to visit them. We realize that the trust in our services has taken a serious blow. A phone conversation with a security expert in the US we had this evening, tells us that this type of attack currently happens frequently. Just like in all software (with MS Windows as the most notorious example, and another example this week with WordPress), there was a security issue in the OpenX Source software. Unfortunately, it turns out there are people who take advantage of this.

Could this have been prevented?

This type of incident is almost impossible to prevent, simply because it’s not know which security problems exist in software until the moment that someone discovers it and starts abusing it. What we can do is learn from this incident, to try and prevent similar incidents from occurring in the future. Unfortunately, this kind of incident occurs frequently but it is almost never publicized by those directly involved. Even software that appears to be safe now because there hasn’t been any problems yet can turn out to be unsafe later.

In closing

We sincerely regret this incident. In the coming days we will be contacting each of our clients to further explain what happened and to discuss if any additional measures from our side are needed.

Update August 3rd

A security analyst who independently monitored our platform and reviewed our approach of this incident commented as follows:

The response by the team was quick, and it appears they covered all the bases that needed to be covered. Simple fact is a determined attacker will penetrate any system s/he wants, eventually.

There was about a 4 hour window where this was seen, and from what I can tell the attacker didn’t have enough time to get the second half of the attack working. All requests to the “payload” domain were met with almost no content and an HTTP 403 response code. That content has now been replaced with a “discussion forum”, which fit the MO of who we believe this attacker to be. Adding new, malicious content to public forums is easy.

Share this on:
  • Twitter
  • Facebook
  • LinkedIn
  • email