Site Hack May 08
Posted: 21 May 2008, 19:31Now that the dust is settling since the hack last weekend, I just wanted to bring everyone up to speed with what happened.
What Was It?
There is a major SQL Injection Trojan sweeping the Internet, and if that means nothing to you, let me explain. SQL is the database technology we use on V-Flyer, as do many millions of sites out there. An SQL Injection is a particular type of hacking attempt which tries to squirt unwanted data into the database by adding SQL commands to the end of legitimate query strings. An SQL Injection Trojan is a virus which automatically probes websites to send SQL commands blindly to every page it finds that use query strings. In this particular case, the trojan infects host PCs, so potentially your own PC becomes the next machine to hack a website. The trojan that swept through V-Flyer last weekend would appear to be a particularly nasty example that may have already infected many millions of PCs, that are all now zombie hacking stations.
What Did It Do?
Our logs show that the attacks started as early as 5th May, but because I've been cleaning up as much of the code on V-Flyer as possible since our last hack, it didn't find a hole until the early hours of the 18th. When it found a hole, it squirted a javascript command into multiple tables, and when that javascript tag is rendered on a vulnerable PC, it turns it into a zombie. If you haven't already done so, make sure you have run a up-to-date virus sweep on your machine.
What Is The Fix?
The initial attempt to clean out the database by searching and replacing for the script tags wasn't totally successful, because of limitations on the types of fields SQL can auto replace on. A decision was made to roll back the data to the 17th May. This means we've lost a chunk of posts from Sunday - Tuesday, but I figured this was better than leaving traces of this particularly nasty virus script in our pages.
I've added code to protect from this particular attack, although there's no guarantee that a different type of trojan couldn't find a way in in the future. Unfortunately, like spam, hacking is just an irritating fact of life.
A side effect of these attack will be the slow-downs we've been seeing on the site of late. The reason being that we're getting millions of page requests from zombie terminals trying to find holes, and that means fully-rendering pages to send that will never be viewed. It eats up server resources. Part of the fix is to now only send a single line error response, which will hopefully reduce the server burden (although we are still being hit thousands of times a day by infected PCs looking for exploits).
I'd like to thank Paul, Ben and Richard for their assistance (and keeping my nerves calm!) over the last few days. It's been a bit tricky trying to fix this from a hotel room (and changing hotels just at the point the hack was at its worst), with the distractions of San Francisco and Vegas just outside the window! Hopefully things will be back to normal, although this hack is still out there and causing havoc across the Internet.
Pete
What Was It?
There is a major SQL Injection Trojan sweeping the Internet, and if that means nothing to you, let me explain. SQL is the database technology we use on V-Flyer, as do many millions of sites out there. An SQL Injection is a particular type of hacking attempt which tries to squirt unwanted data into the database by adding SQL commands to the end of legitimate query strings. An SQL Injection Trojan is a virus which automatically probes websites to send SQL commands blindly to every page it finds that use query strings. In this particular case, the trojan infects host PCs, so potentially your own PC becomes the next machine to hack a website. The trojan that swept through V-Flyer last weekend would appear to be a particularly nasty example that may have already infected many millions of PCs, that are all now zombie hacking stations.
What Did It Do?
Our logs show that the attacks started as early as 5th May, but because I've been cleaning up as much of the code on V-Flyer as possible since our last hack, it didn't find a hole until the early hours of the 18th. When it found a hole, it squirted a javascript command into multiple tables, and when that javascript tag is rendered on a vulnerable PC, it turns it into a zombie. If you haven't already done so, make sure you have run a up-to-date virus sweep on your machine.
What Is The Fix?
The initial attempt to clean out the database by searching and replacing for the script tags wasn't totally successful, because of limitations on the types of fields SQL can auto replace on. A decision was made to roll back the data to the 17th May. This means we've lost a chunk of posts from Sunday - Tuesday, but I figured this was better than leaving traces of this particularly nasty virus script in our pages.
I've added code to protect from this particular attack, although there's no guarantee that a different type of trojan couldn't find a way in in the future. Unfortunately, like spam, hacking is just an irritating fact of life.
A side effect of these attack will be the slow-downs we've been seeing on the site of late. The reason being that we're getting millions of page requests from zombie terminals trying to find holes, and that means fully-rendering pages to send that will never be viewed. It eats up server resources. Part of the fix is to now only send a single line error response, which will hopefully reduce the server burden (although we are still being hit thousands of times a day by infected PCs looking for exploits).
I'd like to thank Paul, Ben and Richard for their assistance (and keeping my nerves calm!) over the last few days. It's been a bit tricky trying to fix this from a hotel room (and changing hotels just at the point the hack was at its worst), with the distractions of San Francisco and Vegas just outside the window! Hopefully things will be back to normal, although this hack is still out there and causing havoc across the Internet.
Pete