I am become Death, destroyer of rootkits

So... last friday, I was at work and witnessed the most amazing thing someone can see in the field of IT. I will tell my grandkids about this one day.

Before I tell you what it was, let me tell you some background story. I work at a small and successful IT company with a few friends from university and other great coworkers and bosses. Our server farm (database and webservers) is physically located several dozen miles away from us, but they all have RealVNC installed so we can access them remotely. RealVNC basically shows you the desktop of the remote server, so you can work on them as if you were sitting right behind them. Our servers even have some kind of 'panic' switch installed on a hardware level that allows us to remotely reboot the servers in case of a serious crash, by sending a text message to a certain cellphone number (which is, of course, the coolest thing ever).

Anyway, two weeks ago I read an article about a vulnerability in RealVNC 4.1.1 at work, but since the proof-of-concept code published by the same site required you to use the server with VNC installed to load their website, which would then check for the vulnerability and tell you, we figured it was a phishing scam and forgot about it. I mean, if somebody told you they could hack your system, but to prove it, you'd have to turn off your firewall and antivirus, and open this strange-looking file... would you do it?

Well, this week we thought, in retrospect, that we should at least have checked the version number on our VNC servers and checked if our servers were actually open to attack.

Because we got attacked. Live. While me and two of my friends were looking at the server's screen at work. We actually witnessed the hacker hacking our server. Hollywood has nothing on this.
We VNC'ed into one of the servers for some routine work, and a command window was open with loads and loads of shell commands scrolling past - adding users, changing registry keys, downloading files, setting attributes (mostly 'hidden' and 'system') in an endless stream of DOS commands that were scrolling by.
At first, we thought it was our sysadmin, doing some complex Win-fu far beyond our IT abilities. But this didn't seem like something he'd do - why would he merge something into the Registry called 'hide2.reg'? Then the stream of shell commands stopped, and to verify what we thought, I typed at the command prompt:
C:\>who is this?
To our surprise and immediate panic, someone typed:
C:\>who is this? I'm PeeWee.
..and quickly closed all the command windows and disconnected from the server.

So we immediately called our sysadmin to tell him what was going on and ask for help. Now, mind you, none of us three (it was early in the morning and only us three had witnessed the owning of one of our servers, a fourth coworker was in the next room) are laymen when it comes to computers. So even while we were talking to our sysadmin on the phone, the other two made plans to shut down RealVNC on all servers and use Terminal Services (Microsoft's own remote access program) to connect to the servers again and upgrade RealVNC to the new updated version that was released on May 20, two days after the earlier proof-of-concept page was published. After we finished talking to the sysadmin, we went to work. One of my friends started upgrading RealVNC on the servers. The other started calling our bosses and tell them what just happened.
They told me to go and clean up the servers, if possible. And I never had as much fun that day at work before. I'm serious.

The first server I checked was the one we witnessed 'PeeWee' working on. Not really knowing where to start, I downloaded a few power tools from SysInternals.com and started snooping around. Mind you, I have written my own security programs, but they really don't compare to the ones you can get off the SysInternals website, all for free. Some even with source code included! They're awesome. And did I mention they started the whole Sony rootkit controversy?

Anyway, after a few hours of snooping around on our poor infected servers, I found the following:

• 
  4 of our 20-ish servers had been hacked.
• 
  All four had been 'converted' into FXP Warez sites. FTP server, remote access, rootkit, hidden folders for the warez, it was all there.
• 
  The hacker was still working on the server we caught him on, so that one hadn't been fully converted. It's possible we would have never found out about this if we hadn't caught him in the act.
• 
  The hacker had most likely found used a portscan on the default RealVNC port, found all of our servers, and used the brand new exploit. The only reason not ALL of our servers had been owned, was because they are all setup to lock themselves after a few minutes of being idle. The four servers that are used most often were not locked, and those were the four that were hacked.
• 
  The hacker had used the same package of tools on all the servers, though configured slightly differently to have the FTP servers identify themselves as part of different FXP networks. In other words, our friend PeeWee was probably a scanner/hacker for multiple FXP boards.
• 
  After I was done fixing the servers (which took me all day, i.e. roughly 7 hours) I had 6 megabytes of crap in zipped folders from each of the four infected servers for our sysadmin to look at. I had already looked at most of it and found a lot of fun things like passwords, FXP board names and the tactics used by these kind of hackers to hide their stuff on a compromised server.
• 
  PeeWee's package of tools consisted mostly of a hacked version of the Serv-U FTP server, a few trojans and droppers, a command-line remote access tool, and Hacker Defender.

Now, not all of you might be familiar with Hacker Defender (aka 'Hxdef'), so take it from me: it's the absolute worst of all rootkits. If there is an entry in the dictionary for 'most perfect rootkit written by most unscrupulous person', there would be a picture of Hacker Defender next to it. The guy who wrote is actually started selling it at some point, in a variety of versions on his website (which doesn't exist anymore), ranging from standard (just the rootkit) to 'Gold' (the rootkit, with all possible antivirus-evading technology, a very strong encryption on the .exe and a unique signature) for prices ranging from $100 to $300 or so. The website also included a movie clip from a Windows machine being infected with it, followed by a few minutes of using over half a dozen anti-rootkit programs on the same system, none of which found Hacker Defender. It's that hard to remove. But it's not perfect. Especially not the version that had been installed on our servers.

Basically, all you have to do to remove the standard 'free' version of Hxdef, is run SysInternals' Rootkit Revealer to find the name of the Hxdef service, and stop that service. The rootkit will then remove all of its hooks from the system, and uninstall itself completely. So stopping the proper service will not only uncloak any hidden stuff, it will also disable and delete the associated service! It's great - like groping for the lightswitch in a dark room, finding it, flipping it, and then seeing a circle of monsters around you that have been frozen by the light.
After you stop the Hxdef service, you are left with three files: the rootkit itself (something .exe), the kernel-mode driver (something .sys) and the configuration file for the rootkit (something .ini). The only thing that's left to do is open the configuration file in Notepad, delete the obfuscating crap that has been put in there to evade antivirus detection, and you know what the Hxdef rootkit was hiding. And then you can delete that. In my case, this was mostly ftp server files, Registry keys for hidden services (like 'Logical Disk Manager client services', 'Microsoft Netlog' and more important-sounding crap), and the rootkit files itself. Since the four new FXP sites were not much older than one hour, there weren't even any warez on them yet!

So is removing Hacker Defender that easy?
And do you understand why?
Because the guys at SysInternals are the smartest bloody programmers I know.
http://www.sysinternals.com/ - Thank you, Mark Russinovich and Bryce Cogswell!