What exactly is it that just happened to Blogger?

December 26, 2001 9:43 AM

What exactly is it that just happened to Blogger? I saw hints that there had been some sort of security breach, and now everything is inaccessible. (And the Blogspot server is outright offline as I write this.) There was some sort of password break?
posted by Steven Den Beste to General Weblog-Related at 9:43 AM (40 comments total)

apparently, it's been hacked into. i'm surprised that sort of thing doesn't happen more often with livejournal and blogger; those are big targets.
posted by moz at 9:45 AM on December 26, 2001

A number of users' passwords were changed.
posted by darukaru at 9:55 AM on December 26, 2001

Somehow, all (some?, most?) of the Blogger passwords got changed to "1". Crackers, database glitch, I don't know. All I know is that I'm alarmed by the following:

"Blogger has had a security breach. It appears no permanent damage has been done, but as a precaution, we advise you change your FTP password if you had it stored on Blogger."

I know FTP passwords are sent to and from Blogger via plain text and therefore relatively accessible to anyone packet sniffing, but shouldn't they be encrypted in the database so that crackers can't easily get at them? If Blogger uses SCP for secure transfers in the future, will those be laying about in the DB unencrypted for the taking as well?
posted by jkottke at 10:00 AM on December 26, 2001

jason:

there may be some good reasons not to encrypt the passwords. as far as the hackers are concerned, an encrypted password is an inevitably accessable one; if he wanted he would crack the encryption at some point. in that sense, encryption is only an inconvenience to the hacker.

encrypting and decrypting text, on the other hand, can be an intensive process on a server. for most scenarios i'd say it wouldn't be a terribly big deal, but for all of blogger's server woes and service outages, i would be wary of the increased server load. particularly i would be wary because i believe blogger has quite a few people using it, making the performance drain that much more significant.

maybe the process would not be so bad and i am overstating the increased load brought upon by adding an encryption and decryption process, but it's something that maybe ought to be considered.
posted by moz at 10:11 AM on December 26, 2001

There really is no excuse not to encrypt such small amounts of data as passwords and usernames. If the suggested scenario of slowing down blogger was to happen then an FTP username/password pop-up box should be an option instead of storing it plaintext. Sniffing is one thing but have a DB full of ftp passwords sitting around is just hacker bait.


posted by skallas at 10:25 AM on December 26, 2001

My password on blogger got changed to 1, & I noticed it this morning & changed it back to what it was before. Just because the passwords were changed doesn't necessarily mean they weren't encrypted -- it'd be easy enough to change them all to 1 using password() in SQL without knowing what they were before if you managed to get direct access to the db.
posted by zempf at 10:28 AM on December 26, 2001

as far as the hackers are concerned, an encrypted password is an inevitably accessable one; if he wanted he would crack the encryption at some point. in that sense, encryption is only an inconvenience to the hacker.

Encryption is not an absolute solution, but if proper encryption is used, it's really not feasible for the hacker to crack those passwords....it would literally take him years to crack all of them.

maybe the process would not be so bad and i am overstating the increased load brought upon by adding an encryption and decryption process, but it's something that maybe ought to be considered.

True, but in my mind, there are a few items of information that companies and services should *always* take great care in keeping safe, including passwords, credit card #s, social security #s, etc. It's not too much to ask for companies to keep these items as safe as they possibly can, inconvenience or no.

Does anyone have any experience with de/encrypting passwords and such as part of a system like Blogger or an online shopping cart? Is such a process resource intensive?
posted by jkottke at 10:33 AM on December 26, 2001

Just because the passwords were changed doesn't necessarily mean they weren't encrypted -- it'd be easy enough to change them all to 1 using password() in SQL without knowing what they were before if you managed to get direct access to the db.

I'm not that concerned about my Blogger password...it's my FTP password that I'm concerned about.

Also, if security is done properly (both by Pyra and Microsoft), using password() in SQL shouldn't be an option for anyone if the passwords are encrypted...there shouldn't be an option to circumvent the encryption that easily.
posted by jkottke at 10:39 AM on December 26, 2001

Actually.. upon rethinking my earlier comment & discussing this with a coworker, I don't know if encrypting the FTP passwords is all that feasible in this case. Using password() in SQL is a one-way function -- that is, there's no way to retrieve the password once you've encrypted it. This would be a good way to encrypt the passwords for the login to Blogger itself, but the FTP passwords have to be easily decrypted so that they can be sent to the user's webserver.
posted by zempf at 10:40 AM on December 26, 2001

Zempf: My password on blogger got changed to 1, & I noticed it this morning & changed it back to what it was before.

You'd be well advised to change it to something new.
posted by Steven Den Beste at 10:49 AM on December 26, 2001

You'd be well advised to change it to something new.

Yes, I realize that at this point. This morning I had no idea they'd been hacked & just assumed it was a DB glitch of some sort. Anyway, I can't change my password while the site is down, though that also means nobody else can do anything with it either. As soon as it's back up it will be switched, believe me.
posted by zempf at 10:55 AM on December 26, 2001

Jason: Scoop encrypts passwords on the server, using the simple expedient of unix crypt() (or, technically, a perl implementation of crypt()). It's not resource-intensive at all. Consider, for a comparison, that any reasonable modern server is perfectly capable of producing SSL-encrypted web traffic at nearly the same rate it can serve unencrypted web pages. That's a hell of a lot more data than anyone's password will ever be. The server load in encrypting passwords is utterly not an issue.

The process basically goes like this: When a password is saved, it is crypted with itself as the salt. crypt() uses a two-character salt, which gets prepended to the ciphertext, so we strip that off too (because it's always the first two characters of the plaintext). Then it's saved in the database.

When someone logs in, you take what they entered for a password, crypt it against itself, strip off the first two characters, and then compare that to the ciphertext in the database. If it matches, then they have the right password.

This is not foolproof by any means. Say someone got their hands on the database full of passwords: They could easily run a cracker against it, which would check each password for common, easy-to-guess words. With a large number to try, they don't have to crack one particular one, they just have to crack something. And they also don't have to actually decrypt anything. Many crackers would already have a library of encrypted words, and checking the database for each one would be trivial, and fast. Encrypting them does slow down someone who isn't already an expert though, and helps prevent a small mistake (i.e. leaving a file full of passwords sitting in public by accident) from turning into a giant breach.

The security of all this could be improved somewhat by using a random salt and leaving it prepended to the password. Then, at least, the cracker couldn't use a pre-generated file of words crypted against themselves. They'd have to try each word with the salt in the password itself, which would take somewhat longer. It's not perfect, but it would slow them down, possibly long enough to notify users and change the passwords. It could also be helped by having the salt stored in a root-readable file, and not including it with the password data at all. The drawback to that is if you compromise your server's salt text, any added benefit from it goes out the window. But it's probably less likely than compromising the password file. Also, if you manage to lose the salt, you're utterly screwed, so it would be a good idea to save it somewhere safe offsite.

The reason Scoop does it like it does is because I didn't know what I was doing when I wrote it. For anyone implementing something like this now, I'd recommend at least using random salts and leaving them prepended. Also, using an algorithm like MD5 would be a good idea, as I think crypt() has some weaknesses that can be exploited to recover the original data, if someone is determined to get that one particular password.

posted by rusty at 10:59 AM on December 26, 2001

jkottke: Somehow, all (some?, most?) of the Blogger passwords got changed to "[ ]".

That was a rather immature way of handling the information. There are hundreds of thousands of bloggers out there who's passwords may still be the changed version even when Blogger comes back. They might not have the time, or be logged on to Blogger right at that moment to change it back to something secure. That will leave them vulnerable not to 'crackers,' but to average MeFi readers with malicious intent. [Not everyone here at MeFi are angels.]

I think darukaru handled it well, giving out enough information, but being cautious enough not to tip off the random 'evildoers.' Details of security breaches should be handled as fooljay said: "in a way which I will not detail right now for security reasons."

Anyhow, I noticed this breach yesterday as I was guest blogging on a site. The only filed that was tampered with was the Blogger password field. User's emails and other information, including ftp passwords were left intact. [But given the knowledge of the breach, others could access the user's information to retreve other personal details stored at a Blogger user's profile.] Given that I only logged on to Blogger only four days ago, I was not familier with much of its nuances and 'things.' So that password change did not 'register' with me right away. Were I more aware of how Blogger works, or a regular user, I think I would've noticed the hiccup as a breach and could've emailed Ev.

I think Ev should change all the passwords back to something generated with a random-word-generator. I don't think generating 350,000 random words will be that time consuming. The users can get those passwords emailed back to them. The users can then log on and change the passwords to something they'll remember. Ev should then put up a notice on Blogger along the lines of: "Your password has been changed by the system and emailed to you. Please check your email for your correct Blogger password. Please change your password once you log on to Blogger with this system generated password."
posted by tamim at 11:00 AM on December 26, 2001

jason:

Encryption is not an absolute solution, but if proper encryption is used, it's really not feasible for the hacker to crack those passwords....it would literally take him years to crack all of them.

you're right in that it would take someone years to crack all of the passwords. but i am more concerned about two things immediately:

1. the cracker has a number of people in mind. were their passwords encrypted, it would still be a matter of days (assuming a standard, and that is to say inferior, method of encryption, my estimate might be generous).

2. the cracker may post the password information on a website or may email it to others. the password information can thus be a matter of public access, and it would not matter if the hacker knows who you are or cares; anyone with a beef can sit down and play around with that information.

on a tangent: there is some interesting information on cryptography, if it is a bit dense. the faq explains a good deal about what is involved with cryptography and how encryption can fail.

rusty:


posted by moz at 11:03 AM on December 26, 2001

oops, i was going to say something to rusty but changed my mind. sorry! thought i'd deleted that trailing bit.
posted by moz at 11:05 AM on December 26, 2001

That was a rather immature way of handling the information.

As you said yourself, there are hundreds of thousands of Bloggers out there whose passwords got changed, & they all got changed to the same thing. I don't think posting to MeTa what the passwords were changed to will add any more to the impact than the several hundred thousand people who already know what the password is.
posted by zempf at 11:06 AM on December 26, 2001

zempf: but the FTP passwords have to be easily decrypted so that they can be sent to the user's webserver.

I assume that Blogger doesn't require you to enter your FTP password each time you post? If it did, you could use a one-way function and do the hash-comparison thing above. If the idea is that you leave your FTP password in Blogger, and it just uses it, then it would have to use a two-way algorithm, and store a server-wide key somewhere the webserver had access to. Something like Blowfish would be good for this. See that page for another speed example: they say that with Blowfish you can encrypt 8.3 Mb of data per second on a Pentium 150. Which means that the time to encrypt a password is effectively zero.

Actually, instead of having a server-wide key, it would work pretty well if you used the user's Blogger password as the blowfish key. So, when you want to post something, the process goes like this:

* Enter your blogger password
* Blogger takes the plaintext password you sent, encrypts it with MD5 as per my previous comment.
* Checks it against the Blogger password it has stored.
* If it checks out, then take the plaintext password that the user just sent you, and decrypt the FTP password with it.
* Use the FTP password as usual.

It's still possible to recover FTP passwords from a cracked database, but now it'll take a lot longer, because you have to first extract plaintext from the Blogger password file, then decrypt FTP passwords with that.

Considering how weak FTP is to begin with, this would probably be more than enough safety on Blogger's part. Does it support scp for posting? FTP is lame. :-)
posted by rusty at 11:10 AM on December 26, 2001

I'm not too familiar with Blogger. Did the FTP passwords get changed to 1 or was it the site passwords? Or are these two kept in sync?

If the site passwords changed, and we're talking about a MySQL database, then someone has obviously got a remote connection to the database server and has then been able to execute an UPDATE query. Table access should really be locked down to as few hosts as possible - there's no excuse for not doing this. I have to admit though in my experience, MySQL is one of the more confusing packages to configure from a security standpoint.
posted by dlewis at 11:17 AM on December 26, 2001

Perhaps those with ideas about making Blogger safer should volunteer to help Ev with this effort. I imagine he would appreciate the help.
posted by shelleyp at 11:17 AM on December 26, 2001

That was a rather immature way of handling the information.

Crap, you're right...revealing that sort of info in a public forum like that isn't the smartest thing to do, despite the small amount of damage it could do (as zempf said). Sorry about that.

Like you said, hopefully Ev will change all the passwords to something random before he puts Blogger back up again.

Also, I guess we should keep in mind that Blogger is free and that we really shouldn't expect too much because of that.
posted by jkottke at 11:27 AM on December 26, 2001

dlewis: General Security Issues and the MySQL Access Privilege System

MySQL ships almost secure. As long as you lock off root@*, you're set. From there on, you have to explicitly add users.

shelleyp: I have too many projects already. Hopefully the advice of someone who's already done this (and made some of the mistakes!) will help. What is blogger written in, anyway? ASP? I always had the impression it was a Windows thing.
posted by rusty at 11:34 AM on December 26, 2001

Does Blogger keep any kind of information that would be of use for financial gain to the hacker? If blogger just stores people's thoughts and a password was all that was stolen then this really isn't that big of a deal (or as big as it would be if it were storing credit card numbers).

For the user to protect himself, he should use different passwords for different "important" services. If someone gets your Blogger password, it can't be used to access your e-mail.

That said FTP is a notroiously easy protocol to hack into. Setting up an SSH transfer system with somekind of PGP support on the server end would be sufficient. I've seen servers with this setup being hacked into all the way to root but the valuable files are left untouch.
posted by geoff. at 12:16 PM on December 26, 2001

Does Blogger keep any kind of information that would be of use for financial gain to the hacker? If blogger just stores people's thoughts and a password was all that was stolen then this really isn't that big of a deal (or as big as it would be if it were storing credit card numbers).

But that's not the extent of it. Even if little to no harm can be caused to the people whose passwords were stolen, the bigger problem is the menace to the Internet represented by access to all of those servers sitting all over the place. And it's not all about financial gain, either, don't we all remember denial of service and distributed denial of service attacks?

Further, while I used to be rather laissez-faire about security, in light of recent events and the way in which much of our (U.S.) critical infrastructure is more and more dependent on the telecommunications infrastructure, security breaches like this are a big deal. While I don't like to pick on the little guy (Blogger), the stability of the Net and related infrastructures depends on everyone doing their part for security. Personally, I never used blogger because of the ftp password in the clear business.

Security in software is hard, and it's often about more than just the compromise of an individual's information.
posted by Medley at 12:51 PM on December 26, 2001

According to Netcraft, Blogger is running IIS on Win2k. I wonder if those recent holes in IIS had anything to do with this.

Does anyone know exactly how the server was compromised?
posted by jaysoucy at 12:58 PM on December 26, 2001

jaysoucy: The most recent security problems with Windows were Windows XP, 95, and 98 related -- having to do with Universal Plug n Play. There was a SQL Server security risk posted on December 20th and this could provide the hole for the type of compromised security we're seeing, but am assuming that Ev would apply all patches as needed.

rusty: I hear you -- I'm also overbooked. And I'm not that hot on security other than fairly standard stuff. Still, I admire Ev for what he's provided; It would be nice to find someway to help him.

geoff: If someone grabbed stored FTP passwords, that's serious.

I can't stand hackers who go after "little guys" like Ev and sites like Blogger. Both are providing services to the community. There used to be a code of honor within the hacking (or cracking) community -- they wouldn't have gone after a site like Blogger; now, any dweeb with a keyboard can crack into a system, they're so full of holes.


posted by shelleyp at 1:20 PM on December 26, 2001

While I don't like to pick on the little guy (Blogger), the stability of the Net and related infrastructures depends on everyone doing their part for security.

i think it's unfair blame evan right now. we don't know the circumstances of the intrusion or the extent to which blogger was secured.

everyone doing their part for security will not result in a secure internet; it may result in a slightly more secure environment, but that will simply push hackers to find new directions for their attacks. security does not prevent intrusion: it merely delays that which is inevitable. for how long and how often you delay things is up to you.
posted by moz at 1:23 PM on December 26, 2001

Random comments:

1) Technically MD5 is a hashing algorithm, not an encryption algorithm. Its intended use is to make sure data isn't being tampered with.
2) It's not difficult or processor intensive to encrypt the FTP password using your blogger password as key material. This, of course, assumes your blogger password would make good key material (i.e. it isn't '12345').
3) Yes, FTP transmits passwords in the clear but it's pretty ubiquitous. SSH/SCP isn't available on many hosts.
4) To do a first class job of securing passwords requires that the channel used to log in be encrypted. This generally means buying a certificate that's signed by a well known CA, which is expensive and most likely not a budget priority for a free operation.
5) Even if you secured the logins, FTP is still sending data unencrypted out the back door, so for all that trouble and expense, you're still hosed, especially in light of point #3.
posted by fushbulb at 1:24 PM on December 26, 2001

I'm at Kinkos now where, not only can you plug in your own laptop and get a high-speed Internet connection (I thought you could just use theirs), it's free, except for printing. Much better (and thanks for the tip).

it's pretty amazing to think that possibly two hundred thousand bloggers are anxiously awaiting their service to be brought back to life thanks to a guy who should be on vacation but instead he's plugged into a high speed pipe at a Kinko's in Iowa.

much respect needs to go out to EV.
posted by tsarfan at 1:57 PM on December 26, 2001

i feel bad. on one hand i love blogger to death, ev and the now departed pyra folks have made a beautiful product worth using day in and day out. right now i feel as if i'm without my own left arm - but fortunatley my left arm's just alseep. of course like any sleeping arm i'll get pins and needles once it wakes back up, but i'll use it again.

but being a security minded individual and being able to host my site off my own server i wonder if i wouldn't be better off setting up moveable type and getting it operable with ssl. sure it's more trouble than i want to go thru, and i'll have to consult the other people who post to the group blog but this is the internet and there are a load of kiddies out there that will own something as simple as a night rider fanpage because they're under the assumption that it'll make them cool. *sigh*

well at least we haven't gotten any livejournal zealots saying "you wouldn't have this problem if you used livejournal to manage your site..." i give them another three hours before they find this thread. anyone in on that pool?
posted by boogah at 1:58 PM on December 26, 2001

oops i got that from the temp Blogger page
posted by tsarfan at 1:58 PM on December 26, 2001

Lest it was unclear, I'm not trying to imply that Ev is to blame, or a bad person for any of this. It sucks getting cracked, and it extra-sucks when people crack services that are trying to do a good thing for free. I wish Ev had designed it a bit more securely, but it's easy to make mistakes in security. God knows I've made some doozies. I'm just trying to point out ways to avoid this, and the potential results of such an intrusion. Hopefully it didn't come off as accusatory or negative. I've been on the receiving end of shenanigans too, and it absolutely bites. Hang in there Ev, we're with you.

Also, fushbulb: SSH is available for pretty much every platform. OpenSSH supports nearly every unix-like OS (including OS X), and runs under Cygwin. F-secure and SSH.com both have windows server products. I'm not sure if there's a free or Free implementation native to Windows, but I know if I were running any kind of public server (and I am!) it would only be accessable through ssh (and it is!). The days where we could safely trust the likes of ftp and telnet are long gone, unfortunately.

And yes, MD5 is a hashing algorithm. It, or even better SHA, are great for applications where you never need to decrypt the data-- you just want to compare it to a known value (like in an authentication situation). The way Blogger uses FTP passwords, something like Blowfish, Twofish, or Rijndael would be better.

posted by rusty at 2:25 PM on December 26, 2001

Knight Rider.
posted by sudama at 2:25 PM on December 26, 2001

sudama: how silly of me. correction noted. "knight rider" indeed. at least i didn't bring up "team knight rider". that show was horrid.

on the upside of all this, i just realized that the ftp login i used had it's shell set to "/bin/noshell" and literally had no rights to edit anything other than a couple files in the directory. oh, and the allowed ip's to ftp in were only pointed to blogger's servers.

still kudos to ev busting his ass on getting everything back up.
posted by boogah at 2:37 PM on December 26, 2001

warning, offtopic tangent ahead

OpenSSH supports nearly every unix-like OS (including OS X)

and you'd be mildly surprised how easily you can turn your os x desktop into a shell server. under x.1 it's as quick as editing a configuration file and ssh is on. natively. of course, at that point you'd need some applications to make things a bit more useful seeing as how the personal edition is a bit bare in that department.
posted by boogah at 2:50 PM on December 26, 2001

boogah,

1. Moveable Type, Greymatter, et al aren't all that difficult to set up. If *I* can do it, anyone can do it. :) And that's no dis on Blogger, or on Ev -- Blogger's awesome, I'm just a control freak who wanted to have *all* the toys.

2. The boys from Livejournal have their own server problems on a regular basis. S'far as I know, they haven't been cracked, but I would guess that they have comparable downtime. Because of this -- and because, frankly, it would be really crappy of them to come in and crow over someone else's security breach -- I'd be a bit surprised to see Livejournal guys coming in and tooting their own horns.


posted by metrocake at 3:53 PM on December 26, 2001

Because of this -- and because, frankly, it would be really crappy of them to come in and crow over someone else's security breach -- I'd be a bit surprised to see Livejournal guys coming in and tooting their own horns.

it's sad, but at one point when blogger was having another problem earlier this year several of their users came on blogger's discussion area and said something to the effect of "y'know, there is another way to blog" and went into the "use livejournal" spiel stating that it was better because it was opensource. there's even been trolls [not so much so now] that seemingly sit in the feature request area and when someone asks for a feature to be added to blogger and say "livejournal already has that, use it instead".

nobody likes a zealot. and belive me the "you folks have just as many - if not more issues" reply was kicked in by a user or two. *sigh* time to go back to editing my site by hand for the time being.
posted by boogah at 4:17 PM on December 26, 2001

Ev writes as follows:

Hi, folks.

As most of you who were online today probably know already, Blogger was hacked yesterday (merry Christmas). I took it down this morning and have been investigating, etc, all day. I'm in the process of recovering and putting it back online now, but I haven't necessarily found the hole, so I'm doing so very cautiously. So, the API services have been down and will continue to be down until I'm able to tighten them up more, which probably won't be tonight (though I expect to have the main interface back up tonight).

Sorry,
Ev.
posted by Steven Den Beste at 4:56 PM on December 26, 2001

This, of course, assumes your blogger password would make good key material (i.e. it isn't '12345').

Phew! I use my credit card number with its expiration date.
posted by skallas at 5:02 PM on December 26, 2001

Ev seems to be communicating through here.
posted by Steven Den Beste at 5:10 PM on December 26, 2001

It looks like it is back up.
posted by geoff. at 9:39 PM on December 26, 2001