Udo's Techblog

Comment from John Scalzi, via Wil Wheaton's blog:

indeed, as a late-thirties balding man of modest height, weight and physical attractiveness, I am practically invisible to anyone under the age of 30, and visible to anyone over that age only to the extent that they have to walk around me, or have to have some limited amount of social interaction with me as we stand in a line or some such.

Wow, it's like he's my twin! Fiendish though as I am, I sometimes enjoy it when people are forced to interact with me. Over the years I have come to derive some form of perverse pleasure from observing their barely concealed pain while they have to talk (or worse: listen) to me, all the while wondering when they can get back to, you know, real people. Ah, good times!

Funny thing happened to my blog recently: not only was it being hacked, there was also a DoS attack going on. The attack originated from 65.55.107.111, which made me revise my initial impression that the two events were somehow connected. See, this IP is owned by Microsoft, and the USER_AGENT string identifies the server (which made well over 1.5 Million HTTP requests in a short time frame) as an MSN search bot.

I'm hosted at MediaTemple, using the Grid Service hosting plan. That means, an attack of this sort cannot likely disable the server, since there is a whole grid that can absorb the load. However, this also means that I have to pay not only for bandwidth used but also for cluster resources such as CPU time. So what's a site owner supposed to do in this case? Now that the episode seems to be over, I still don't have a comprehensive answer - but maybe telling the story will help someone somewhere in some way some day. Here's what happened:

The good thing with MediaTemple is that you get almost realtime reports regarding your resource usage. That's how I saw that something was not right: most of my billable resources were being consumed by pages on my blog that couldn't possibly be valid URLs. And there were already hundreds of thousands of such requests occurring targeting these URLs. Well, after downloading the logs for that day it became pretty obvious the originating server was 65.55.107.111, which resolves to

OrgName: Microsoft Corp
OrgID: MSFT
Address: One Microsoft Way
City: Redmond
StateProv: WA
PostalCode: 98052
Country: US

NetRange: 65.52.0.0 - 65.55.255.255
CIDR: 65.52.0.0/14
NetName: MICROSOFT-1BLK
NetHandle: NET-65-52-0-0-1
Parent: NET-65-0-0-0-0
NetType: Direct Assignment
NameServer: NS1.MSFT.NET
NameServer: NS5.MSFT.NET
NameServer: NS2.MSFT.NET
NameServer: NS3.MSFT.NET
NameServer: NS4.MSFT.NET
Comment:
RegDate: 2001-02-14
Updated: 2004-12-09

So far so good. Could be Microsoft, could be a spoofed attack pretending to come from MSN's IP address. The reason why I thought an MSN server was the genuine source were the nature of the URLs used: they looked highly recursive, like someone made a horrible mistake programming their bot. And now it was stuck in an infinite loop querying my site!

First measure: block those URLs

The first and easiest thing to do was go into the Wordpress code and hardwire the response to those URLs. Since they would never occur during normal web browsing anyway, this was an easy choice. I made it so WP would return no data upon such calls, so there was no further HTML that could be parsed for more recursive mayhem to be added to the bot's to-do list. So far, so good. Because now execution was canceled as soon as the URL was called, excess CPU cycles had been cut by one-fifteenth. Not bad! But still, at the rate those requests were made, it was clear that by the end of the billing cycle I would still be well above my alotted limit. However, I felt this was all I could do on the technical side of things.

Second: contacting Microsoft

Allright, what do you do if there is a company out there, hammering your server? You write them a nice notice, informing them that they have a rogue bot, of course. Oh, how naive I was. I thought it was actually possible to contact someone, they would listen, surely discover their mistake and fix it! Ha, maybe they'd even apologize for causing me costs and workload, I thought. The hubris! There are maybe a hand full of email addresses that you can use to contact MS in case of problems. However, half of them return error message right back at you. The other half, I imagine, are just huge data graves where emails go to die. Of course, there was no help coming. It's just not possible to reach someone who cares. I was at least hoping to get the infamous Condescending Automated Response, but apparently my problem wasn't even worthy of that.

Third: what about MediatTemple?

Well, if MSN wasn't going to do anything at all, maybe I could turn to my provider for help. Of course, the thing you have to keep in mind is, that MT is profiting from such things happening to their customers. Nevertheless, I wrote a diligent message detailing the problem to MediaTemple's support. In the beginning I was even hopeful, because some first-responder sent me a mail right back explaining that my request had been escalated to a sysadmin. However, this state of hopefulness quickly faded away, when the sysadmin finally gave me my Condescending Automated Response. It explained things along the lines of "if you don't want bots spidering your site, you can exclude them by editing the robots.txt". Bloody brilliant, like I hadn't already forbidden MSN to crawl my site. Like these million requests were part of a normal indexing run, sure!

Upon explaining these things again to MT support, I got a semi-useful message back: there just wasn't anything they could do, period. Blocking this IP would mean other customers' sites couldn't be indexed by MSN. And I could always use an .htaccess rule to further cut down on CPU cycles. But otherwise, that's just the risk of running a site.

And then, everything went quiet

I'm not really sure what happened next. The attack suddenly stopped. Maybe MediaTemple had suddenly recognized the fact that this wasn't a normal bot running its index and blocked it, though I doubt it. Maybe MSN finally rebooted their server, though I'm fairly sure they didn't even get the message that anything was wrong. Maybe it will even happen again come next indexing run. Who knows? It's not like you get any curtesy information out of any of those companies. And if it happens again? Well, I'll just have to pay up then, won't I?

What little can be done

I've excluded MSN bots from spidering the site at several levels. It's the least I could do. And it's also not like there is any meaningful traffic coming through MSN search there, too. I would encourage other people to do this as well, because if an MSN bot goes rogue, there is absolutely nothing you can do against that as a lowly site owner. The least you can do to protect yourself is to pull your stuff from Microsoft-related indexes.

Udo's Techblog

All things considered, the attack did turn out to be not so bad, but I certainly didn't enjoy the hacking and the posting of spam in my name. These recent events have added to much of the negativity that is currently in sum making up my life. Things have been going downhill for a long time now, I just don't know where the bottom is, yet. I guess this is also the reason for the shocking lack of original content recently. I haven't decided what to do with the blog, yet. If nothing else, it certainly has allowed unpleasant people in my life another angle of attack. The blog comes up as the first result when someone googles my name and Analytics is telling me lots of people have been doing exactly that, recently. That's nice as long as everything is going great. But if you're bankrupt and overall not doing so well, it becomes another thing entirely.

Looks like my blog is being hacked, I apologize for the stuff some of you might have received in their RSS feeds. I think I know who did this, if it was indeed aimed at me personally (which I do believe, maybe because humans are good at pattern recognition even given random data).


Apparently, there is also a DoS/overload attack going on, which seems to confirm the initial suspicion.

This is way cool!



Why wasn't Firefox there, first?

Help me test a little Facebook app: FQL Explorer. It's very straightforward, does exactly what you'd expect it to - it's a tool that lets you access your data via the FQL API. Sign up and help me fix the bugs, willya? :-P



Update: fixed the link - but for some reason there's a cataclysmic bug when using IE, trying to figure it out...

Hory made a PHP eval() class that allows for safer script execution in environments where you want users to enjoy some level of customization but can't give them full access under the PHP "all or nothing" model. Good job and nice website, too!

In the last years, we have gotten used to web applications handling almost every aspect of our life. Each day, we create a huge amount of content: emails, instant messages, twittergrams, Facebook interactions, chats, blog posts and comments. In a very real sense, this data is our life, it's what we've been spending all this time on. Yet, most online apps don't actually let us own our content. They store it "for" us, making sure we don't get to access or use it in any way that exceeds the bare minimum functions needed to prevent migration to a competitor's platform.

Robert ran some data mining script on his Facebook account, and - very predictably - he was soon deported from Facebookland. Of course, they let him back in; everything else would have been just a disastrous PR move reminiscent of what happened to Second Life when they kicked him out. Seen any SL buzz lately? Yeah, me neither. Not that I'm complaining, it's an awful piece of software running a bleak and senseless virtual world. But none of that is the point.

The actual point is, when they banned Robert, all his content disappeared from Facebook. Think about that for a while. Our digital lives are not only threatened by bankruptcy and datacenter-scale disasters. In a setting where you're not allowed to take your own data anywhere, you're completely at the mercy of whoever gets to run those services. Somehow, we have gotten to a point where instead of owning your personal content, you just license it under whatever conditions they decide on, RIAA style.

Let's take a look into the future, where we'll spend even more time accumulating even more stuff on even fewer sites. Imagine using a social service for 20 years! Every byte of data is effort that goes into building your digital identity. Every second spent building that identity increases the mental energy needed to switch to another service and start from scratch. And then, one day, for whatever reason, all the stuff you've been doing just vanishes. It's as if you never existed.

This is a huge deal. Maybe the time isn't right for this idea to enter mainstream yet. But it will be. Someone has to start thinking about this. Sooner or later online service companies need to let us take control of our data. Otherwise, why even bother creating it? Some day, some important blogger will wake up and raise the issue and everyone will act like it's a totally novel concept. People will get ridiculous amounts VC money to solve this problem. Why doesn't Facebook preemptively solve this? Especially since it's almost possible to do today with the existing API?

Oh yes, before I forget: OpenSocial doesn't even begin to address this problem.

People absolutely love their content-neutral blurry one-minute clips. The king of short video is of course still YouTube. Ah, the joy of clicking through a 15-part series of meaningless 50-second webisodes!

Not me though. Google Video may live on borrowed time, but I totally dig both the ">20 minutes" search option and the fact that there are lots of very cool long videos that are hosted on the doomed service itself. I guess that means I'm finally getting old and have completely lost touch with trends and reality. Or maybe I'm the last person on earth who hasn't either succumbed to catastrophic ADD or lost all interest in content that can't be boiled down to a 50 second tabloid soundbite?

Via Responsible Nanotechnology: CRN is running an article called The Fermi Death Sentence, which infers that there must be some barrier preventing advanced civilizations from forming.

The Fermi Paradox, of course, concerns itself with the mathematical probability of life and finds that, even when using very conservative numbers, the universe is teeming with it. More importantly though, Fermi asks the question: if the universe is full of life, why isn't the sky full of flying saucers? Why isn't our radio spectrum totally polluted with extrasolar transmissions? Why are there no artifacts on earth suggesting somebody was here before us?

CRN and many others conclude from this that all intelligent life is doomed and thus a colonialization or even exploration of the galaxy a scientific impossibility. Since CRN concerns itself with nanotechnology, they think about how maybe the development of nanotech could be the final nail in the coffin of all intelligent beings.

Using statistics to make "definite" assumptions about singular instances (a.k.a. humanity's existence in this case), Mike Treder allows for 3, and only three, possible scenarios:

We are the first intelligent beings capable of expanding into the cosmos and making our presence known. There have been no others.

There have been others before us, but all of them, without exception, have chosen -- or somehow been forced -- to expand in such a way that they are presently undetectable by our most sophisticated instruments.

There have been others, but all of them, without exception, have run into a cosmic roadblock that either destroys them or prevents their expansion beyond a small radius.

If nothing else, this example beautifully illustrates the dangers of applying statistics to single events. I'm sure, everyone of us can think of dozens of reasons for the status quo and none of them requires the scientifically improbable conclusion that every single intelligent race wiped itself out with nanotechnology (or bio warfare, or climate change, or whatever).

Just a few of them might include or be any combination of:

• We might be living in the galactic equivalent of a national park.


• Evidence is there, but it's being covered up.


• Our part of space has not been colonized yet or it was "recently" sterilized by a cosmic event.


• Our patch of space was colonized, but so long ago that the evidence is gone now.


• Only primitive civilizations use radio waves (because they're very impractical at long distances).


• The Singularity already happened and/or we're living in a simulation.


• The advancement of virtual reality makes space exporation unattractive.

Diclaimer: These are in no particular order, by no means complete, I'm not believing in any particular one of them. These are just more theoretical possibilities.

I am just trying to show that the Fermi Death Sentence argument attempts use misguidedly applied mathematics to elevate a point that would otherwise be merely one more theoretical possibility among many. As they say in reality TV: you decide.

I can't decide whether this AP story on CNN is sponsored by some airline association, or rather directly by filtering software companies. Claiming to act in the best interests of their customers, airlines will eventually install WLAN on their planes (hefty surcharges apply, no doubt) but won't give them unhindered access because that would supposedly aggravate luddite passengers.

See, if you had free (as in not filtered) internet on planes, people would be "yapping endlessly" on VOIP, which is so much more disruptive than today's non-tech loudmouths who believe their neighbors really want to hear awesome stories about their personal achievements to no end, or the constantly crying and bickering children of the jumbo-sized family in row 15. Of course the censoring of VOIP connections would have absolutely nothing to do with the horrendously expensive cell phone and in-cabin phone access the airlines will be offering soon.

It's not really like you can vote with your feet, either, mainly because they all provide equally sucky service and equally horrendous prices at an equally low service level.