Web outage blamed on zombies

Grok Headline matches for Web outage blamed on zombies

Squirrel Blamed for Outage, Traffic Jam
(AP)

Squirrel Blamed for Outage, Traffic Jam
(AP) 08/27/2004 01:59 PM
AP - A hungry squirrel has been blamed for a power outage that snarled rush-hour traffic in this city north of Portland, Ore.

All You Zombies

All You Zombies 07/13/2004 05:32 AM

Lastest reports suggest that most of the spam and viruses out there come from "Zombie" machines -- machines that have been infected with viruses, spyware and other malware -- often unknown to the owner -- that sit and pump out their nasty effluent for the highest bidder.

Is your machine among the infected? Or likely to become so? Who among your acquaintences badly needs some clues and advice?

Here's a simple guide to how to protect yourself, your computer, and your data.

Find out for sure if you're part of the problem -- or how to insure you don't become a part of it.

Zombies on the web

Zombies on the web 06/22/2005 02:58 AM
There are actually three different kinds of zombies. All of them are like humans in some ways, and all of them are lacking something crucial (something different in each case).

Hollywood zombies. These are found in zombie B-movies...
Haitian zombies. These are found in the voodoo (or vodou) tradition in Haiti...
Philosophical zombies. These are found in philosophical articles on consciousness...

Zombies on the web

How Much For A Network Of Zombies?

How Much For A Network Of Zombies? 09/08/2004 05:26 AM
USA Today is running a series of articles about various "bad things" happening on the internet, from spyware to phishing attacks to zombies, and one of the sidebar items looks more closely at the zombie problem, suggesting a network of 20,000 zombie machines can be leased for $2,000 to $3,000. Of course, they don't say what the terms of the lease are. Is it for a day? A month? A certain number of emails or DDoS attacks sent? One interesting (and slightly scary) note, however, is that some zombie networks now appear to be using a rotational system to avoid detection. Rather than have all 20,000 zombie machines blasting spam at once, it rotates. This way, not only is it tougher to track down which machines are actually zombified, the owners of those machines are much less likely to notice there's any kind of a problem with their machine -- and, thus, much less likely to clean the machine to free it from its zombie masters.

BBC Says There are One Million Zombies
on Web

BBC Says There are One Million Zombies
on Web 03/19/2005 02:44 AM
"More than one million computers on the net have been hijacked to attack websites and pump out spam and viruses."

Una de (ordenadores) zombies

Una de (ordenadores) zombies 07/28/2004 05:53 AM

Zombies all, and glad of it

Zombies all, and glad of it 08/17/2004 07:40 AM
Chicago Tribune Aug 17 2004 12:12PM GMT

What are zombies and trolls?

What are zombies and trolls? 02/05/2005 09:26 PM
Dave, I’m surrounded by strange jargon and I feel like an extra in a B-grade horror movie. One mailing list I’m on has “trolls”, and the latest virus can turn my computer into a “zombie”, and my ISP keeps talking about “demons”! Egads! Do I need some sort of exorcist to use my computer now??…

Direct and Related Links for 'What are zombies and trolls?'

Zombies for sale

Zombies for sale 09/10/2004 03:04 PM
Compromised PCs are valuable these days, at least to spammers. Some are willing to pay over US$2,000 for a network of zombie PCs.

Plight of the zombies

Plight of the zombies 02/17/2004 09:56 AM
USA Today Feb 17 2004 1:58PM GMT

Zombies at Starbucks

Zombies at Starbucks 04/28/2004 12:14 AM

This particularly ghoulish scene from the movie Security Scenarios from Hell has three actors: WiFi, Zombies, and Spyware.

Perils of WiFi are well known and well publicized (i.e. Wireles s Networks are in Big Trouble, a classic Wired from 2001).  If you are a geek, here is a more technical version of the same from Secur ity-Forums.com.  While the perils were preached before their subjects have, WiFi is now commonly available which means those perils are now common as well.

Zombies are also well publicized.  Typically, they are poorly protected servers or home PCs with broadbands which are hijacked by hackers, supposedly even traded like Yu-Ki-Oh cards in the hacker community, and used to increase scalability to their attacks and to reduce likelyness of capture.

Spyware is software running on desktops that monitors user activities and report back to it's master.  Most of them are just privacy violators, some are used for more sinister purpose and are called trojans.  Earthlink recently claimed that PCs had, on the average, 28 spyware installed.  While I think the claim is over-hype d to fit their agenda, spyware is nonetheless common place and it's not difficult to place one on anyone's compure.  If your PC is more than six months old, chances are that there were plenty of opportunities for hackers to seed it with spyware.

So here is the scene: imagine a new class of spyware that monitors wireless network packets using code from these open source wiretapping tools.  AirSnort and one of the ARP poisoning packages should be enough.  Now imagine this spyware being delivered to laptops with WiFi cards that supports features AirSnort needs.  The laptop just became a new kind of zombie, which I call wireless zombie, that only wakes up when the WiFi card is used.

All that is missing from the scene is the stage: a WiFi hotspot like Starbucks.  The laptop owner sits in a corner and access the Net through the WiFi, it could even be someone like me writing this very blog post.  The spyware wakes up and starts monitoring the wireless traffic looking for passwords and credit card numbers.  If very strong encryption is used, wireless zombies can form a global grid and split up the work of cracking encryption keys.  Once a month, the zombies reports back to their master via USENET posts.

This Zombies at Starbucks scenario is particularly nasty because the potential number of compromises is just staggering.  Maybe the FCC will have to dictate higher level of standards and send out a warning that helps WiFi users detect wireless zombies by the unusual fan activities triggered by the zombie grid working overtime.

Spammers: Looking for a few good zombies

Spammers: Looking for a few good zombies 05/21/2004 06:59 PM
ZDNet May 21 2004 10:31PM GMT

Other News: Windows Zombies

Other News: Windows Zombies 05/27/2004 09:13 AM
PC zombies created by Windows worms aren't just a "theoretical" problem - they're actually delivering massive loads of spam for their remote masters right now.

Invasion Of The iPod Zombies

Invasion Of The iPod Zombies 02/14/2004 09:28 AM

NY Times On Spam Zombies

NY Times On Spam Zombies 06/24/2005 06:17 PM

Brightmail tackles zombies

Brightmail tackles zombies 07/01/2004 10:05 AM
Antispam vendor releases new version of software to try to keep zombie PCs at bay.

iPod Zombies In New Yrok -- It's True

iPod Zombies In New Yrok -- It's True 03/06/2004 02:06 AM
The iPod is allowing Apple to reach people that may not have given the company a second look a couple of years ago. By Jim Dalrymple (Macworld via MyAppleMenu)

Brightmail's new software tackles
zombies

Brightmail's new software tackles
zombies 07/01/2004 10:35 AM
ZDNet UK Jul 1 2004 1:49PM GMT

Attack of Comcast's Internet zombies

Attack of Comcast's Internet zombies 05/24/2004 06:03 AM

Comcast Thinks About Stopping Zombies

Comcast Thinks About Stopping Zombies 05/24/2004 08:21 PM

Week in review: Attack of the zombies

Week in review: Attack of the zombies 06/18/2004 02:13 PM
A denial-of-service outage hits Akamai and some of its top customers, and smart phones get their first-ever call from a worm.

Zombies may spoil Microsoft's spam plan

Zombies may spoil Microsoft's spam plan 06/04/2004 12:27 PM
ZDNet Jun 4 2004 3:27PM GMT

Government Machines Turned Into Spam
Zombies

Government Machines Turned Into Spam
Zombies 08/31/2004 10:06 AM

Hackers hijack federal computers: Not even the federal government is immune. This must be the zombie controllers Holy Grail, 'cause I imagine the feds have some decidedly fast machines lying around.

Hundreds of powerful computers at the Defense Department and U.S. Senate were hijacked by hackers who used them to send spam e-mail, federal authorities say.

More on zombie networks here and here.

Click here to comment on this entry

"When the zombies take over, how long
till the electricity fails?"

"When the zombies take over, how long
till the electricity fails?" 06/18/2004 11:20 AM

The Inquirer: 'Apple zombies attack
RealNetworks'

The Inquirer: 'Apple zombies attack
RealNetworks' 08/19/2004 05:48 AM
Nick Farrell, writing for The Inquirer, has an "interesting" take on Apple fans and the recent Real/Apple petition snafu...

ISPs Teaming Up To Standardize Plans To
Stop Zombies

ISPs Teaming Up To Standardize Plans To
Stop Zombies 06/22/2004 02:02 PM
Six large ISPs are now teaming up to take on the issue of zombie machines spewing spam. AOL, Yahoo, Earthlink, Comcast, Microsoft and BT are all working together to come up with a standard way to fight spam zombies, but some of their plans may upset certain users. While it's good to see them (finally) taking this issue seriously, one of the proposed solutions is to limit how many emails a customer can send out per hour or per day. While this is unlikely to be a problem for most users, it could cause problems for people who legitimately need to send out a large number of messages. Besides, as some have been suggesting, all this really means is that spammers will have incentive to get their software on even more machines in order to keep on spamming.

Outage

Outage 12/29/2003 10:28 PM
Here's what happened earlier today, according to our web hosting provider:

Today there was a outage for about one hour. The word from the Datacenter is that there was a faulty fiber in their connection which needed to be replaced without warning. They are finishing up the repairs and you may experience intermittent latency as this completes. We apologize for the probelms this has caused as it was out of our control.

Straight Dope Staff Report: When the
zombies take over, how long till the
electricityfails?

Straight Dope Staff Report: When the
zombies take over, how long till the
electricityfails? 06/16/2004 01:06 PM
Straight Dope Staff Report: When the zombies take over, how long till the electricity fails? .. Important hypothetical question .. how long would power last .. Now you know .. StraightDope .. zombies

straightdope.com/mailbag/mzombiepower.html
track this site | 7 links

MIT power outage

MIT power outage 05/04/2004 03:12 PM
real reporting, complete with charts!

Host Outage

Host Outage 07/13/2004 03:22 PM

Our web host had emergency maintenance last night that lasted nearly 12 hours. They took the site down and put up a older drive which had dated news. We apologize for the confusion. Nothing like messing up posting of the daily articles.

Power Outage

Power Outage 12/14/2002 07:13 PM
It's raining and blowing like mad in the Bay Area today. I just had a 3.5 hour power outage. Yuck. Oh, well. It could be worse. At least it doesn't snow here....

Yesterday's outage

Yesterday's outage 04/14/2004 10:27 AM
My host's server died yesterday and didn't come back until this morning. Sorry for the interruption. I don't know yet what will happen to email you sent me yesterday. Apparently it's all going to arrive soon. Sorry for the inconvenience....

LiveJournal Outage

LiveJournal Outage 02/01/2005 10:05 PM
Due to a power failure affecting all of Internap's data center, LiveJournal is currently completely inaccessible, and we're waiting on...

Outage seen at Hotmail

Outage seen at Hotmail 05/07/2004 01:36 PM
CNET May 7 2004 5:13PM GMT

Planned outage

Planned outage 03/25/2005 09:07 PM
NewsGator Online will be down for approximately 8 hours starting Saturday, March 26 at 9:00am MST.  We will be implementing a major system upgrade to enhance our service...

the story of an outage

the story of an outage 01/16/2004 11:27 AM

a tale of mistakes, backups, recovery (by a hair), and why permalinks are not so permanent after all

out·age (ou?tij) noun

1. A quantity or portion of something lacking after delivery or storage.
2. A temporary suspension of operation, especially of electric power.

   When I woke up yesterday after a brief sleep I started to log back in to different services and as I'm seeing something's funny with my server, Jim over at #mobitopia asks "is your site down?".

   Damn.

   As I checked what was happening, I could see that all sorts of things were not working on the server. I was starting to fear the worst ("the worst" in abstract, nothing specific) when I remembered that I had seen similar symptoms a couple of months ago, and back then it had been a disk space problem. I run "df" and sure enough, the mountpoint where a bunch of data related to the services (including logs) is stored was full (since November the number of pageviews a month has increased to over 200,000, which creates pretty big logfiles). As the last time, the logs were the culprits. Still half-asleep, I start to compress, move things around and delete files, when suddenly after a delete I stop cold: "No such file or directory".

   What? But I had just seen that file...

   I look up the console history and four rm commands had failed similarly.

   Uh-oh.

   I run "pwd". Look at the result. "That's not right...". I was not where I thought I was.

   At that point, I woke up completely. Nothing like adrenaline for shaking off sleepiness.

   I look through the command history. At some point in my switching back and forth from one directory to another, I mistyped a "cd -" command and it all went downhill from there. Adding to the confusion was the fact that I used keep parallel structures of the same data on different partitions, "just in case". I stopped doing that once I got DSL back in May last year, opting instead to download stuff to my home machine, but the old structure, with old data, remained. And, even more, my bash configuration for root doesn't display the current directory (the first thing I did after I realized that was add $PWD to the prompt, but of course by then it was too late).

   I had just wiped out the movable type DB, the MT binaries (actually, all the CGI scripts), the archives, and a bunch of other stuff in my home directory.

   I took a deep breath and finished creating space, and moved on.

   First thing I did was restart the services, now that disk space wasn't longer an issue. Then I reinstalled the binaries that I had just wiped out, which I always keep in a separate directory with some quick instructions on how to install them. That turned out to be a lifesaver, one of the many in this little story.

   After that I put up a simple page that explaining the situation (he re's a copy for... err... "historical reference"), plus a hand-written feed and worked on the problem in breaks between work.

   Then I realized that all the links that were coming in from the outside (through other weblogs, google, etc) were getting a 404. So as a temporary measure I redirected the archive traffic to the main page through a mod_rewrite clause:

   RewriteRule /d2r/archives/(.*) /d2r/ [R=307]

   That would return a temporary redirect (code 307) while I got things fixed (one fire out! 10 to go).

   So what next? The data of course. When I came back to Ireland at the beginning of January I started doing backups of different things (a "new year, new backups" sort of thing), and I backed up all the server data directories on Thursday, and then on Saturday I did what I thought was a backup of my weblog data, through MovableType's "Export" feature. As things turned out, the latter proved useless, and it was the "binary" backup that saved the day.

   Why? Well, as I started looking at things, I went to MT's "import" command in cavalier fashion and was about to start when the word "permalink" popped up in my head. Then it grew to a question: "What about the permalinks?".

   The question was valid because my permalinks are directly based on the MT entry ids. Therefore, if an import changed the entry IDs, it would also break all the permalinks. I started cursing for not switching over to using entry-based strings for permalinks, but that didn't help. So I did a little digging and I realized that I was right. MT assigns entry IDs on a system-wide basis. So if you have multiple weblogs on the same DB (which I have, some of them private, some for testing, etc) OR if you have to recover the data from an export (which I had to do) you're out of luck. More likely than not, the permalinks will not work anymore. The exported file did not include IDs. Re-importing would generate the IDs again. Different IDs. Different links. Result: broken links all over the place, both within the weblog and from external sources.

   This is clearly an issue with the MT database design, which doesn't seem too well adapted to the idea of recovery. To be fair, however, I am not sure how other blogging software deals with this problem, if at all. I think this is one big hole in the weblog infrastructure that we haven't yet completely figured out, both for recovery and for transitions between blog software (As Don noted recently).

   This is when I started thinking that things would have been much easier if I had written my own weblog software. :) That thought would return a few times over the next 24 hours, but luckily I was busy enough with other things not to indulge in it too much.

   After looking online and finding nothing on the topic, I came to the conclusion that my only chance was to do a direct restore of the "binary" copy (that is, replacing the clean database with the backup directly) I had from last Thursday. I did the upload, put everything in place, and things seemed to go well, I could log in to MT and the entries up to that point where right where they had to be. So far so good. I was going to do a rebuild and I thought that maybe now was a good time to close off all comment threads in all entries (to avoid ever-increasing comment spam) and I spent some time trying to figure out how to use the various MT tools to close comments on old entries. However, they all seem to be ready for MySQL rather than BerkeleyDB. It wasn't a hard decision to set it aside and move on.

   So I started a full rebuild. The first 40 entries went along fine, albeit slowly. Then nothing happened. Then, failure. I thought for a moment that, for some strange reason, the redirect I had set up yesterday was causing the problem, so I removed it, restarted the server, and Tried again. Failed again. No apparent reason.

   I got angry for a second but then I remembered that the "binary" backup was of everything, including the published HTML files. Aha! I uploaded those,crossed my fingers, and did a rebuild only of the index files, and everything was up again. Actually, this was important for another reason, since the uploaded images that are linked from the entries end up by default in the archives directory, you need a backup of that or the images (and whatever else you upload into MT) will be gone if you lose the site.

   So the solution up until this point had been a lot simpler than I thought at the beginning.

   But wait! All the entries after last Thursday were missing, and I didn't have a backup for those. That was when RSS came to the rescue in three different forms: 1) I download my own feeds into my aggregator, so there I had a copy up to a point. 2) Some kind souls, along with their condolences for the problem, sent along their own copy of the latest entries (Thanks!!--and Thanks to those who sent good wishes as well). 3) Search engines, (Feedster was the most up to date--btw, it was Matt that suggested yesterday, also on #mobitopia, that I check out Feedster as a source of information, a great idea that really applies to many search engines if their database is properly updated), had cached copies that I could use to check dates and content. So armed with all that information I set out to recreate the missing entries.

   Here the problem of the permalinks surfaced again. I had to be careful on the sequencing, or the IDs wouldn't match. So I re-created empty entries, one-by-one, to maintain the sequencing (leaving them unpublished), actually posted a couple of updates< /a> of what was going on, and then I published the recovered entries as I entered the content and set the right dates.

   So. All things are restored now (except for the comments from the last week, which are truly lost--this makes me think that setting up comment feeds would be a good idea. However, that doesn't address how would I recreate the comments given what happened. Would I post them myself under the submitter's name? That doesn't seem right at all. Another problem with no obvious solution given the combination of export/ID issues with MT).

   What's strange is that there's been slight a breakdown in continuity now, because I did "post" some updates to that temporary index file, but it couldn't be part of the regular blogflow. Hopefully this entry fixes that to the extent possible.

   Okay, lessons learned?

   1. Backups do work. :) I am going to do another full backup today, and I'll try to set up something automated to that effect. (Yes, I know I should have done it before, but as usual there are no simple solutions, and then you leave it for the next day... and the next...). Plus, backups for MT installations, should always be both of the DB and the published data, to make recovery quick. (I have about 1500 entries, which amount to something like 20MB of generated HTML--additionally, the images are posted directly on the archives directory, so if you're not backing that up, you've lost them).
      
   2. For MovableType, the export feature is not so great as far as backups are concerned. The single-ID-per-database problem is a big one IMO, and I don't think MT is alone in this. We need to start looking at recovery and transition in a big way if weblogs are going to hit the mainstream (and we want permalinks to be really permanent)
      
   3. Solutions are often simpler than you think, if you have the right data. Having a full backup makes recovery in this case easy and fast.
      
   4. This stuff is still too hard. What would a less technically-oriented user do in this situation? Granted, it was my knowledge (since I was fixing stuff directly on the server) that actually created the problem in the first place, but there are lots of ways in which the same result could have been "achieved", starting from simple admin screwups, hardware failures, etc.
   Overall, this has been a wake-up call in more than one sense, and it has set off a number of ideas and questions in my head. How to solve these problems? I'll have to think about it more.

   Anyway. Back to work now, one less thing on my mind.

   Where was I?

   
   

   Impact of Outage Minimal

   
   Impact of Outage Minimal 06/17/2004 04:38 PM
   “Akamai Technologies (akamai.com) said yesterday that the “sophisticated, large-scale” denial of service attack it suffered earlier this week that impacted its naming functionality had only a minimal impact on its customers.”
   

   Other News: Akamai Outage

   
   Other News: Akamai Outage 06/16/2004 10:22 AM
   Yesterday's blackout of Apple's and other major web sites is was apparently caused by a mysterious Internet attack on Akamai name servers.
   

   Comcast's Offer for Outage: $1.43 a Day

   
   Comcast's Offer for Outage: $1.43 a Day 04/15/2005 12:36 PM
   After experiencing three nights of network outages in less than a week, BetaNews has learned that in at least one case in southeast Michigan, a customer received a credit of $2.86 on their bill to compensate for the two days of service he complained about.
   Grok Description matches for Web outage blamed on zombies
   GrokA matches for Web outage blamed on zombies
   

   Web outage blamed on zombies

The following phrases have been identified by the grok system as matching this entry: