Home Graffiti About Sitemap Visual DevWork WebCam Email Brian GPG Key

GRAFFITI -- March 03 thru March 09, 2003 >> Link to the Current Week << Last Week << Mon   Tues   Wed   Thu   Fri   Sat   Sun >> Next Week Search this site : Welcome to Orb Graffiti, a place for me to write daily about life and computers. Contrary to popular belief, the two are not interchangeable.   About eMail - I publish email sometimes. If you send me an email and you want privacy or anonymity, please say so clearly at the beginning of your message..

MONDAY    Tues    Wed    Thu    Fri    Sat    Sun   
March 03, 2003 -    Updates at 1515 EST

Good afternoon. About the delay - our network connection here at home was pretty dicey this AM, and then it was bust-ass to get over to work, then out to Sterling to install a rack-mount Dell box at a colo facility for a customer. We did well, and finished up there with about a total of 1-3/4 hours onsite - all the preparation paid off.

Overall, though, it wasn't a good weekend. I've declined the book that I was talking about, as I couldn't sign the contract in the form that the publisher wanted. It's sort of okay, that is, I could see his point of view, I just couldn't accept some of the terms from my perspective. C'est la vie! Then I've managed to play so hard in the unstable side of the Gentoo package tree that I've broken this box. That is, some important bits just don't work, and I don't know why. I have a couple of different possible solutions before I just blow the whole thing away and start from scratch, so I'd best get busy with those, and with the laundry.

I'll keep you up to date with my progress, such as it is. Thanks for stopping in.

Top  /  Email Brian

Mon    TUESDAY    Wed    Thu    Fri    Sat    Sun   
March 04, 2003 -    Updates at 0645

Good morning. I'll make up for yesterday's very late post, by putting up one a bit early today, alright? First off, there's good news and bad news about the network kinks I've been observing here in the last 24 hours. The good news is that I know what it is. The bad news is a slightly longer story.

First, let me set the stage. When I got home yesterday, the Activity LED on the Motorola SURFboard cable modem was still blinking like a gerbil on methamphetamines. So I started from basics. I powered everything off, and brought the network up one piece at a time, from the cable modem out. Solo, with nothing plugged into the Ethernet connection of the cable modem, all is quiet. The moment a live link is established, the traffic resumes. I presume that it's always on the line, just cut off until there's something to talk to.

I tried just plugging directly into the NetGear firewall/router, and directly into Grendel, the test server. No difference, no joy. I was able to determine that it was not much in the way of outbound traffic, and precious little destined for the machines on public IPs here on this network. So, lots of traffic on subnets that I'm not on. I can tell that because while the packet counts were climbing at a rate of about, oh, 1500 packets a minute, none of them showed up while I was running tcpdump on Grendel.

I called and talked to the tier 1 folks at Comcast. On the very first call, I got a good guy, by the name of Joe. I went through all of the fun with him, and we decided that indeed it was outside my network, but from his side of things, it appeared not to be affecting my connectivity much. But he opened a ticket anyway, so that I could call back in and pick up the process, if more was learned, or the problems continued.

Later yesterday evening, after the dog had been walked, fed and settled down, I sat down and thunk about it for a while. Hmmm. Traffic that I can't see. Why don't I reset the IP address for the NIC on Grendel to 0.0.0.0. Then I can use tcpdump to listen to all of the traffic. So I executed the three following commands:

ifdown eth1

ifconfig eth1 inet 0.0.0.0

tcpdump -i eth1

With an address like 0.0.0.0, I had to explicitly specify that the hardware type for the device was inet, or tcpdump wouldn't see it. Once I had that running, I could see what was happening: It was an ARP request flood, from the Comcast gateway/router boxes (assorted subnets, all n.n.n.1 addresses). Ah-hah! This was something that I could tell Comcast about. So I called back, gave them the case number for the open ticket, and had the guy try to get the well-clued Joe back on the line. Bummer, he was unavailable. So I ran through the schpiel with the new guy, who agreed that this was interesting, and sent me up to talk to Tier 2. Now this is progress... right up to the time where their phone system dropped me. For the second time. The first time was when I was talking with Joe - he called me right back. This new guy didn't, probably because he thought he'd successfully transferred me.

So I call right back, and get guy number three. Now things start to go downhill in a bumpy sort of way. I call, give the case number, and ask to be sent up to Tier 2, where I was destined whent the phone system cut me off. After a few questions, the guy put me on hold ... I assumed in order to transfer the call. After about 5 minutes he came back online and said he'd tried to locate either of the two people I'd talked to before, but he apparently couldn't find them. (It was hard to tell, because he had a rather thick accent, Indian I think.) But he didn't understand why I needed to get to Tier 2. Looking over the notes, he didn't see the reason for escalation. So in short words, I explained again to this guy that I'd discovered what appeared to be a problem that can be traced to their equipment on their network, and can he do anything about that? No, I thought not. Please escalate me up to Tier 2. Yes, thank you, I'll hold.

Then, after about 10 minutes or so stranded in the George Benson circle of hold music hell, I got on the phone with Rick. Hey, that rhymes with thick, doesn't it? How, um, appropriate. I explain to Rick the symptoms on the cable modem, briefly what I tried in diagnostics, and what I learned when I set up the Linux box to monitor all of the traffic on the cable modem. Hoo-ey, I said, You folks have a router that's flooding at least my segment of your network with ARP requests. You appear to have a box that needs some serious resetting.

He says something like "Sure, let me check a few things. Hold on for a few." This is promising - the sort of response I expect from people that can get things done. So I hold, back in the Benson zone, until the conversation resumes, plunging off the precipice above the pointy volcanic rocks. "Um, I talked to my supervisor, and we don't support Linux." Dude, are we having the same conversation? This isn't about my running Linux or Windows or OS-X, this isn't even about the cable modem or my NetGear firewall/router. This is a piece of your equipment, on your network, flooding an entire segment with ARP requests. Somebody needs to reset a box in one of your NOCs... "Oh."

Sigh. Thick Rick says that he's escalating to the local office of Comcast, since he's in Nova Scotia, and can't do anything with the network stuff. Supposedly they're going to call me. Right. Meantime the ARP flood continues. In the first six hours after restarting the NetGear, it logged something like 420,000 packets received on the WAN interface. Sheesh. I can SSH up to Rocket. But I can't get to Grendel, right next to me???

Top  /  Email Brian

Mon    Tues    WEDNESDAY    Thu    Fri    Sat    Sun   
March 05, 2003 -    Updates at 0715

Good morning. I'm composing this on a newer version of Bluefish than I am used to, and doing so on Gryphon, the Sony laptop. That's because I spent so much time playing on the bleeding edge with Gentoo over on Goldfinger that unstable became a state of being, rather than just a goal. Heh. But then, that's what I do with boxes - test stuff out, see what breaks, figure out how to fix it and report my results.

In this case, the behaviour was two-fold. First, VMware wasn't running anymore - I was getting X errors back. Then bits of Gentoo would fail to build when I would apply updates to the system. I first noted some of this stuff back on last Saturday or Sunday. Well, what I've been trying over the last couple of days was to rebuild the whole system from the ground up, in place. That is, I tried to successfully run the command: emerge -e world, which rebuilds all of the packages installed on the system, theoretically in dependency order. I can't tell, because I couldn't get the process to complete.

So, enough! I carefully backed up the important bits of my home off to another system or three, then popped in the Gentoo 1.4rc3 disc that I downloaded last week. I blew away the /boot and / partitions, leaving out /home (the backups are for safety net, not necessity, well...), and I've bootstrapped (last evening) and emerged the base system (overnight), following the directions here. Before I leave for the first client stop of the morning, I'll finish the install, and be set to reboot and start installing X and sundry other bits. By tomorrow morning, I'll be back up to speed. The difference this time? I've left the unstable flag in the configuration file unset. That's right, I'm pulling Goldfinger back from the edge. There's too much going on to down this box every three or four weeks. I'll give that task to Garcia, shortly.

Out here in the world, I've been pretty busy. There's some fun projects in store with one of our main clients, and I've been staying pretty busy up there - I'll be there perhaps 50% time this month, which is a lot, but we're making up for snow days and ear fluid/infections (from which I am still suffering, but enough about that already)and lord knows what all else. We've got hours due them, and they've got projects waiting to drop into our laps, so the timing's right.

I hadn't paid it much attention yet, but there is this Sendmail vulnerability that's been making the news in the last few days. I've been reasonably unconcerned, because on the public systems that I'm responsible for, I run Postfix. But this is a remote-root vulnerability, folks. So if you've got Sendmail running on any of your boxen, get that puppy patched or updated. Many vendors have got their updates available. Props to Roland Dobbins for being first out with the news as far as my inbox was concerned.

On the topic of my fun flood problem with Comcast, there's just one bit of news...

From: Alan Stevens
Subject: Comcast Traffic Flood
Date: Tue, 04 Mar 2003 22:18:51 -0500

Hi Brian,

Thanks for sharing the results of your network sleuthing. I woke up Saturday to a light show on my Comcast cable modem, but I couldn't find the traffic, nothing in the logs etc. I could get out, and had my brother verify he could get in. I was stumped. At least now I feel confident it wasn't an attack against me personally.

Alan--

You're welcome. It's continuing at this moment, and there's certainly been no effort to contact me as promised last night. Why am I not surprised?

Top  /  Email Brian

Mon    Tues    Wed    THURSDAY    Fri    Sat    Sun   
March 06, 2003 -    Updates at 0730

Good morning. Things are beginning to return to normal around here, computing environment-wise. By the time I got home last night, X, Fluxbox, Galeon (& Mozilla), Bluefish, and a few other choice tidbits were installed and ready for me. The only fly in the ointment is that VMware still isn't working. At some point, one or more packages have been upgraded past VMware's ability to install properly. I need to do some explorations of that. But I am very, very pleased that everything else is installing without a hitch. Last night I installed OpenOffice.org, Samba, NTP tools, a few other things. Overnight I started The Gimp and KDE installing, and that's not done yet. I'm not surprised - that's a long build. Let me check here and see how much is left...

goldfinger root # emerge -p kde

These are the packages that I would merge, in order:

Calculating dependencies ...done!
[ebuild  N   ] kde-base/kdeaddons-3.1  
[ebuild  N   ] kde-base/kdeedu-3.1-r1  
[ebuild  N   ] app-arch/rpm-4.0.4-r4  
[ebuild  N   ] kde-base/kdeadmin-3.1  
[ebuild  N   ] kde-base/kde-3.1  

Not much is left, probably an hour or two's worth. Darn it! That's what I get for opening my big mouth. The build just died in the kdeaddons package. I'll have to figure that out tonight. Oh, well.

Yesterday was a good day. I spent the time with a new client out in Annapolis, helping them deal with a virus-plagued server. You see, they didn't have any A/V on the box at all, and it was the file server. So... well, anyway, it's fixed up now. Now I'm running late and I've got to get to Sam's. Have a lovely day.

Top  /  Email Brian

Mon    Tues    Wed    Thu    FRIDAY    Sat    Sun   
March 07, 2003 -    Updates at 0700

Good morning, and woo hoo to you, too. Yeah, almost everything's now working on Goldfinger, the dual Athlon box, again. XMMS isn't working, and I've seen the behaviour before, but I can't put my finger on how I fixed it in the past. Noatun, the KDE media player, is working just fine, though, so I've got tunes. And VMware is up and running. It just took reconfiguring after a reboot. Odd, that, but who am I to complain.

On this day in 322 BC, Aristotle died after complaining of a stomach illness. Also this is the day, in 1530, when Henry VIII had his request for a divorce denied by the Pope. In retrospect, this seems a shortsighted and altogether bad choice. It lead to the second major schism in a double decade, following the nailing of the 95 Theses at Castle Church by Martin Luther back in 1517. In 1857, baseball was determined to be a complete game after 9 innings, rather than 9 runs. And in 1981, on March 7th, the first homicide at Disney land was committed as an 18 year old was stabbed to death. Now if I were James Burke, I could weave all of these events into a compelling and momentarily diverting revision of history. But instead, as we look forward to likely war, I would encourage you to think not just of the future, but also of history. And yes, it's possible that we're doomed. But it's better, I think, to go down fighting. And we don't have to have another Neville Chamberlain, do we?

We make war that we may live in peace.
- Aristotle

Now, back to our regular tech fare... Slashdot informed me of the first 10K RPM IDE drive, in beta and written up over at Storage Review. Hopefully by the time you're reading this the swampage of their server will have abated somewhat. Then you can see what I see, a drive that's all dressed up and finds itself in a biker bar, surrounded by bigger, tougher types... In Linux distribution news, Gentoo's at version 1.4rc3, Red Hat has its Phoebe in Beta 2 (with images dated mid-February at the mirrors I found), Mandrake is showing off 9.1rc2 - it came out just three days ago. Debian continues upon the stately, conservative course that makes it my favorite choice for servers, with 3.0r1 the latest version. SuSE's 8.1 edition deserves another look too, and I'll be getting to that shortly. Patrick's been busy, busy, busy - so Slackware 9.0-rc1 was released on Monday. TurboLinux, under new management (SRA) is up at the plate with their 8.0 release in workstation and server versions. I haven't installed a TurboLinux distro in ages! I need to rectify that, too (if I can, many of their products appear to be Japan/China distribution only). Lycoris and Lindows continue to, um, bore me. Sorry.

Topics that I'm thinking about for the near future include musings on the fate of the Red Hat Network, and why I may consider not re-entering the authoring market again. We'll see what bubbles out of the back of my brain sooner or later. Meantime, enjoy this last day of the work week.

Oh, and a special message for our patriots in the Gulf region: When it comes time, kick ass. I'm proud to be an American, and proud of y'all.

Top  /  Email Brian

Mon    Tues    Wed    Thu    Fri    SATURDAY    Sun   
March 08, 2003 -    Updates at 0900 1330

Well. Rocket is down and I don't know why. Greg and I are working on it. Shit.

1330 - We're back. Whooo, here 's an expurgated version of the post-mortem that I sent to all the people hosted on Rocket.

From: Brian P. Bilbrey Subject: Saturday 3-8-2003 Downtime post-mortem... Date: 08 Mar 2003 13:32:27 -0500 Howdy, Folks. Here's the post-game report on our 6 hours of downtime for Saturday, March 8, 2003. At approximately 0650 EST, access from the Internet to Rocket was lost. I first noticed at about 0850 when trying to check email for the first time (late, but it's Saturday morning, neh?). I called Greg, then we got in touch with RackShack, our dedicated server provider. After some investigation lasting about 45 minutes, it was determined that another server that was operating at [another IP in our subnet] had also taken our IP address. They rebooted the other box, and we were up for about 15 minutes. Then when the other box came back online, we were off again. We called and read them the riot act. Unfortunately (from our perspective), Rackshack policy is to leave the offending box up and notify the end user to mend his configuration. We didn't much care for that, but there was no one there in a position of authority to rescind that policy, on a Saturday morning. They contacted the end user of the other box, and at about 11:40 EST, we were back online for... oh, about 20 minutes. Then we lost connection to Rocket again. So I called Rackshack and raised the red flag. By 1300, they had pulled the offender's box off of the network, and he's not getting it back. We're going to be emailing the powers that be at Rackshack about the downside of their policy for boxes that are denying service to other machine. Consequences: During the span of time that our packets destined for Rocket were going instead to this other machine, our sites were offline. Additionally, and worse, all inbound email was FAILING. That is, it wasn't bounced saying that the server was down. A server was running on that box, and rejecting messages. So any mail that was sent to you between approximately 0700 and 1300 on March 8 is gone. This may also affect mailing lists that you subscribe to. Some lists do auto-unsubscribe if message delivery fails. So pay heed to your incoming mail, and check your subscription status for lists. Thanks for your patience.

Top  /  Email Brian

Mon    Tues    Wed    Thu    Fri    Sat    SUNDAY   
March 09, 2003 -    Updates at 0930

Good morning. Bob may hate RackShack - I can't summon up that level of feeling about yesterday's downtime. Mistakes were made by their staff, and policies were apparently mis-understood or mis-interpreted by lower level people. That lead to the longer extent of the downtime. We currently have involvement from the uppermost levels of the company, as our posts to their public forums yesterday gained notice. We've requested that some of the technical changes recommended to us by Roland Dobbins be implemented to protect us (and others) from the same sort of event in the future. More as we know more.

We had a nice night, at the last, yesterday evening. I stepped away from the computer and didn't come back. We went out to a restaurant called On The Border over off of Highway 301 here in Bowie. It's the first Mexican place out here that is good enough to warrant a second visit. We had been losing hope about finding a good place like

Following supper, we came home and watched all of the second season of Black Adder, featuring Rowan Atkinson, of course, and our favorite: Miranda Richardson as Queenie (Elizabeth I).

My favorite line from Queenie: "It's up to you: either you can shut up, or you can have your head cut off." That's Lord Percy's decision to make, after relating how his uncle's great oak table, and in fact all of his house and possessions, and his uncle himself, all disappeared in the Great Fire of Stepney (or some such). It's a mystery, you know...

Now, on with the Sunday. There's shopping, cleaning, and an assortment of other duties that await. I hope that your day is less stressful than my yesterday, and more relaxing than my today. Have a great one!

Top  /  Email Brian

Last Week << Mon   Tues   Wed   Thu   Fri   Sat   Sun >> Next Week

Visit the rest of the DAYNOTES GANG, a collection of bright minds and sharp wits. Really, I don't know why they tolerate me <grin>. My personal inspiration for these pages is Dr. Jerry Pournelle. I am also indebted to Bob Thompson and Tom Syroid for their patience, guidance and feedback. Of course, I am sustained by and beholden to my lovely wife, Marcia. You can find her online too, at http://www.dutchgirl.net/. Thanks for dropping by.

All Content Copyright © 1999-2003 Brian P. Bilbrey.