People have claimed that three is, in fact, the magic number. Some people favor 42, some people e, engineers often 8, maths people e or 3.14 (really just a corruption of 3, right?), 14.7 has some significance in combustion, and so on. Today, two is important.
So way back in 2001, there were two cities on the east coast that learned a lesson about computers. Sometimes, you could have a cluster, an UPS, a big diesel, nice HVAC, and everything. Massive mainframes, and so on. Normally, only one computer actually goes away, or one part of it, or one generator, or one PDU, or one battery, and so on.
Then, unexpectedly, two buildings vanished, and a third was damaged (well, okay, sometimes 3 is important). It turned out your batteries and diesels and HVAC and all that shiny stuff was Just Not Enough.
The Bush administration, in their typical lock-step with Foreign Policy, Public Opinion, and Reality, noticed this. And thus Critical Infrastructure Protection became more than just talk. Some agencies were told that their assets were "national critical infrastructure". The NIAC was born in the same festering pool that DHS was born out of, and absorbed Bush's original council into its own. The original, 75-page CIP document published by the Bush administration became hundreds of pages of guidance at DHS. Actually, that's not quite right. The original document was twelve pages, followed by a 75-page document, followed by the deluge that became the NIAC guidance.
Okay, Okay. Why do I mention this?
Well, all us sysadmins know about disaster recovery ("DR"). I mean, before, we called it "failover." But now you can't just fail over, you need to fail over to a remote site. I recently interviewed with the Census, as I mentioned. Well, the Census has a huge datacenter in Bowie, MD. Well, not AOL huge, but still Pretty Big. They told me this was their only datacenter, and they were quite proud of this. I used the AOL (who is next to IAD, Dulles Airport) lingo and asked, "what happens if a plane falls on the building?" So, just for example (don't you go pointing any planes at the census, people), the Census fails over to a datacenter in Indiana. That's geographically removed. Dig? At ACS, they had a failover site in Columbus, OH (so DC to OH is also good). Some people go as far as to put them in other countries, on the other side of the planet, and so on.
But I've been working with NOAA. You know, the weather people! NOAA recently got told that the GOES and POES satellite services they were providing were national critical infrastructure (I'd use "NCIS", but that's already been taken by something completely unrelated), and that they had better harden them against failure. Well, the ground stuff is largely mainframe-based, and so redundant. But, of course, the question is not now asked, "what happens if we lose a proc or HVAC?" but rather "what happens if the Suitland facility is a smoking hole in the ground?"
That question has been answered. Sort of. Some of you read about Chincoteague Island when you were kids, and may know where Asateague is. Some of you have camped there, despite the grit and mosquitoes. Near that place, the tiny little toehold on Maryland that Virginia has, is Wallops Island. The whole island (a thin, barrier island) and the surrounding peninsula are referred to collectively as "Wallops, VA" although no such area is incorporated.
Anyways, so when they wanted to fail stuff over from the federal center at Suitland, the choice was made to fail over to Wallops. This makes sense, as Wallops has some Big Dishes (including one called "Satan"), as well as very high speed connectivity over land. Now, that is NASA's gear, but I am sure if there's a smoking hole in Suitland, MD, NASA won't mind sharing bandwidth with NOAA. At least we know there's fibre in the ground. Most of what we do with the GOES birds however, is bounced off of, well, other birds.
That's the project I'm working on now. I am building out the stuff that goes to Wallops, and in the process, figuring out the tangled mess of stuff that connects GOES, POES, our consumers, and even (you had to ask) DISA. Priceless moment: on my first or second day, I am asking what our requirements and tolerances are. I am trying to articulate what I mean and I just say "well, let me fall back on military stuff. basically if I want to plug this cable into that wall, I have to ask DISA if it's okay. and they'll hand me a 400 page book that says what I need to do to make sure it is okay, and then they will audit me and make sure. I then plug it in." I am however, working for a civilian organization, and there's no DISA. "So who's DISA?"
There is no DISA. While nobody has much love for DISA (except, well, DISA), I have never missed it more. There is so much griping and buck-passing that I never know where my responsibilities end or what I've really been asked to do. Because what I've been asked to do makes no sense, and if I say what I think I should do, they say, no, do what we asked you to do. So I have to come up with some compromise in the middle and do that. As a consultant, this troubles me, because I'm usually asked to fix problems with specific bounds. Fuzzy logic works for rice cookers, but it's not so great for contracts.
Today I learned (it's been a week) that we have an AIX guy, an HPUX guy, a Network guy, and a Linux guy (me). We all report to the same person, and we don't hate eachother. So when things go tits up with the installation, we should be able to sit down and figure out where the problem is. This pleases me. So things are going a little more smoothly.
That's it. I had expected to update people sooner, but I've had pneumonia, and I'm getting over it. Or rather, am mostly over it. The original culprit, supposedly a 5mm nodule in my right lung, is "suspicious", but we're going to have a look again in 6-12 months to make sure it's either not there, less suspicious, or static. Or something. Less spooky. That second look will be another CT. Also at Virginia Hospital Center, which is fabulous (now; it used to suck and smell like old socks).
No comments:
Post a Comment