15 June, 2009
13 June, 2009
Not much to say
... LOVELY LOVELY, moaned the Weaver, THE SNIPSNAP OF SUPPLICATION AND YET THOUGH THEY SMOOTH EDGES AND ROUGH FBRES WTH COLD NOISE AN EXPLOSION IN REVERSE A FUNNELLING IN A FOCUS I MUST TURN MAKE PATTERNS HERE WITH AMATEURS UNKNOWING ARTISTS TO UNPICK THE CATASTROPHIC TEARNG THERE IS BRUTE ASYMMETRY IN THE BLUE VISAGES THATWILL NOT DO IT CANNOT BE THAT THE RIPPED UP WEB IS DARNED WITHOUT PATTERNS AND IN THE MINDS OF THESE DESPERATE AND GUILTY AND BEREFT ARE EXQUISITE TAPESTRIES OF DESIRE THE DAPPLED GANG PLAIT YEARNINGS FOR FRIENDS FEATHERS SCIENCE JUSTICE GOLD...
Somehow I feel this characterizes things of late. And I'm not even a real big fan of MÃeville.
09 June, 2009
failure
I have not written any fiction in so long I am truly embarrassed with myself. I have so many 1/3 novels, short stories waiting for my editor, ideas needing to be made into outlines, and I get more every day. Yet work, it pushes so hard to fill my free time and my "9-5" that anything else suffers. I can't just write on weekends, like I do with the bikes. I can't pick up Sunday where I left off the previous Saturday. Writing doesn't work like that.
I feel like such a failure. Writing is so important to me, and I've just let it become this big mark of "yeah, you fucking failed" in my life. I wish I could express the things I do with writing on a motorcycle.
It can't be over. I refuse to stop writing. I mean, I've stopped, but I will not lose this battle. I just don't see a way forward. Self-loathing is so ugly, and so pathetic.
I feel like such a failure. Writing is so important to me, and I've just let it become this big mark of "yeah, you fucking failed" in my life. I wish I could express the things I do with writing on a motorcycle.
It can't be over. I refuse to stop writing. I mean, I've stopped, but I will not lose this battle. I just don't see a way forward. Self-loathing is so ugly, and so pathetic.
03 June, 2009
Hey, wtf?
I'm drowning at work. No twitter, no aim. Sorry. It may be a while.
Behold today's wisdom:
$num_redbulls = $num_hours / 4;
$num_bananas = $num_hours / 8;
$breakfast = { redbull => $num_redbulls, bananas => $num_bananas };
Behold today's wisdom:
$num_redbulls = $num_hours / 4;
$num_bananas = $num_hours / 8;
$breakfast = { redbull => $num_redbulls, bananas => $num_bananas };
27 May, 2009
SHE RIDES THE HAMMER!
Pay no attention to the Crocs, dear reader! $wife fits the hammer, and I intend to ease her onto it and show her what a real terror of a bike is like. Which is to say, sort of like injecting cocaine. At least, that's what I'm told.
It just occurred to me
That some of the most rewarding things I've ever done, professionally as an engineer, have involved writing. I love writing documentation, specification files, creating policy, creating diagrams of architectures, and helping to explain these to people.
Maybe, in addition to my loving being a writer, this is the career change I've been looking for. Maybe there is hope.
Maybe, in addition to my loving being a writer, this is the career change I've been looking for. Maybe there is hope.
26 May, 2009
I sometimes wonder why I bother
Sure, writing good code is hard. Making big decisions about what you're going to do for the next 2-4 years is hard. Picking a vendor is hard. Teaching developers is hard. Writing meaningful diagrams is hard. Asking useful questions is hard.
But none of this is as hard as not having done any of it.
What is the point of keeping a wiki if nobody uses it? Why on earth would anyone bother researching products and vendors, technologies and organizations if only to throw their hands up and say "you know, it didn't work last time I tried..."?
I'm a serious, serious pessimist. I see failure everywhere. But the important difference between me and Mr. Joe-You're-Ugly is that where I see failure, I see an opportunity to fix something; to make something functional and beautiful. I see failure all around me, every day, from potholes, to laws, to people driving poorly, and, yes, in the workplace. I strive to fix these things, and people hate it. They feel like, once a shortcoming has been identified — even if you've sworn to help them make it better — that immediately, you're the enemy, and they'd rather clutch on to that failure and be proud of it because, dammit, it's their failure, and they worked hard on it.
It's not about blame, folks. It's about being better people. Let's work together and get past the petty bickering. We don't have to all hug and live in a group home in the Castro, but can we please look at the possibility of positive change as a good thing, rather than something terrifying?
If this happens to me again, people deciding they're scared of growth, I may well lose my faith in humanity all together. I'd really, if that's true, rather go back to missile guidance and warfare simulators. Killing people, at least, has well-defined objectives, you know when you're doing "better," and you know there are always more people to kill.
But none of this is as hard as not having done any of it.
What is the point of keeping a wiki if nobody uses it? Why on earth would anyone bother researching products and vendors, technologies and organizations if only to throw their hands up and say "you know, it didn't work last time I tried..."?
I'm a serious, serious pessimist. I see failure everywhere. But the important difference between me and Mr. Joe-You're-Ugly is that where I see failure, I see an opportunity to fix something; to make something functional and beautiful. I see failure all around me, every day, from potholes, to laws, to people driving poorly, and, yes, in the workplace. I strive to fix these things, and people hate it. They feel like, once a shortcoming has been identified — even if you've sworn to help them make it better — that immediately, you're the enemy, and they'd rather clutch on to that failure and be proud of it because, dammit, it's their failure, and they worked hard on it.
It's not about blame, folks. It's about being better people. Let's work together and get past the petty bickering. We don't have to all hug and live in a group home in the Castro, but can we please look at the possibility of positive change as a good thing, rather than something terrifying?
If this happens to me again, people deciding they're scared of growth, I may well lose my faith in humanity all together. I'd really, if that's true, rather go back to missile guidance and warfare simulators. Killing people, at least, has well-defined objectives, you know when you're doing "better," and you know there are always more people to kill.
25 May, 2009
Rolling what?
I guess most of the dicks had left town today, and we were welcomed at the only motorcycle shop open – a Harley shop in Fairfax. They didn't give us the filthy ricer look when we parked...
Yes, that sexy red thing in the front is my baby, the one behind that is the two fiddy, which $wife dutifully followed me on. Unfortunately, the trip we had planned, Route 50 to Lee Highway (29) to 211 out to Skyline Drive (which we refer to fondly as the Ebisu circuit after the cat-and-mouse chase in Initial D with the Sil-80 on the downhill), was hampered by rain. Me, I'll ride in anything — snow, salt, rain, ice, cold, hot, humid, hail, and so on. But $wife, following tandem with me on the two fiddy, hadn't ridden in this kind of stuff, and I began to worry for her safety, as traffic was becoming aggressive (I guess they wanted a piece of Ebisu, too).
So, instead of making it all the way out to Ebisu, we pulled over at Bull Run Battlefield Park, looked over at its soggy field (I really would have wanted to look around, but trying on the bikes in the gravel, mud, etc., was a recipe for disaster), and turned around to head home.
You might think we didn't get a ride in. We'd planned about two hundred miles of rides today, and instead got in about eighty. But we got a bunch of intangible benefits in:
If the Subaru needs an air filter or an oil change, I'm off to the dealer. Screw that noise. Cars... man, they're a fucking drag. I really, really wish the wife would let me sell the Subaru and get maybe a regular Rex or something like that. Maybe even a Mazdaspeed 3. Maybe I could get the Z all legal-like, and we'd have a car that was sort-of practical.
But, between buying her a BMW bike on the used market, my financing a new super-tourer, and wanting to do work on the Z, we're probably going to need to have equity in the stupid Subaru just to convince the bank they can steal our belongings if we default.
Choice quote of the day, by the way, from a neighbor, looking over the hammer
Guy then relates to me that he used to own a Harley, but that when it got to 120 or so, it hunted and didn't feel stable. I noted that the hammer has all kinds of fairings to keep it safe in the wind, and that the front suspensions on our respective bikes (inverted forks on mine, for example), are way, way, different. Poor guy. I just don't understand the Harley crowd.
So, instead of making it all the way out to Ebisu, we pulled over at Bull Run Battlefield Park, looked over at its soggy field (I really would have wanted to look around, but trying on the bikes in the gravel, mud, etc., was a recipe for disaster), and turned around to head home.
You might think we didn't get a ride in. We'd planned about two hundred miles of rides today, and instead got in about eighty. But we got a bunch of intangible benefits in:
- $wife got in eighty more miles of not riding to work. She gets more proficient every time she gets on the bike, and I love watching her. We can now actually ride tandem, as close as a foot or two from eachother, in the same lane, and stop at exactly the same spot at a light, every single time. This is different when we started riding together.
- I got to watch $wife ride, and saw that she has indeed improved tremendously (although I did notice she kind of leans off one side of the bike which might mean the suspension needs a little tweaking; this is hard — the 250 does not have an adjustable suspension). I did, however, notice that she needs a new chain. There's a lot of slack in that chain, and since she's been complaining about throttle slop, this may actually be take-up in the chain. I'm going to have a look and see if I can fix it with the tensioner, and if not, I may just replace the chain and sprockets (and might even sneak in a -1/+2 on her).
- She complained the brakes were a little squishy. Well, compared to the hammer, they are a little squishy, and I might have had a hard time following the hammer today on her bike. However, I've ridden that bike a couple thousand miles, and when you get used to the brakes, it's fine. I adjusted her lever so the pull is quicker, and I think I'm going to adjust the cable for her, so it's even snappier. I just don't want her to lock up the brake.
- The front brake rotor (not two!!) is glazed. I should probably replace the rotors and pads, bleed the system, and feed it new Motul. But, do I want to do this on a bike we're probably going to sell?
- The rear brake may be lagging a little, but pulls hard enough to stall the bike out if you're not careful, so it may just be time for a good brake service.
- She complained the transmission was kind of sloppy. Now, bear in mind the bike makes, literally, a hundred less horsepower than the hammer, so when I go to take the two fiddy out to see if its transmission is sloppy, I don't get the characteristic kick in the ass from first gear (on the hammer, you stomp down into first, and it kicks you back just as hard and the front wheel wants to rise on up). Now, her brake lever is adjustable, and adjusted as of today, but the clutch lever is not. So, I can adjust the clutch cable, which I haven't done before, but I have the factory service manual and all the tools, and, hell, I'm a competent mechanic. The other thing is, and you gotta understand motorcycles to understand this, the shift lever is adjustable for throw. You can adjust two nuts, one on the actual shift knob on the transmission (crank case?) and the other at the base of the shift lever. If you adjust both of these so that the connecting link is as short as it can be, the throw is consequently maybe a half an inch or an inch shorter, which is a lot when you are shift with a pedal. So, I think I'm gonna do that for her.
- The bike is filthy. I rode it out in the snow, salt, and rain, and the crankcase has all kinds of white filth on it, and everything is just dirty. It also needs a turn signal. So I'll order the signal at the dealer (I ordered flush signals, but they just look sheisty with their fake carbon fibre plastic, and they require dremeling the fascias, which I'm just not gonna do). So, I'm probably gonna simple green the crankcase, honda-polish and buff the rest of the bike, and clean the glass of her bubble (which is smoked, but alas...) and her mirrors. For this, I hope she will love me eternally.
- The other thing is we took both bikes out and made sure that the tire pressures were correct (in my case, I opted for a little more pressure; she's just about right), and we started feeding her premium gas. She'd been feeding it 87. The manual says this is okay, but this is before they started both oxygenating gas and adding 10-15% methanol to the pumps. So we go to the station that only does 10% instead of 15%, and with a tank of premium, her bike made more power (it was funny to watch her be surprised by the solid "thunk" of a shift from time to time today), doesn't smoke under wide open throttle, and generally runs better. Hell, at $6 a tank, and her getting 70mpg, it's not like we're breaking the bank.
- My bike's been overheating (or close to) in 395 traffic during rush hour. I picked up a bottle of Redline Water Wetter at the auto parts store, and I think I'm gonna execute a coolant flush. It being summer, I think I may just run 90/10 water and glycol (we do get down in the 40's sometimes, there's no telling if some demonic cold snap would bust open my block, so best to have at least a little glycol in there...). So, this is good.
If the Subaru needs an air filter or an oil change, I'm off to the dealer. Screw that noise. Cars... man, they're a fucking drag. I really, really wish the wife would let me sell the Subaru and get maybe a regular Rex or something like that. Maybe even a Mazdaspeed 3. Maybe I could get the Z all legal-like, and we'd have a car that was sort-of practical.
But, between buying her a BMW bike on the used market, my financing a new super-tourer, and wanting to do work on the Z, we're probably going to need to have equity in the stupid Subaru just to convince the bank they can steal our belongings if we default.
Choice quote of the day, by the way, from a neighbor, looking over the hammer
Say, how fast does that thing go?
Oh, about a hundred and eighty.
Really? Do you do that, like, on a track?
A track? Nah. 395. I've only ever had it up to 140 or so. It's like absolute power, you know, it corrupts absolutely. You just reach for a hand full of throttle, and at the top of fourth gear, you know the bike will take you all the way to the top of sixth, and you're going to be doing north of 170mph. It's absolutely irresistible. I can't stand cars anymore, and I've got two fast ones.
Guy then relates to me that he used to own a Harley, but that when it got to 120 or so, it hunted and didn't feel stable. I noted that the hammer has all kinds of fairings to keep it safe in the wind, and that the front suspensions on our respective bikes (inverted forks on mine, for example), are way, way, different. Poor guy. I just don't understand the Harley crowd.
Trips
$wife doesn't want to do the Yukon, even if I get her an R1150GS or R1300GS. Instead, I am planning a trip from DC to Seattle via Reno and the Gerlach, then back down through Portland to Santa Cruz, Monterey, hopefully with a stay at the Hyatt Highlands Inn, and then back west through Yosemite, and maybe back up through Montana or Yellowstone on the way home. I want to put 10-12,000 miles on the bike, and I want her to, also, so her skills get honed to the point where I feel safe taking her anywhere. Any bike she gets will likely be a boxer or a KLR, and have big old crashbars up front, protecting boxer cylinder heads, feet, and the bike in general. I'm real worried about taking her out on a shaft-drive bike, because if the shaft breaks, you're done. Chains and sprockets on the other hand, you're okay with. It's not fun to fix, but it's something you don't have to weld, and you can even do it in the field if you have a spare, which you can actually carry on a bike. (original credit for the shaft photo Doug Grosjean)Me, I love my ZX-7. The hammer is a splendiferous bike. But, it lacks any carriage other than what I can net to it, and I am not going to ride ten thousand frickin miles with everything on my back. For my part, I've been looking at Hayabusas (because they can have hard bags), Concours 14's, and that new BMW K1300GT (aforementioned shaft-drive problem acknowledged).
So here they are:
The Suzuki Hayabusa:
Although I really think I like some of the other color schemes a bit more.
All this really puts a cramp in my style, because ever since I heard it running around the Ascari track, I've wanted a Katoom. Behold, the RC8:
The good news is, Charley Boorman, minor celebrity, but a man of like-mind, and someone who believes people should get out and see the world (showing that people are all just people, and this whole business of killing eachother over which bit of brush you own is just outdated, retarded, and pointless), has a site for planning trips at Big Earth. Google Earth, for all its directions, satellite imagery, overlays, and all that, is not real good at multi-destination maps.
The important thing is that we're able to carry laptops with us (probably the Air and a PC), a GPS, paper maps, and some token clothing. GPS-wise, I am thinking of a Nokia N96 or N97, and of course $wife's BMW will come with its own GPS.
Then there's that whole "taking six weeks off work" thing. I can't believe that someone would fire me for wanting to do that, especially if I'd proven that I was a worthwhile member of the team. My wife's employer is not real thrilled by the idea of her being gone that long either. But, really, why would employers begrudge their employees the opportunity, no, necessity, of getting out, figuring out what you're made of, and seeing the world? Why consign us all to these concrete caves, cubicles, office chairs, and shackle us to computers? The computers will all be there when we get back, along with the chairs and offices, and even if I'm gone for six weeks, your company isn't going to fall apart. And furthermore, it's going to take six fucking weeks to hire somebody to replace either of us.
No, our respective employers would fire us out of spite, for having the audacity to think that we could go be human somewhere, instead of wage slaves. This, I surmise, and I may be proven wrong, but I doubt it.
24 May, 2009
And I'm done for the day.
I had a lot of stuff saved up; my comcast modem caught fire or something over the weekend, and instead of working on Puppet, I was offline reading CJ Cherryh. Worse things have happened, to be sure.
It is worth noting that the guy that came out was really solid, and when I told him I was under no circumstances rebooting any of my machines, he let me be a prick about it. Good show.
Fucking harleys. Bring on the rain.
(note: we have business service, so they'll come see us on a sunday. fat chance if you're just a residential puke)
It is worth noting that the guy that came out was really solid, and when I told him I was under no circumstances rebooting any of my machines, he let me be a prick about it. Good show.
Fucking harleys. Bring on the rain.
(note: we have business service, so they'll come see us on a sunday. fat chance if you're just a residential puke)
Rolling Thunder
Guys, you make me fucking sick. You don't wave. You don't wear proper gear. Some of you asshats aren't even wearing helmets. You parked your goddamn trailers all up and down my street, most of which were emblazoned with "ROLLING THUNDER" and variations of eagles and flags on the side. You think it's real cool to roll full-throttle onto first with your flatulent pipes at 0300 on route 1, which has a population of over 12,000 people. You come out here, to washington dc, from your dentist jobs, with the harleys you leave on a trickle charger for the rest of a year, and under a cover at that, and you bring your wife with you. But, heaven forbid she actually know how to ride a fucking motorcycle. You're all riding these enormous fucking harleys with big, cushy rear seats she's sitting in, waiting to get killed when you can't make a fucking u-turn in three lanes and you topple over.
You're impolite. You're here because it's a fucking circus. You're here congesting the roads and making lives for the residents unpleasant. And you haven't even got the decency to be respectful of the locals.
I washed my fucking bike today. I took it out and I lubed the chain, I checked the tires, and I got me a tank of gas. I figured I'd go out and see some of you harley fuckers, and maybe some of you wouldn't be the absolute pricks you were last year. But, indeed you were. To a one. Every single one of you bastards ignored me. I rode up behind a pair of you riding two-wide in a lane, and pulled into the lane to the right and looked over, and I guess I wasn't fucking good enough for you to even fucking acknowledge me.
Consider, if I'd passed you on one wheel and given you the finger, you'd have said something like "fucking ricer squid," and yet, when I ride, polite as can be, next to you, look over and smile, you're saying the same fucking thing.
Go retire, you decrepit fucks. Sell your harleys and buy RV's. Go tour the grand canyon. But you're not here for the Armed Forces, and I really, really, don't care if you fucking served. I work with people who serve presently and they concur you're a bunch of douchebags.
Get the fuck out of DC, or shape the fuck up. You all make me sick. I hope these thunder clouds break open and cover your cute little trailered million-pound choppers with the ugliest, coldest rain there is. I hope you all cower in the fucking hilton for the duration of your stay here. Me, I'd go out, rain or shine, just to enjoy being on two wheels, but I can't abide being near your unfortunate asses another second.
You disgust me.
You're impolite. You're here because it's a fucking circus. You're here congesting the roads and making lives for the residents unpleasant. And you haven't even got the decency to be respectful of the locals.
I washed my fucking bike today. I took it out and I lubed the chain, I checked the tires, and I got me a tank of gas. I figured I'd go out and see some of you harley fuckers, and maybe some of you wouldn't be the absolute pricks you were last year. But, indeed you were. To a one. Every single one of you bastards ignored me. I rode up behind a pair of you riding two-wide in a lane, and pulled into the lane to the right and looked over, and I guess I wasn't fucking good enough for you to even fucking acknowledge me.
Consider, if I'd passed you on one wheel and given you the finger, you'd have said something like "fucking ricer squid," and yet, when I ride, polite as can be, next to you, look over and smile, you're saying the same fucking thing.
Go retire, you decrepit fucks. Sell your harleys and buy RV's. Go tour the grand canyon. But you're not here for the Armed Forces, and I really, really, don't care if you fucking served. I work with people who serve presently and they concur you're a bunch of douchebags.
Get the fuck out of DC, or shape the fuck up. You all make me sick. I hope these thunder clouds break open and cover your cute little trailered million-pound choppers with the ugliest, coldest rain there is. I hope you all cower in the fucking hilton for the duration of your stay here. Me, I'd go out, rain or shine, just to enjoy being on two wheels, but I can't abide being near your unfortunate asses another second.
You disgust me.
Tell us about system load, Alex
Let's first look at this script:
Many people don't understand the difference between system load, and processor load. Earlier today, when I was running a Windows VM, a Solaris VM, handbrake, Visualhub, two instances of iTunes, deleting 65,000 files, and using Windows to unrar two 12-gb files segmented into 100-rar-file elements, I had the load well above twenty. And yet, if we look at the actual idle percentage of the CPU, we see that it's not doing a whole lot.
The reason for this is that you're waiting for data to move across the network (even if it is a loopback, host-only network, in the case of the VMs), you're waiting on disks to cough up their data and their reads/writes to finish, and, sadly, you're waiting for data to be pushed across RAM. Let me show you one of Apple's favorite documents:
When we look at the Intel design, we see one that is slightly more mature, in that there are fewer bottlenecks between devices, processors, memory, buses, and so on, but they have made the same mistakes. You're still going to struggle to get data from one side of the computer to the other.
This is what we call "iowait." Sometimes (like on Solaris), it's called iow, FreeBSD calls it something similar, and it is the enemy of computing. Because you paid a heck of a lot of money for those processors, and if they're sitting at 48% idle, why on earth did you spend all that money?
Sun's solution with the SPARC and with the newer Niagara processors and architecture has been to make machines so massively parallel and multi-linked (note: IBM is doing this too) and having such enormous bandwidth between components (Sun owns technology they purchased, jointly, with SGI, from Cray, which Cray at the time called CrayLink — it's sort of like God tapping you on the shoulder and saying "Oh, hai there, I got ur bits..."). But, even a Sun or an IBM SMP will choke on IO, and you'll wind up with a system that is fantastically loaded, and yet, at the same time, it is indeed idle.
The other demon that plagues us as systems administrators, when we're trying to figure out why our systems reporting software is saying "ZOMG THE LOAD IS EIGHT JIGGOWATTS" is "sys". Sys is a little more insidious. Let's say that it's the kernel's job to overcome all the failings of a given architecture, such as those indicated above (and let me be fair to Apple; those are very nice architectures, and for desktop- or commodity-level machines, it's pretty goddamn good to be "like an Octane." Anyone who can guess the price delta between an R12k Octane2 and a PowerMac of the same day gets extra credit). You're running (as we've previously talked about in the virtualization soliloquy), say, sixty webservers, and each of those webservers has maybe four thousand connections. It's easy to do the math here, and realize that you've got GOBS AND GOBS of open connections. The kernel, because it handles things like TCP (which, sadly, your web serving daemon does not), is frantically trying to assemble TCP packets, your firewall (also part of sys) is probably trying to either drop or reassemble fragmented packets, and it's trying to maintain state when you have a jillion open, half-open, talking, and closing connections.
Let me show you how this looks on my, fairly idle, desktop:
Now, my kernel is trying to maintain state on all of these connections, to make sure that people get the bits they want, that people who don't want to talk to us get their sockets closed, and that people who do want to talk to us get a new socket to listen on.
This takes up a lot of time. And unfortunately, there's nothing we can do about it, because we use computers to talk to people. The kernel spends an enormous amount of time just making sure we can do precisely that. So if you start experimenting with commands like iostat(1), vmstat(1), or sysctl on the mac, you're going to notice that a vast majority of the time, the system isn't actually crunching your transcode from matroshka to mp4. Instead, it's desperately waiting for that giant USB array you have attached to your computer to feed it the bits it needs to actually do that transcoding. Even if you have a really, really, fast processor.
Other things that impact sys are logical volume management (although with lufs, this is sort of changing), sar and other accounting processes, interprocess communication, and managing multiprocessor systems (maintaining things like processor affinity and process "nice"ness).
You can sort of reduce the load your system is going to spend on TCP by getting what's called a TCP Offload Engine or TOE, which is usually built into your NIC. Most companies aren't shipping these, but Alienware and other boutique vendors will ship you a "Extreme Gaming Network Card" that incorporates features of a TOE. If you're living in a cluster, you can get NICs with RDMA, and of course using ultrafast filesystems like myrinet and infiniband will also allow you to use your processor, rather than waiting around for your stupid disks to spin up, hit the right sector, or whatever.
So next time you're looking at every newbie's favorite command, uptime, and you see OMG THE LOAD IS SO HIGH, have a little deeper look, and figure out what your computer is doing, rather than assuming that you need something faster. An Ultra 80 with four UltraSPARC 450mhz cpu's is still a badass monster of a machine if you stuff it full of ram and 15krpm SCSI 320 drives. Take this to heart. Don't just go buy the fastest proc you can. Get the fastest disks you can, and attach them internally, or on a PCI card, and make sure they have the highest rotational speed you can find. Get the highest speed ram you can fit in your machine, and get it with fancy cooling fins so your bits move faster through the ram.
Your video card, trust me, is unimportant, unless you have a tesla, and if you have a tesla, you probably didn't even bother to read this far.
I'm real tired, though, of people worrying about system loads. I've seen system loads over a thousand. No shit, the load on a six-processor E3500 was creeping up from 970 to briefly over a thousand before it settled down. It was having a bad FCAL day. But those processors were idle, folks.
Computers are complicated, and that stuff you read on Ars and Toms Hardware, and BYTE Magazine and Accelerate Your Mac — it's all bullshit. Really. You need to learn you some engineering if you're going to tell me that your system is overloaded.
#!/bin/bash
# Developed on Darwin. This won't work on Solaris, and probably won't work on Linux, either.
echo -n "number of cores available: " ; \
sysctl hw.ncpu | cut -d: -f 2
echo -n "users on system: " ; \
ps xawu | cut -f 1 -d ' ' | sort -u | wc -l ;
echo -n "system load averages: " ; \
sysctl vm.loadavg | tr -d '{}' | cut -d: -f 2 ;
echo -n "distinct processes on system: "; \
ps xa | wc -l ;
echo -n "actual cpu idle percentage: " ; \
iostat 1 | head -3 | tail -1 | tr -s ' ' | cut -d ' ' -f 11
# aja // vim: ts=2 noet
We get an output that looks something like this:number of cores available: 8 users on system: 16 system load averages: 7.44 8.72 8.20 distinct processes on system: 136 actual cpu idle percentage: 48
Many people don't understand the difference between system load, and processor load. Earlier today, when I was running a Windows VM, a Solaris VM, handbrake, Visualhub, two instances of iTunes, deleting 65,000 files, and using Windows to unrar two 12-gb files segmented into 100-rar-file elements, I had the load well above twenty. And yet, if we look at the actual idle percentage of the CPU, we see that it's not doing a whole lot.
The reason for this is that you're waiting for data to move across the network (even if it is a loopback, host-only network, in the case of the VMs), you're waiting on disks to cough up their data and their reads/writes to finish, and, sadly, you're waiting for data to be pushed across RAM. Let me show you one of Apple's favorite documents:
And their earlier, much more SGI Octane-like G5,
We very quickly start to see the problems with this. With the G5 for starters, we see the disks and all the external buses, like 1394a/b, USB, Bluetooth, and even wireless, living off on their own controller. This is, incidentally, what caused so much heat on my Macbook that it self-immolated and burned traces off the board, melting the case, and resulting in my getting an Air (which, let's face it, doesn't really have an architecture so much as magic smoke that just works).
When we look at the Intel design, we see one that is slightly more mature, in that there are fewer bottlenecks between devices, processors, memory, buses, and so on, but they have made the same mistakes. You're still going to struggle to get data from one side of the computer to the other.
This is what we call "iowait." Sometimes (like on Solaris), it's called iow, FreeBSD calls it something similar, and it is the enemy of computing. Because you paid a heck of a lot of money for those processors, and if they're sitting at 48% idle, why on earth did you spend all that money?
Sun's solution with the SPARC and with the newer Niagara processors and architecture has been to make machines so massively parallel and multi-linked (note: IBM is doing this too) and having such enormous bandwidth between components (Sun owns technology they purchased, jointly, with SGI, from Cray, which Cray at the time called CrayLink — it's sort of like God tapping you on the shoulder and saying "Oh, hai there, I got ur bits..."). But, even a Sun or an IBM SMP will choke on IO, and you'll wind up with a system that is fantastically loaded, and yet, at the same time, it is indeed idle.
The other demon that plagues us as systems administrators, when we're trying to figure out why our systems reporting software is saying "ZOMG THE LOAD IS EIGHT JIGGOWATTS" is "sys". Sys is a little more insidious. Let's say that it's the kernel's job to overcome all the failings of a given architecture, such as those indicated above (and let me be fair to Apple; those are very nice architectures, and for desktop- or commodity-level machines, it's pretty goddamn good to be "like an Octane." Anyone who can guess the price delta between an R12k Octane2 and a PowerMac of the same day gets extra credit). You're running (as we've previously talked about in the virtualization soliloquy), say, sixty webservers, and each of those webservers has maybe four thousand connections. It's easy to do the math here, and realize that you've got GOBS AND GOBS of open connections. The kernel, because it handles things like TCP (which, sadly, your web serving daemon does not), is frantically trying to assemble TCP packets, your firewall (also part of sys) is probably trying to either drop or reassemble fragmented packets, and it's trying to maintain state when you have a jillion open, half-open, talking, and closing connections.
Let me show you how this looks on my, fairly idle, desktop:
thunder% netstat -an | grep -vi listen | grep -v dgram | grep -v stream Active Internet connections (including servers) Proto Recv-Q Send-Q Local Address Foreign Address (state) tcp4 0 124 10.1.10.140.49344 24.86.166.100.31224 ESTABLISHED tcp4 0 0 10.1.10.140.49343 74.125.47.19.443 ESTABLISHED tcp4 0 0 10.1.10.140.49341 78.180.207.90.6348 ESTABLISHED tcp4 0 26 10.1.10.140.49335 88.192.26.203.61845 ESTABLISHED tcp4 0 0 10.1.10.140.49332 116.49.123.10.11881 ESTABLISHED tcp4 0 0 10.1.10.140.49319 194.109.248.110.36921 ESTABLISHED tcp4 0 0 10.1.10.140.49316 82.161.104.88.46031 ESTABLISHED tcp4 0 19250 10.1.10.140.49314 60.241.73.188.25000 ESTABLISHED tcp4 0 36656 10.1.10.140.49312 190.212.10.170.64283 ESTABLISHED tcp4 0 0 10.1.10.140.49310 58.70.117.71.16499 ESTABLISHED tcp4 0 0 10.1.10.140.49309 194.109.248.110.36921 ESTABLISHED tcp4 0 733 10.1.10.140.49308 85.228.205.19.36917 ESTABLISHED tcp4 0 0 10.1.10.140.49296 58.70.117.71.16499 ESTABLISHED tcp4 0 0 10.1.10.140.49291 96.21.108.102.30965 ESTABLISHED tcp4 0 0 10.1.10.140.49287 85.186.177.129.27194 ESTABLISHED tcp4 0 19964 10.1.10.140.49272 83.8.19.193.27797 ESTABLISHED tcp4 0 0 10.1.10.140.49269 60.241.73.188.25000 ESTABLISHED tcp4 0 0 10.1.10.140.49268 65.23.200.241.24890 ESTABLISHED tcp4 0 0 10.1.10.140.49260 151.53.4.153.57150 ESTABLISHED tcp4 0 0 10.1.10.140.49255 151.53.4.153.57150 ESTABLISHED tcp4 0 291 10.1.10.140.49247 194.109.248.110.36921 LAST_ACK tcp4 0 0 10.1.10.140.49245 173.128.165.180.3724 ESTABLISHED tcp4 0 0 10.1.10.140.49244 81.70.142.212.55986 ESTABLISHED tcp4 0 0 10.1.10.140.49242 85.83.1.95.8900 ESTABLISHED tcp4 0 0 10.1.10.140.49240 24.68.36.126.39271 ESTABLISHED tcp4 0 0 10.1.10.140.49239 80.99.34.174.6881 ESTABLISHED tcp4 0 0 10.1.10.140.49228 86.198.85.136.40695 ESTABLISHED tcp4 0 51691 10.1.10.140.49226 85.225.232.241.60167 ESTABLISHED tcp4 0 0 10.1.10.140.49225 85.241.164.146.53034 ESTABLISHED tcp4 0 8 10.1.10.140.49224 79.136.62.72.64299 ESTABLISHED tcp4 0 0 10.1.10.140.49205 92.133.223.226.59472 LAST_ACK tcp4 0 0 10.1.10.140.49204 88.192.26.203.61845 ESTABLISHED tcp4 0 0 10.1.10.140.49203 79.17.234.126.45930 ESTABLISHED tcp4 0 0 10.1.10.140.49193 151.53.4.153.57150 ESTABLISHED tcp4 0 0 10.1.10.140.65530 65.23.200.241.24890 ESTABLISHED tcp4 0 0 10.1.10.140.65518 88.192.26.203.61845 ESTABLISHED tcp4 0 739 10.1.10.140.65501 78.180.218.159.6348 ESTABLISHED tcp4 0 299 10.1.10.140.65494 78.180.218.159.6348 ESTABLISHED tcp4 0 39204 10.1.10.140.65485 60.241.130.179.6501 ESTABLISHED tcp4 0 0 10.1.10.140.65478 200.75.235.203.46125 ESTABLISHED tcp4 0 385 10.1.10.140.65469 78.180.218.159.6348 ESTABLISHED tcp4 0 0 10.1.10.140.65439 82.161.104.88.46031 ESTABLISHED tcp4 0 0 10.1.10.140.65438 81.217.78.140.7777 ESTABLISHED tcp4 0 0 10.1.10.140.65432 79.17.234.126.45930 ESTABLISHED tcp4 0 0 10.1.10.140.65430 78.105.187.201.25321 ESTABLISHED tcp4 0 0 10.1.10.140.65425 74.125.93.191.80 ESTABLISHED tcp4 0 0 10.1.10.140.65422 67.172.251.35.62523 ESTABLISHED tcp4 0 0 10.1.10.140.65324 70.83.32.117.48156 ESTABLISHED tcp4 0 0 10.1.10.140.65305 78.105.187.201.25321 ESTABLISHED tcp4 0 12291 10.1.10.140.65254 60.241.73.188.25000 FIN_WAIT_1 tcp4 0 0 10.1.10.140.65172 88.192.26.203.61845 ESTABLISHED tcp4 0 65605 10.1.10.140.65154 91.153.113.50.56319 ESTABLISHED tcp4 0 0 10.1.10.140.65145 88.90.235.18.49153 ESTABLISHED tcp4 0 39116 10.1.10.140.65136 79.172.92.36.37531 ESTABLISHED tcp4 0 0 10.1.10.140.65132 60.241.73.188.25000 CLOSING tcp4 0 13477 10.1.10.140.65122 69.242.186.238.45696 ESTABLISHED tcp4 0 34912 10.1.10.140.65119 79.172.92.36.37531 ESTABLISHED tcp4 0 0 10.1.10.140.65053 85.134.20.25.47537 ESTABLISHED tcp4 0 0 10.1.10.140.65025 60.241.130.179.6501 ESTABLISHED tcp4 0 0 10.1.10.140.64921 58.182.194.24.31822 ESTABLISHED tcp4 0 31980 10.1.10.140.64874 92.133.223.226.59472 FIN_WAIT_1 tcp4 0 66608 10.1.10.140.3689 10.1.10.40.57528 ESTABLISHED tcp4 0 10486 10.1.10.140.64747 97.104.232.8.35450 ESTABLISHED tcp4 0 53 10.1.10.140.64708 190.212.10.170.64283 FIN_WAIT_1 tcp4 0 0 10.1.10.140.64700 85.241.164.146.53034 ESTABLISHED tcp4 0 0 10.1.10.140.64648 78.105.187.201.25321 ESTABLISHED tcp4 0 0 10.1.10.140.64594 82.161.104.88.46031 ESTABLISHED tcp4 0 0 10.1.10.140.64593 96.49.23.203.34162 ESTABLISHED tcp4 0 0 10.1.10.140.64587 77.41.89.217.60847 ESTABLISHED tcp4 0 6144 10.1.10.140.64583 98.14.25.190.24852 ESTABLISHED tcp4 0 0 10.1.10.140.64467 24.87.89.150.32828 ESTABLISHED tcp4 0 0 10.1.10.140.64431 173.128.165.180.3724 ESTABLISHED tcp4 0 0 10.1.10.140.64374 190.207.230.140.41352 ESTABLISHED tcp4 0 30489 10.1.10.140.64320 94.21.56.185.58172 ESTABLISHED tcp4 0 11406 10.1.10.140.64315 93.6.97.49.25583 ESTABLISHED tcp4 0 49208 10.1.10.140.64273 93.6.97.49.25583 ESTABLISHED tcp4 0 0 10.1.10.140.64112 94.66.9.233.63929 ESTABLISHED tcp4 0 40942 10.1.10.140.64034 83.57.55.155.52328 ESTABLISHED tcp4 0 0 10.1.10.140.63965 82.131.25.19.45060 ESTABLISHED tcp4 0 0 127.0.0.1.63827 127.0.0.1.63829 ESTABLISHED tcp4 0 0 127.0.0.1.63829 127.0.0.1.63827 ESTABLISHED tcp4 0 0 10.1.10.140.63650 213.93.230.16.51002 ESTABLISHED tcp4 0 32794 10.1.10.140.63354 220.233.193.86.31914 ESTABLISHED tcp4 0 61210 10.1.10.140.63334 173.48.24.137.52734 ESTABLISHED tcp4 0 52175 10.1.10.140.63327 173.48.24.137.52734 ESTABLISHED tcp4 0 27686 10.1.10.140.63004 95.222.204.238.55253 ESTABLISHED tcp4 0 64268 10.1.10.140.62840 58.70.117.71.16499 ESTABLISHED tcp4 0 43379 10.1.10.140.62805 76.10.186.223.34335 ESTABLISHED tcp4 0 31077 10.1.10.140.62644 85.166.87.17.37397 ESTABLISHED tcp4 0 49005 10.1.10.140.62396 85.166.87.17.37397 ESTABLISHED tcp4 0 20365 10.1.10.140.62256 208.107.112.237.65000 ESTABLISHED tcp4 0 14144 10.1.10.140.62246 118.166.46.152.27888 ESTABLISHED tcp4 0 66608 10.1.10.140.61869 72.229.126.193.51413 ESTABLISHED tcp4 0 8 10.1.10.140.61853 24.68.36.126.39271 FIN_WAIT_1 tcp4 0 15707 10.1.10.140.61834 78.115.126.26.58474 ESTABLISHED tcp4 0 63133 10.1.10.140.61718 72.198.3.164.10051 ESTABLISHED tcp4 0 13870 10.1.10.140.61674 84.29.95.34.51978 ESTABLISHED tcp4 0 34791 10.1.10.140.61557 85.222.85.23.61687 ESTABLISHED tcp4 0 11113 10.1.10.140.61360 69.249.68.41.13708 ESTABLISHED tcp4 0 32794 10.1.10.140.61229 85.142.54.17.19999 ESTABLISHED tcp4 0 51745 10.1.10.140.60973 60.241.130.179.6501 ESTABLISHED tcp4 0 66240 10.1.10.140.60969 90.178.189.57.18596 ESTABLISHED tcp4 0 1766 10.1.10.140.60913 85.186.35.54.55933 ESTABLISHED tcp4 0 20744 10.1.10.140.60690 91.140.99.154.41854 ESTABLISHED tcp4 0 65894 10.1.10.140.60664 58.70.117.71.16499 ESTABLISHED tcp4 0 55729 10.1.10.140.60619 58.70.117.71.16499 ESTABLISHED tcp4 0 30477 10.1.10.140.60552 79.172.92.36.37531 ESTABLISHED tcp4 0 65894 10.1.10.140.60449 116.49.123.10.11881 ESTABLISHED tcp4 0 61362 10.1.10.140.60193 83.247.44.159.59045 ESTABLISHED tcp4 0 0 10.1.10.140.60131 24.87.89.150.32828 ESTABLISHED tcp4 0 23400 10.1.10.140.60126 77.106.172.230.51897 ESTABLISHED tcp4 0 2810 10.1.10.140.60051 24.86.166.100.31224 ESTABLISHED tcp4 0 654 10.1.10.140.59486 84.29.95.34.51978 ESTABLISHED tcp4 0 0 127.0.0.1.389 127.0.0.1.59452 ESTABLISHED tcp4 0 0 127.0.0.1.59452 127.0.0.1.389 ESTABLISHED tcp4 0 64985 10.1.10.140.59451 72.81.236.231.15206 ESTABLISHED tcp4 0 0 10.1.10.140.59191 24.86.166.100.31224 ESTABLISHED tcp4 0 0 10.1.10.140.59190 24.86.166.100.31224 ESTABLISHED tcp4 0 32754 10.1.10.140.59172 76.65.69.240.46795 ESTABLISHED tcp4 0 35133 10.1.10.140.59142 76.10.186.223.34335 ESTABLISHED tcp4 0 0 10.1.10.140.58987 79.186.250.63.43609 ESTABLISHED tcp4 0 29150 10.1.10.140.58893 76.65.69.240.46795 ESTABLISHED tcp4 0 0 10.1.10.140.58858 88.192.26.203.61845 ESTABLISHED tcp4 0 7527 10.1.10.140.58783 60.241.73.188.25000 ESTABLISHED tcp4 0 25266 10.1.10.140.58781 213.46.222.187.45896 ESTABLISHED tcp4 0 29472 10.1.10.140.58704 76.190.208.111.57017 ESTABLISHED tcp4 0 16045 10.1.10.140.58558 60.241.130.179.6501 ESTABLISHED tcp4 0 0 10.1.10.140.4111 10.1.10.140.58528 ESTABLISHED tcp4 0 0 10.1.10.140.58528 10.1.10.140.4111 ESTABLISHED tcp4 0 0 173.8.15.237.4111 173.8.15.237.49176 ESTABLISHED tcp4 0 0 127.0.0.1.5347 127.0.0.1.49175 ESTABLISHED tcp4 0 0 127.0.0.1.49175 127.0.0.1.5347 ESTABLISHED tcp4 0 0 127.0.0.1.5347 127.0.0.1.49166 ESTABLISHED tcp4 0 0 127.0.0.1.49166 127.0.0.1.5347 ESTABLISHED tcp4 0 0 127.0.0.1.5347 127.0.0.1.49165 ESTABLISHED tcp4 0 0 127.0.0.1.49165 127.0.0.1.5347 ESTABLISHED tcp4 0 0 127.0.0.1.5347 127.0.0.1.49164 ESTABLISHED tcp4 0 0 127.0.0.1.49164 127.0.0.1.5347 ESTABLISHED tcp4 0 0 127.0.0.1.5347 127.0.0.1.49163 ESTABLISHED tcp4 0 0 127.0.0.1.49163 127.0.0.1.5347 ESTABLISHED tcp4 0 0 127.0.0.1.5347 127.0.0.1.49162 ESTABLISHED tcp4 0 0 127.0.0.1.49162 127.0.0.1.5347 ESTABLISHED tcp4 0 0 127.0.0.1.5347 127.0.0.1.49161 ESTABLISHED tcp4 0 0 127.0.0.1.49161 127.0.0.1.5347 ESTABLISHED tcp4 0 0 127.0.0.1.5347 127.0.0.1.49160 ESTABLISHED tcp4 0 0 127.0.0.1.49160 127.0.0.1.5347 ESTABLISHED tcp4 0 0 10.1.10.140.65516 24.175.70.220.56601 FIN_WAIT_2 tcp4 0 0 10.1.10.140.49337 10.1.10.140.139 TIME_WAIT tcp4 0 0 10.1.10.140.49238 74.125.47.19.443 TIME_WAIT
Now, my kernel is trying to maintain state on all of these connections, to make sure that people get the bits they want, that people who don't want to talk to us get their sockets closed, and that people who do want to talk to us get a new socket to listen on.
This takes up a lot of time. And unfortunately, there's nothing we can do about it, because we use computers to talk to people. The kernel spends an enormous amount of time just making sure we can do precisely that. So if you start experimenting with commands like iostat(1), vmstat(1), or sysctl on the mac, you're going to notice that a vast majority of the time, the system isn't actually crunching your transcode from matroshka to mp4. Instead, it's desperately waiting for that giant USB array you have attached to your computer to feed it the bits it needs to actually do that transcoding. Even if you have a really, really, fast processor.
Other things that impact sys are logical volume management (although with lufs, this is sort of changing), sar and other accounting processes, interprocess communication, and managing multiprocessor systems (maintaining things like processor affinity and process "nice"ness).
You can sort of reduce the load your system is going to spend on TCP by getting what's called a TCP Offload Engine or TOE, which is usually built into your NIC. Most companies aren't shipping these, but Alienware and other boutique vendors will ship you a "Extreme Gaming Network Card" that incorporates features of a TOE. If you're living in a cluster, you can get NICs with RDMA, and of course using ultrafast filesystems like myrinet and infiniband will also allow you to use your processor, rather than waiting around for your stupid disks to spin up, hit the right sector, or whatever.
So next time you're looking at every newbie's favorite command, uptime, and you see OMG THE LOAD IS SO HIGH, have a little deeper look, and figure out what your computer is doing, rather than assuming that you need something faster. An Ultra 80 with four UltraSPARC 450mhz cpu's is still a badass monster of a machine if you stuff it full of ram and 15krpm SCSI 320 drives. Take this to heart. Don't just go buy the fastest proc you can. Get the fastest disks you can, and attach them internally, or on a PCI card, and make sure they have the highest rotational speed you can find. Get the highest speed ram you can fit in your machine, and get it with fancy cooling fins so your bits move faster through the ram.
Your video card, trust me, is unimportant, unless you have a tesla, and if you have a tesla, you probably didn't even bother to read this far.
I'm real tired, though, of people worrying about system loads. I've seen system loads over a thousand. No shit, the load on a six-processor E3500 was creeping up from 970 to briefly over a thousand before it settled down. It was having a bad FCAL day. But those processors were idle, folks.
Computers are complicated, and that stuff you read on Ars and Toms Hardware, and BYTE Magazine and Accelerate Your Mac — it's all bullshit. Really. You need to learn you some engineering if you're going to tell me that your system is overloaded.
running nginx on OSX
I run Leopard Server, and I have no need for Apache 1.3, Apache 2.x, Apple's wikid, their calendar server, and a bunch of other things that are all fixated on Apache and its configurations in launchd. Right now would be a good time for you to familiarize yourself with the launchd and launchctl man pages, although the referenced ("See Also") command, launch, does not exist.
I've written an nginx launchd configuration file which happily starts and stops on command, starts on boot of course, and has the required configurations. nginx, in general, is a simpler, better written, faster httpd than apache is (which is, after all, a patchy web server). It follows:
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>Label</key>
<string>nginx</string>
<key>Program</key>
<string>/usr/local/sbin/nginx</string>
<key>KeepAlive</key>
<true/>
<key>NetworkState</key>
<true/>
<key>StandardErrorPath</key>
<string>/var/log/system.log</string>
<key>LaunchOnlyOnce</key>
<true/>
</dict>
</plist>
You will want to place this in /System/Library/LaunchDaemons/nginx.plist, and it is helpful to issue "launchctl load -F /System/Library/LaunchDaemons/nginx.plist", after which it should behave correctly. Of course, the corresponding unload command is simply "unload", with or without the -F. Note that this launches nginx on boot, rather than when you choose it to. It also tells it to log stderr to syslog, which may not be to your preference (nginx likes to log to /usr/local/log/nginx.err). But I think you get the idea.
I've written an nginx launchd configuration file which happily starts and stops on command, starts on boot of course, and has the required configurations. nginx, in general, is a simpler, better written, faster httpd than apache is (which is, after all, a patchy web server). It follows:
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>Label</key>
<string>nginx</string>
<key>Program</key>
<string>/usr/local/sbin/nginx</string>
<key>KeepAlive</key>
<true/>
<key>NetworkState</key>
<true/>
<key>StandardErrorPath</key>
<string>/var/log/system.log</string>
<key>LaunchOnlyOnce</key>
<true/>
</dict>
</plist>
You will want to place this in /System/Library/LaunchDaemons/nginx.plist, and it is helpful to issue "launchctl load -F /System/Library/LaunchDaemons/nginx.plist", after which it should behave correctly. Of course, the corresponding unload command is simply "unload", with or without the -F. Note that this launches nginx on boot, rather than when you choose it to. It also tells it to log stderr to syslog, which may not be to your preference (nginx likes to log to /usr/local/log/nginx.err). But I think you get the idea.
22 May, 2009
Woe is me
My wife won't be home until after ten. I have a week coming up that is punctuated by meeting after meeting after meeting after ... you get it. I worked sixteen hours yesterday, I've already worked nine and I'm pulling for a twelve, I'll be working eight tomorrow (provided everything goes well), and I still feel like I'm getting nowhere.
What is the point of having a wiki, of writing documentation, of laying out a roadmap and explaining paths and tools and everything, if nobody wants to read them because it takes time? What is the point of trying to improve anything if nobody wants to spend the time to do the improving? Why bother? Why not just leave everything broken?
Tools, like virtualization, automated deploy programs like Puppet, and packaging systems like CSW and SunFreeware exist to make your life easier through modularity, consistency, patchability, and currency. But if you don't use them, why do you even bother developing? You reinvent the wheel, every. single. time.
And that means somebody has to come along, take your wheel, shave the corners off it, and fit it to the vehicle it's supposed to be attached to.
Really, it's enough to make you cry.
What is the point of having a wiki, of writing documentation, of laying out a roadmap and explaining paths and tools and everything, if nobody wants to read them because it takes time? What is the point of trying to improve anything if nobody wants to spend the time to do the improving? Why bother? Why not just leave everything broken?
Tools, like virtualization, automated deploy programs like Puppet, and packaging systems like CSW and SunFreeware exist to make your life easier through modularity, consistency, patchability, and currency. But if you don't use them, why do you even bother developing? You reinvent the wheel, every. single. time.
And that means somebody has to come along, take your wheel, shave the corners off it, and fit it to the vehicle it's supposed to be attached to.
Really, it's enough to make you cry.
21 May, 2009
Whither thine virtualization?
Listen, if you're not a sysadmin, you don't want to read this. It's really long, and it's got a lot of info, but you're going to be bored to tears unless this is the sort of thing that keeps you up at night.
It's no big secret that I'm a fan of Virtualbox and xVM. Before the unholy alliance of Sun and Oracle, it seemed to me that Sun had VMWare beat with a better product, a better filesystem (ufs had snapshots, and zfs has even better snapshots). But the waters have been muddied somewhat, not only at Sun, but among the Linuxes, as well as in the IBM camp, and the various vendors offering virtualization solutions (including Microsoft).
So, what are we trying to accomplish with virtualization? I see a couple of tasks. If we look at the image above, we see the fairly vanilla use of virtualization: I've got various hosts I use for testing out various things, and in the case of work I need a Windows XP machine to (yes, it sucks, yes, I know, but I've worked around it) fill out my weekly timecards. Note that this is VirtualBox running on my MacPro at home.
Another purpose of virtualization is sort of like the one I have described above, only it works on a larger scale. Let's say you're a development shop and you've got sixty developers. Rather than buy each and every one of them their own development workstation, or expect them to develop on their laptops or desktops, we can buy one bigass machine, run a virtualization host, and "carve off slices" for each of these developers so that they are free to break their development machine. Furthermore, they can take snapshots, and if their machine is so irretrievably broken they can't recover it, a sysadmin might take pity on them and simply carve them off a new virtual machine.
There are two resource constraints with the above model (the one I describe prior to this is handled by a MacPro with 12gb of ram and about 4tb of disk). First, it needs to have enough processor capacity to deal with whatever the developers want to do. Developers have bad habits like running make -j, or telling their database it can allocate all the ram that /etc/system will let it have. So rest assured, your "Virtual Machine Host" is going to need to have multiple CPU's and the current standard seems to be a four-by-four-core Opteron or Xeon machine, and anywhere from sixteen to sixty-four gigs of ram. Remember, each of these users is going to need a few gigs of ram as well. Your mileage may vary, and you may decide that your developers don't get to have a few gigs; they'll just have to make do with one. And, really, Solaris and Linux can get along just fine in less than a gig if nothing exotic is going on.
A third model for virtualization exists, of course, and this is the one where all the money is. Long ago, us grey-beard Sparc admins realized that we really wanted E3500's to run Oracle, but for the most part, the load on the machine never really got above 1, and sometimes it might get as high as 3 or 4. For a machine with four CPU's, this isn't a big deal, and half the time this wasn't because Oracle was pegging the CPU as such, but rather because it was waiting for the SAN to come back with the data it wanted for a given query it was executing (this is called iowait). There were also other niggling details like waiting on swap (kernels, back in the day, had a habit of swapping stuff out even when they didn't really need to; there's a huge discussion about this on the LKML, and even RedHat will tell you to use swap, even when you have tons of RAM) that would raise the system load, even though the processors might actually be idle.
It is important for this reason to understand what the actual meaning of "system load" is in terms of Unix. For a really, seriously, in-depth discussion of it, albeit a bit out of date, there's the Marlin book from O'Reilly. It's a great book, and is probably required reading for anyone who wants to read any further than we're going to go right here.
So we old farts decided, hey, most of the time, our systems are idle. And yet, we have datacenters full of 280R's and E4500's and E3500's and we've got racks and racks of A5200's and yet, most of the time, they're just drawing power, creating heat, costing us money by way of support contracts and failing components (disks, power supplies, cpu's, dimms, etc). Why is this?
Someone – whose name I do not know – got it into their head that, if these systems spend most of their time idle, why not segment the machines into multiple virtual machines? If we take the example of my Mac, I have eight cpu's, and I'm rarely using more than four (between my wife, myself, and iTunes, we really do use four). What about those other four cpu's? What about that extra four gigs of ram that I'm not using? Indeed, the machine is idle. The load may well be 4.0, but by and large, the system is healthy.
So, VirtualBox comes along (although, I think to be fair, VMWare Fusion was first on the Mac, and I don't really know anything about Parallels, but it's been around a while, too; both of these sort of made Boot Camp obsolete), and it says, well, let's carve off sections of disk, and we'll treat them just like they were hard disks themselves. We can mount ISO images (or physical OS media) as though they were in the CDROM of this virtual machine, and we can install an operating system image onto that file that's pretending to be a disk. What the virtualization software in this case does is it lets the "guest" operating system run its own code in a virtual machine (hence the name), and both Intel and AMD have made a sort of "pass-through" option on their CPUs so that it's not even really necessary to emulate – the virtualization software can just say "hey, I'm running software on top of the OS, please just run this guest OS right on the iron, if you would," and the proc complies.
What we get, then, is myself logged in and doing evil things with postgres, mail.app, photoshop, and handbrake, my wife logged in looking at the latest news and downloading Bleach torrents, and our iTunes user logged in serving up content to the AppleTV. BUT, we also have an instance of Windows, or Solaris, or Nexenta, CAOS, or Ubuntu, running as a guest in Virtualbox. That guest really believes it's a real machine, and I can shell into or out of it, apply updates, and so on, and when I've gotten it to a point I'm happy with (e.g., a particular operating system revision, or I'm ready to install a new piece of software that may hose up the guest), I can take a snapshot of the guest which I can then revert back to if something breaks.
This is pretty cool shit.
But can you see where this is going? In our datacenters, Sun is no longer really selling us SPARCs (that's Fujitsu's job these days I suppose); they're selling us Xeon and AMD based machines. Which means, for better or for worse, they, too, can run these magical guests. We can give them to our developers, we can use them as test environments, or, and this is the important part, we can apportion them out into production servers.
So, this is good technology, because it lets our developers be developers, and it lets our admins have one single point of administration (the big 16-way machine). On xVM, for example, we even have this great command-line interface where we can simply say VBoxManage metrics collect, and it shows us what is going on in the virtual machines running within the virtual environment without us having to log into the actual guests. We can of course save this data, use it for reporting, and to smack developers when they break something. But, speaking of smacking developers, remember they have guest machines on our one big host, and if they manage to let the smoke out of their machine, they only break that machine, and the remaining fifteen webservers are all happy, and we can throttle the one developer who broke his machine. Or, restart it, or revert to a snapshot, or whatever. I mean, this is nirvana, right?
Here's how I have things set up:
As you can see, I have disks in their own directory, the ISO images they boot off of, and their snapshots saved in their own separate directory. This is on a separate disk (e.g., not the root volume), which is mirrored in case one of my disks tanks, and in theory gives me a little bit of round-robin read performance. But I'm not too worried about that.
Here's where it gets complicated.
VMWare (owned now by EMC, who doesn't exactly produce anything on the cheap) is kind of an industry darling because they were first to market with an enterprise solution. They have a product that runs on MacOS, on Linux, and on Solaris. They can do anything that xVM and VirtualBox can do (except, as far as I can tell, automatically compile their own kernel module, which is accomplished by vboxdrv on Ubuntu; this means I can switch to a realtime kernel and VirtualBox just keeps a-workin'). So lots of shops have adopted VMWare and their associated costs.
If you think about this in the worst terms, if you have a VMWare solution (and, to be honest, I do like the VMWare 2 engine...), and you're running it on Solaris x86, you can't call Sun and ask them to fix your virtualization problem, and VMWare might just tell you it's a Solaris problem and, frankly, they would rather you ran on Linux, or whatever strikes their fancy at the moment.
But this isn't the point of this diatribe, because we don't all run Solaris, or Linux. The point is more subtle. In addition to the VMWare and VirtualBox products, we have the Virtualbox-OSE package, which is open-source and supported by the Debian community (and presumably by Sun engineers in their off-hours). There is also a virtualization product called Xen, which is more like a sort of glorified netbooting, but all the same provides a way to run guests on hosts (although not monolithically like the rest of the products). And of course there's KVM, the kernel virtual machine, which has a novel approach to virtualization:
Basically, while KVM provides the so-called "hypervisor" (think of a hypervisor as a sort of "super-kernel" that's able to run "slave kernels" much the way VMWare and xVM use the processor extensions as a pass-through to the iron), it still relies on the system kernel. I'm not going to get into the security vulnerabilities or other problems with this model, other than to say it is probably the most innovative way of going about things, but also the most fragile.
Now, I've been nattering on for quite a long time, explaining the various approaches to virtualization, but I haven't gotten to the meat of things, and I haven't discussed the confusing messages coming from Sun with respect to virtualization. Sun, starting with 5.10, and being released into 5.11, has what they call zones. Now, my virtualization experience is largely centered around the above-mentioned virtualization solutions – that is, a monolithic guest that is run by a hypervisor process which may or may not just pass the bytecode through to the iron.
The Solaris Zone is a little bit of a different approach. I've always been one to keep /usr mounted read-only, leaving /usr/local mounted read-write (Ubuntu broke me of this convention, something I'm still kind of sore at them about). With a Solaris zone, you can say, I'd like to create an alias on eri0, let's call it eri0:1, and I want to give that the last octet of .221. Let's call eri0 .200. I can then say that eri0:1 is now attached to a new zone (the details of this are abstracted by a set of commands dealing specifically with zones, as we'd expect from Sun). If we want to be pedantic about it, we can call .200 "master" and .221 "slave." But, while Sun will tell you that zones are virtualization, they share little in common with the other virtualization products on the market. Your zones, for example, will likely share /usr (which is probably a good thing), but it's going to be immutable so that one zone (say, slave2, at .222) can't clobber /usr/lib for the other zones.
But what have we accomplished by doing this? We can use commands like zoneadm to peek in on our zones, but we don't quite have the introspection we might have from a command like VBoxManage. We literally have to "shell into" the zone to have a look at what it's doing. But one thing Sun doesn't seem to be advertising that zones offer which monolithic virtual machines do not offer is standardization.
Let's say I'm a really responsible systems operator and I have installed a bunch of system v packages in /opt/site. Let's say that not only am I really responsible, but I'm kind of a fascist, and I want to make sure that the developers use those libraries rather than installing their own stuff in their own zones. I can give each developer, in their own zone, their own writeable /var directory (which is traditionally for runtime files anyways, right?), and of course their own home directory. But in the earlier case of having sixty developers (or however many; the principle of zones scales pretty far), we can keep /opt/site immutable, just like we keep /usr, and the developers are going to install their software in the writable places like /usr/local/appname, and they're going to throw their e.g., apache configurations in /var/httpd/conf.d/appname.conf, and as a systems administrator, I know where their configurations are, I don't have to worry about fifteen different versions of apache installed, or one application linked to some library that lives in ~joebob/lib/.
This, to me, strikes me as a tremendous win. If you can get all your developers on the same page, and say, hey, guys, we're all going to be developing from the /opt/site packages, and you all have your own zones, and you can install your software in /usr/local/appname, and use the current convention of installing your apache fragment in the conf.d directory, we'll all get along great, and when I get that 0300 call, I don't have to spend two hours reverse-engineering your application.
There's another benefit to this, if you can believe that. By encouraging developers to all code to a specific set of libraries and paths, you can create automated deploy scripts (in perl land we call this the Makefile.PL, python has its own conventions, ruby its own, and of course there's the Unix convention of having a configure script and a Makefile.) You can even detect if the libraries you expect to be there are there, and complain if they're not. Why is this useful? Well, because you're not always going to have me installing your software.
The admin who's never seen your software before is going to have a hell of a time installing it if he has to go download and install fifteen different pieces of software before he actually gets to installing your software, and that's assuming it even works when it's installed. By coding against standard conventions, paths, libraries, and having scripts that check our environment before hand, we ensure that whether the admin is trying to install our software on IRIX, Darwin, AIX, Solaris, or Linux, we can be reasonably assured that it will work.
But back to these zones. From the above, I think we can assume that zones are a good thing. In fact, they're a very good thing. But you're going to run into one problem. zoneadm lets you "clone" a zone, using a zfs snapshot, but this is not useful to anyone who isn't running zfs, nor is it useful to anyone who isn't running Solaris.
So, what then, do we do when we want our developers to write software we like, and we want to be able to move virtual machines, zones, client kernels, and so on, around? Unfortunately, it looks like the only solution is to use the monolithic-guest applications, like VMWare and Virtualbox. The good news is both these pieces of software have tremendous tools for sysadmins, and their footprint is actually fairly small (an xml file and a "virtual disk" file). Since the files are self-contained, provided you're not crossing bitness (e.g., moving a 64-bit guest to a 32-bit host), you should be able to just pick the image up, move it to the new host (or hand it to a customer, another department, or even another developer) and it should Just Work.
The problem with this approach is that it lacks the benefits I describe with zones. There is a solution, but it's a little bit of a pain in the ass. If you have an environment you like working on – let's just say Solaris 5.10 as that's kind of a recurring theme today, you build a virtual Solaris 5.10 machine you like, take a snapshot of that, and then clone it, passing snapshots out to your users, developers, or whomever, as necessary.
The obvious drawback, compared to zones, KVM, and Xen is that you're going to be stuck with the kernel you rolled into that image, along with all those packages. So each of those snapshot virtual machines you handed out, you're going to have to update manually or come up with an infrastructure for updating them. Now, that having been said, there's no reason you can't (and this is really thinking out of the box) create a virtual machine, a monolithic virtual machine, that netboots, picking up the appropriate kernel, packages via NFS/SMB/AFS/whatever, and is as up-to-date as needs be.
The reason I've hashed all this out, if you hadn't guessed, is I'm in the process of building this environment at work. We're going to hash out the virtualization solution we like, and we're going to be stuck with it, probably for at least a few years. Unfortunately, none of them really has all the features we/I want, and they all have drawbacks that are difficult to address.
But to ignore virtualization because it has a few niggling issue is veritably spitting in the wind. The tide is changing, folks, and if you don't get moving in that direction, you're going to find that you're missing very important skills as sysadmins in a couple years.
Hell, even developers are going to be expected to understand virtual machines and zones in the next couple years if not already.
Best of luck... and if you've got some magic solution, drop me a line.
It's no big secret that I'm a fan of Virtualbox and xVM. Before the unholy alliance of Sun and Oracle, it seemed to me that Sun had VMWare beat with a better product, a better filesystem (ufs had snapshots, and zfs has even better snapshots). But the waters have been muddied somewhat, not only at Sun, but among the Linuxes, as well as in the IBM camp, and the various vendors offering virtualization solutions (including Microsoft).So, what are we trying to accomplish with virtualization? I see a couple of tasks. If we look at the image above, we see the fairly vanilla use of virtualization: I've got various hosts I use for testing out various things, and in the case of work I need a Windows XP machine to (yes, it sucks, yes, I know, but I've worked around it) fill out my weekly timecards. Note that this is VirtualBox running on my MacPro at home.
Another purpose of virtualization is sort of like the one I have described above, only it works on a larger scale. Let's say you're a development shop and you've got sixty developers. Rather than buy each and every one of them their own development workstation, or expect them to develop on their laptops or desktops, we can buy one bigass machine, run a virtualization host, and "carve off slices" for each of these developers so that they are free to break their development machine. Furthermore, they can take snapshots, and if their machine is so irretrievably broken they can't recover it, a sysadmin might take pity on them and simply carve them off a new virtual machine.
There are two resource constraints with the above model (the one I describe prior to this is handled by a MacPro with 12gb of ram and about 4tb of disk). First, it needs to have enough processor capacity to deal with whatever the developers want to do. Developers have bad habits like running make -j, or telling their database it can allocate all the ram that /etc/system will let it have. So rest assured, your "Virtual Machine Host" is going to need to have multiple CPU's and the current standard seems to be a four-by-four-core Opteron or Xeon machine, and anywhere from sixteen to sixty-four gigs of ram. Remember, each of these users is going to need a few gigs of ram as well. Your mileage may vary, and you may decide that your developers don't get to have a few gigs; they'll just have to make do with one. And, really, Solaris and Linux can get along just fine in less than a gig if nothing exotic is going on.
A third model for virtualization exists, of course, and this is the one where all the money is. Long ago, us grey-beard Sparc admins realized that we really wanted E3500's to run Oracle, but for the most part, the load on the machine never really got above 1, and sometimes it might get as high as 3 or 4. For a machine with four CPU's, this isn't a big deal, and half the time this wasn't because Oracle was pegging the CPU as such, but rather because it was waiting for the SAN to come back with the data it wanted for a given query it was executing (this is called iowait). There were also other niggling details like waiting on swap (kernels, back in the day, had a habit of swapping stuff out even when they didn't really need to; there's a huge discussion about this on the LKML, and even RedHat will tell you to use swap, even when you have tons of RAM) that would raise the system load, even though the processors might actually be idle.
It is important for this reason to understand what the actual meaning of "system load" is in terms of Unix. For a really, seriously, in-depth discussion of it, albeit a bit out of date, there's the Marlin book from O'Reilly. It's a great book, and is probably required reading for anyone who wants to read any further than we're going to go right here.
So we old farts decided, hey, most of the time, our systems are idle. And yet, we have datacenters full of 280R's and E4500's and E3500's and we've got racks and racks of A5200's and yet, most of the time, they're just drawing power, creating heat, costing us money by way of support contracts and failing components (disks, power supplies, cpu's, dimms, etc). Why is this?
Someone – whose name I do not know – got it into their head that, if these systems spend most of their time idle, why not segment the machines into multiple virtual machines? If we take the example of my Mac, I have eight cpu's, and I'm rarely using more than four (between my wife, myself, and iTunes, we really do use four). What about those other four cpu's? What about that extra four gigs of ram that I'm not using? Indeed, the machine is idle. The load may well be 4.0, but by and large, the system is healthy.
So, VirtualBox comes along (although, I think to be fair, VMWare Fusion was first on the Mac, and I don't really know anything about Parallels, but it's been around a while, too; both of these sort of made Boot Camp obsolete), and it says, well, let's carve off sections of disk, and we'll treat them just like they were hard disks themselves. We can mount ISO images (or physical OS media) as though they were in the CDROM of this virtual machine, and we can install an operating system image onto that file that's pretending to be a disk. What the virtualization software in this case does is it lets the "guest" operating system run its own code in a virtual machine (hence the name), and both Intel and AMD have made a sort of "pass-through" option on their CPUs so that it's not even really necessary to emulate – the virtualization software can just say "hey, I'm running software on top of the OS, please just run this guest OS right on the iron, if you would," and the proc complies.
What we get, then, is myself logged in and doing evil things with postgres, mail.app, photoshop, and handbrake, my wife logged in looking at the latest news and downloading Bleach torrents, and our iTunes user logged in serving up content to the AppleTV. BUT, we also have an instance of Windows, or Solaris, or Nexenta, CAOS, or Ubuntu, running as a guest in Virtualbox. That guest really believes it's a real machine, and I can shell into or out of it, apply updates, and so on, and when I've gotten it to a point I'm happy with (e.g., a particular operating system revision, or I'm ready to install a new piece of software that may hose up the guest), I can take a snapshot of the guest which I can then revert back to if something breaks.
This is pretty cool shit.
But can you see where this is going? In our datacenters, Sun is no longer really selling us SPARCs (that's Fujitsu's job these days I suppose); they're selling us Xeon and AMD based machines. Which means, for better or for worse, they, too, can run these magical guests. We can give them to our developers, we can use them as test environments, or, and this is the important part, we can apportion them out into production servers.
Let's say I have a big 16-way machine with 64gb of ram and a petabyte of disk (and this is really not so far fetched anymore, is it?). We all know that the load on most webservers is approximately 0, but our developers have decided that they really want to keep their applications separate, so what we do is we create sixteen virtual machines, all running apache or nginx or lighttpd for them, and let them go to town. And instead of sixteen 1U machines sitting in the rack, we have a single 4U machine, and maybe we have a 3U disk array underneath it.
So, this is good technology, because it lets our developers be developers, and it lets our admins have one single point of administration (the big 16-way machine). On xVM, for example, we even have this great command-line interface where we can simply say VBoxManage metrics collect, and it shows us what is going on in the virtual machines running within the virtual environment without us having to log into the actual guests. We can of course save this data, use it for reporting, and to smack developers when they break something. But, speaking of smacking developers, remember they have guest machines on our one big host, and if they manage to let the smoke out of their machine, they only break that machine, and the remaining fifteen webservers are all happy, and we can throttle the one developer who broke his machine. Or, restart it, or revert to a snapshot, or whatever. I mean, this is nirvana, right?
Here's how I have things set up:
thunder% find Disks Isos Snaps -type f | grep -v DS
Disks/caos-dvldev.vdi
Disks/chimchim.vdi
Disks/laserwolf.vdi
Disks/needles.vdi
Isos/XP.iso
Isos/caos-nsa-1.0.i386.iso
Isos/caos-nsa-1.0.x86_64.iso
Isos/chaos-0.7.iso
Isos/chaos-1.5.iso
Isos/chaos-1.6.iso
Isos/kubuntu-8.10-dvd-amd64.iso
Isos/kubuntu-9.04-beta-desktop-i386.iso
Isos/sol-10-u6-ga1-x86-dvd.iso
Snaps/{22decce8-35d4-466c-92bc-b853b9ff3b7e}.vdi
Snaps/{36dcd85b-b133-4f56-83de-3f5f5c6778a7}.sav
Snaps/{9d839033-02df-4b0f-a7f2-b052f99dbe8a}.vdi
Snaps/{e166da89-904f-438c-9cee-624a0f33d4e0}.sav
Snaps/{eedd4c80-e4e3-47c0-b416-b0dca5d15b39}.vdi
Snaps/{fcb9b02a-74ae-43e4-94e7-d8dd15804929}.vdi
Disks/caos-dvldev.vdi
Disks/chimchim.vdi
Disks/laserwolf.vdi
Disks/needles.vdi
Isos/XP.iso
Isos/caos-nsa-1.0.i386.iso
Isos/caos-nsa-1.0.x86_64.iso
Isos/chaos-0.7.iso
Isos/chaos-1.5.iso
Isos/chaos-1.6.iso
Isos/kubuntu-8.10-dvd-amd64.iso
Isos/kubuntu-9.04-beta-desktop-i386.iso
Isos/sol-10-u6-ga1-x86-dvd.iso
Snaps/{22decce8-35d4-466c-92bc-b853b9ff3b7e}.vdi
Snaps/{36dcd85b-b133-4f56-83de-3f5f5c6778a7}.sav
Snaps/{9d839033-02df-4b0f-a7f2-b052f99dbe8a}.vdi
Snaps/{e166da89-904f-438c-9cee-624a0f33d4e0}.sav
Snaps/{eedd4c80-e4e3-47c0-b416-b0dca5d15b39}.vdi
Snaps/{fcb9b02a-74ae-43e4-94e7-d8dd15804929}.vdi
As you can see, I have disks in their own directory, the ISO images they boot off of, and their snapshots saved in their own separate directory. This is on a separate disk (e.g., not the root volume), which is mirrored in case one of my disks tanks, and in theory gives me a little bit of round-robin read performance. But I'm not too worried about that.
Here's where it gets complicated.
VMWare (owned now by EMC, who doesn't exactly produce anything on the cheap) is kind of an industry darling because they were first to market with an enterprise solution. They have a product that runs on MacOS, on Linux, and on Solaris. They can do anything that xVM and VirtualBox can do (except, as far as I can tell, automatically compile their own kernel module, which is accomplished by vboxdrv on Ubuntu; this means I can switch to a realtime kernel and VirtualBox just keeps a-workin'). So lots of shops have adopted VMWare and their associated costs.
If you think about this in the worst terms, if you have a VMWare solution (and, to be honest, I do like the VMWare 2 engine...), and you're running it on Solaris x86, you can't call Sun and ask them to fix your virtualization problem, and VMWare might just tell you it's a Solaris problem and, frankly, they would rather you ran on Linux, or whatever strikes their fancy at the moment.
But this isn't the point of this diatribe, because we don't all run Solaris, or Linux. The point is more subtle. In addition to the VMWare and VirtualBox products, we have the Virtualbox-OSE package, which is open-source and supported by the Debian community (and presumably by Sun engineers in their off-hours). There is also a virtualization product called Xen, which is more like a sort of glorified netbooting, but all the same provides a way to run guests on hosts (although not monolithically like the rest of the products). And of course there's KVM, the kernel virtual machine, which has a novel approach to virtualization:
Basically, while KVM provides the so-called "hypervisor" (think of a hypervisor as a sort of "super-kernel" that's able to run "slave kernels" much the way VMWare and xVM use the processor extensions as a pass-through to the iron), it still relies on the system kernel. I'm not going to get into the security vulnerabilities or other problems with this model, other than to say it is probably the most innovative way of going about things, but also the most fragile.
Now, I've been nattering on for quite a long time, explaining the various approaches to virtualization, but I haven't gotten to the meat of things, and I haven't discussed the confusing messages coming from Sun with respect to virtualization. Sun, starting with 5.10, and being released into 5.11, has what they call zones. Now, my virtualization experience is largely centered around the above-mentioned virtualization solutions – that is, a monolithic guest that is run by a hypervisor process which may or may not just pass the bytecode through to the iron.
The Solaris Zone is a little bit of a different approach. I've always been one to keep /usr mounted read-only, leaving /usr/local mounted read-write (Ubuntu broke me of this convention, something I'm still kind of sore at them about). With a Solaris zone, you can say, I'd like to create an alias on eri0, let's call it eri0:1, and I want to give that the last octet of .221. Let's call eri0 .200. I can then say that eri0:1 is now attached to a new zone (the details of this are abstracted by a set of commands dealing specifically with zones, as we'd expect from Sun). If we want to be pedantic about it, we can call .200 "master" and .221 "slave." But, while Sun will tell you that zones are virtualization, they share little in common with the other virtualization products on the market. Your zones, for example, will likely share /usr (which is probably a good thing), but it's going to be immutable so that one zone (say, slave2, at .222) can't clobber /usr/lib for the other zones.
But what have we accomplished by doing this? We can use commands like zoneadm to peek in on our zones, but we don't quite have the introspection we might have from a command like VBoxManage. We literally have to "shell into" the zone to have a look at what it's doing. But one thing Sun doesn't seem to be advertising that zones offer which monolithic virtual machines do not offer is standardization.
Let's say I'm a really responsible systems operator and I have installed a bunch of system v packages in /opt/site. Let's say that not only am I really responsible, but I'm kind of a fascist, and I want to make sure that the developers use those libraries rather than installing their own stuff in their own zones. I can give each developer, in their own zone, their own writeable /var directory (which is traditionally for runtime files anyways, right?), and of course their own home directory. But in the earlier case of having sixty developers (or however many; the principle of zones scales pretty far), we can keep /opt/site immutable, just like we keep /usr, and the developers are going to install their software in the writable places like /usr/local/appname, and they're going to throw their e.g., apache configurations in /var/httpd/conf.d/appname.conf, and as a systems administrator, I know where their configurations are, I don't have to worry about fifteen different versions of apache installed, or one application linked to some library that lives in ~joebob/lib/.
This, to me, strikes me as a tremendous win. If you can get all your developers on the same page, and say, hey, guys, we're all going to be developing from the /opt/site packages, and you all have your own zones, and you can install your software in /usr/local/appname, and use the current convention of installing your apache fragment in the conf.d directory, we'll all get along great, and when I get that 0300 call, I don't have to spend two hours reverse-engineering your application.
There's another benefit to this, if you can believe that. By encouraging developers to all code to a specific set of libraries and paths, you can create automated deploy scripts (in perl land we call this the Makefile.PL, python has its own conventions, ruby its own, and of course there's the Unix convention of having a configure script and a Makefile.) You can even detect if the libraries you expect to be there are there, and complain if they're not. Why is this useful? Well, because you're not always going to have me installing your software.
The admin who's never seen your software before is going to have a hell of a time installing it if he has to go download and install fifteen different pieces of software before he actually gets to installing your software, and that's assuming it even works when it's installed. By coding against standard conventions, paths, libraries, and having scripts that check our environment before hand, we ensure that whether the admin is trying to install our software on IRIX, Darwin, AIX, Solaris, or Linux, we can be reasonably assured that it will work.
But back to these zones. From the above, I think we can assume that zones are a good thing. In fact, they're a very good thing. But you're going to run into one problem. zoneadm lets you "clone" a zone, using a zfs snapshot, but this is not useful to anyone who isn't running zfs, nor is it useful to anyone who isn't running Solaris.
So, what then, do we do when we want our developers to write software we like, and we want to be able to move virtual machines, zones, client kernels, and so on, around? Unfortunately, it looks like the only solution is to use the monolithic-guest applications, like VMWare and Virtualbox. The good news is both these pieces of software have tremendous tools for sysadmins, and their footprint is actually fairly small (an xml file and a "virtual disk" file). Since the files are self-contained, provided you're not crossing bitness (e.g., moving a 64-bit guest to a 32-bit host), you should be able to just pick the image up, move it to the new host (or hand it to a customer, another department, or even another developer) and it should Just Work.
The problem with this approach is that it lacks the benefits I describe with zones. There is a solution, but it's a little bit of a pain in the ass. If you have an environment you like working on – let's just say Solaris 5.10 as that's kind of a recurring theme today, you build a virtual Solaris 5.10 machine you like, take a snapshot of that, and then clone it, passing snapshots out to your users, developers, or whomever, as necessary.
The obvious drawback, compared to zones, KVM, and Xen is that you're going to be stuck with the kernel you rolled into that image, along with all those packages. So each of those snapshot virtual machines you handed out, you're going to have to update manually or come up with an infrastructure for updating them. Now, that having been said, there's no reason you can't (and this is really thinking out of the box) create a virtual machine, a monolithic virtual machine, that netboots, picking up the appropriate kernel, packages via NFS/SMB/AFS/whatever, and is as up-to-date as needs be.
The reason I've hashed all this out, if you hadn't guessed, is I'm in the process of building this environment at work. We're going to hash out the virtualization solution we like, and we're going to be stuck with it, probably for at least a few years. Unfortunately, none of them really has all the features we/I want, and they all have drawbacks that are difficult to address.
But to ignore virtualization because it has a few niggling issue is veritably spitting in the wind. The tide is changing, folks, and if you don't get moving in that direction, you're going to find that you're missing very important skills as sysadmins in a couple years.
Hell, even developers are going to be expected to understand virtual machines and zones in the next couple years if not already.
Best of luck... and if you've got some magic solution, drop me a line.
19 May, 2009
Fucking cagers
Why would somebody in an M5 (the previous M5, the V8) try so desperately to outrun a sportbike? I was just minding my own business when the guy came flying up on my rear, changed to the lane to my left, looked over, and then yanked in front of me way too close to comfort.
The only rational reaction to this is to just let the idiot do his thing. I wasn't feeling real rational, though, so I taped down two gears, yanked left, yanked right again in front of him, topped out one gear, then another, and was going so fast that the rear wheel came up when I went to stop at the next light. Mr. M5, Mr. I'm-so-fast-in-my-cage, pulled up next to me and pretended nothing had happened. The bike was doing well over 130mph, and my guess is he was struggling to make a hundred from his "start" of about 65mph.
Fucking cagers. Listen to me. Bikes are not cars. They are very fast. They do not like you. They are not impressed when you cut them off. And if they feel unsafe around you, they're going to blister the road until they get far enough away from you. So, please, keep your dick in your pants, your eyes on the road (and your fucking mirrors, you dolt), and maybe, please, keep within a good fifteen percent of the speed limit, and, please, please, pick a fucking lane and stay in it. We'll all be a lot safer, and you won't look like such a fucking tool.
The only rational reaction to this is to just let the idiot do his thing. I wasn't feeling real rational, though, so I taped down two gears, yanked left, yanked right again in front of him, topped out one gear, then another, and was going so fast that the rear wheel came up when I went to stop at the next light. Mr. M5, Mr. I'm-so-fast-in-my-cage, pulled up next to me and pretended nothing had happened. The bike was doing well over 130mph, and my guess is he was struggling to make a hundred from his "start" of about 65mph.
Fucking cagers. Listen to me. Bikes are not cars. They are very fast. They do not like you. They are not impressed when you cut them off. And if they feel unsafe around you, they're going to blister the road until they get far enough away from you. So, please, keep your dick in your pants, your eyes on the road (and your fucking mirrors, you dolt), and maybe, please, keep within a good fifteen percent of the speed limit, and, please, please, pick a fucking lane and stay in it. We'll all be a lot safer, and you won't look like such a fucking tool.