27 May, 2009
SHE RIDES THE HAMMER!
Pay no attention to the Crocs, dear reader! $wife fits the hammer, and I intend to ease her onto it and show her what a real terror of a bike is like. Which is to say, sort of like injecting cocaine. At least, that's what I'm told.
It just occurred to me
That some of the most rewarding things I've ever done, professionally as an engineer, have involved writing. I love writing documentation, specification files, creating policy, creating diagrams of architectures, and helping to explain these to people.
Maybe, in addition to my loving being a writer, this is the career change I've been looking for. Maybe there is hope.
Maybe, in addition to my loving being a writer, this is the career change I've been looking for. Maybe there is hope.
26 May, 2009
I sometimes wonder why I bother
Sure, writing good code is hard. Making big decisions about what you're going to do for the next 2-4 years is hard. Picking a vendor is hard. Teaching developers is hard. Writing meaningful diagrams is hard. Asking useful questions is hard.
But none of this is as hard as not having done any of it.
What is the point of keeping a wiki if nobody uses it? Why on earth would anyone bother researching products and vendors, technologies and organizations if only to throw their hands up and say "you know, it didn't work last time I tried..."?
I'm a serious, serious pessimist. I see failure everywhere. But the important difference between me and Mr. Joe-You're-Ugly is that where I see failure, I see an opportunity to fix something; to make something functional and beautiful. I see failure all around me, every day, from potholes, to laws, to people driving poorly, and, yes, in the workplace. I strive to fix these things, and people hate it. They feel like, once a shortcoming has been identified — even if you've sworn to help them make it better — that immediately, you're the enemy, and they'd rather clutch on to that failure and be proud of it because, dammit, it's their failure, and they worked hard on it.
It's not about blame, folks. It's about being better people. Let's work together and get past the petty bickering. We don't have to all hug and live in a group home in the Castro, but can we please look at the possibility of positive change as a good thing, rather than something terrifying?
If this happens to me again, people deciding they're scared of growth, I may well lose my faith in humanity all together. I'd really, if that's true, rather go back to missile guidance and warfare simulators. Killing people, at least, has well-defined objectives, you know when you're doing "better," and you know there are always more people to kill.
But none of this is as hard as not having done any of it.
What is the point of keeping a wiki if nobody uses it? Why on earth would anyone bother researching products and vendors, technologies and organizations if only to throw their hands up and say "you know, it didn't work last time I tried..."?
I'm a serious, serious pessimist. I see failure everywhere. But the important difference between me and Mr. Joe-You're-Ugly is that where I see failure, I see an opportunity to fix something; to make something functional and beautiful. I see failure all around me, every day, from potholes, to laws, to people driving poorly, and, yes, in the workplace. I strive to fix these things, and people hate it. They feel like, once a shortcoming has been identified — even if you've sworn to help them make it better — that immediately, you're the enemy, and they'd rather clutch on to that failure and be proud of it because, dammit, it's their failure, and they worked hard on it.
It's not about blame, folks. It's about being better people. Let's work together and get past the petty bickering. We don't have to all hug and live in a group home in the Castro, but can we please look at the possibility of positive change as a good thing, rather than something terrifying?
If this happens to me again, people deciding they're scared of growth, I may well lose my faith in humanity all together. I'd really, if that's true, rather go back to missile guidance and warfare simulators. Killing people, at least, has well-defined objectives, you know when you're doing "better," and you know there are always more people to kill.
25 May, 2009
Rolling what?
I guess most of the dicks had left town today, and we were welcomed at the only motorcycle shop open – a Harley shop in Fairfax. They didn't give us the filthy ricer look when we parked...
Yes, that sexy red thing in the front is my baby, the one behind that is the two fiddy, which $wife dutifully followed me on. Unfortunately, the trip we had planned, Route 50 to Lee Highway (29) to 211 out to Skyline Drive (which we refer to fondly as the Ebisu circuit after the cat-and-mouse chase in Initial D with the Sil-80 on the downhill), was hampered by rain. Me, I'll ride in anything — snow, salt, rain, ice, cold, hot, humid, hail, and so on. But $wife, following tandem with me on the two fiddy, hadn't ridden in this kind of stuff, and I began to worry for her safety, as traffic was becoming aggressive (I guess they wanted a piece of Ebisu, too).
So, instead of making it all the way out to Ebisu, we pulled over at Bull Run Battlefield Park, looked over at its soggy field (I really would have wanted to look around, but trying on the bikes in the gravel, mud, etc., was a recipe for disaster), and turned around to head home.
You might think we didn't get a ride in. We'd planned about two hundred miles of rides today, and instead got in about eighty. But we got a bunch of intangible benefits in:
If the Subaru needs an air filter or an oil change, I'm off to the dealer. Screw that noise. Cars... man, they're a fucking drag. I really, really wish the wife would let me sell the Subaru and get maybe a regular Rex or something like that. Maybe even a Mazdaspeed 3. Maybe I could get the Z all legal-like, and we'd have a car that was sort-of practical.
But, between buying her a BMW bike on the used market, my financing a new super-tourer, and wanting to do work on the Z, we're probably going to need to have equity in the stupid Subaru just to convince the bank they can steal our belongings if we default.
Choice quote of the day, by the way, from a neighbor, looking over the hammer
Guy then relates to me that he used to own a Harley, but that when it got to 120 or so, it hunted and didn't feel stable. I noted that the hammer has all kinds of fairings to keep it safe in the wind, and that the front suspensions on our respective bikes (inverted forks on mine, for example), are way, way, different. Poor guy. I just don't understand the Harley crowd.
So, instead of making it all the way out to Ebisu, we pulled over at Bull Run Battlefield Park, looked over at its soggy field (I really would have wanted to look around, but trying on the bikes in the gravel, mud, etc., was a recipe for disaster), and turned around to head home.
You might think we didn't get a ride in. We'd planned about two hundred miles of rides today, and instead got in about eighty. But we got a bunch of intangible benefits in:
- $wife got in eighty more miles of not riding to work. She gets more proficient every time she gets on the bike, and I love watching her. We can now actually ride tandem, as close as a foot or two from eachother, in the same lane, and stop at exactly the same spot at a light, every single time. This is different when we started riding together.
- I got to watch $wife ride, and saw that she has indeed improved tremendously (although I did notice she kind of leans off one side of the bike which might mean the suspension needs a little tweaking; this is hard — the 250 does not have an adjustable suspension). I did, however, notice that she needs a new chain. There's a lot of slack in that chain, and since she's been complaining about throttle slop, this may actually be take-up in the chain. I'm going to have a look and see if I can fix it with the tensioner, and if not, I may just replace the chain and sprockets (and might even sneak in a -1/+2 on her).
- She complained the brakes were a little squishy. Well, compared to the hammer, they are a little squishy, and I might have had a hard time following the hammer today on her bike. However, I've ridden that bike a couple thousand miles, and when you get used to the brakes, it's fine. I adjusted her lever so the pull is quicker, and I think I'm going to adjust the cable for her, so it's even snappier. I just don't want her to lock up the brake.
- The front brake rotor (not two!!) is glazed. I should probably replace the rotors and pads, bleed the system, and feed it new Motul. But, do I want to do this on a bike we're probably going to sell?
- The rear brake may be lagging a little, but pulls hard enough to stall the bike out if you're not careful, so it may just be time for a good brake service.
- She complained the transmission was kind of sloppy. Now, bear in mind the bike makes, literally, a hundred less horsepower than the hammer, so when I go to take the two fiddy out to see if its transmission is sloppy, I don't get the characteristic kick in the ass from first gear (on the hammer, you stomp down into first, and it kicks you back just as hard and the front wheel wants to rise on up). Now, her brake lever is adjustable, and adjusted as of today, but the clutch lever is not. So, I can adjust the clutch cable, which I haven't done before, but I have the factory service manual and all the tools, and, hell, I'm a competent mechanic. The other thing is, and you gotta understand motorcycles to understand this, the shift lever is adjustable for throw. You can adjust two nuts, one on the actual shift knob on the transmission (crank case?) and the other at the base of the shift lever. If you adjust both of these so that the connecting link is as short as it can be, the throw is consequently maybe a half an inch or an inch shorter, which is a lot when you are shift with a pedal. So, I think I'm gonna do that for her.
- The bike is filthy. I rode it out in the snow, salt, and rain, and the crankcase has all kinds of white filth on it, and everything is just dirty. It also needs a turn signal. So I'll order the signal at the dealer (I ordered flush signals, but they just look sheisty with their fake carbon fibre plastic, and they require dremeling the fascias, which I'm just not gonna do). So, I'm probably gonna simple green the crankcase, honda-polish and buff the rest of the bike, and clean the glass of her bubble (which is smoked, but alas...) and her mirrors. For this, I hope she will love me eternally.
- The other thing is we took both bikes out and made sure that the tire pressures were correct (in my case, I opted for a little more pressure; she's just about right), and we started feeding her premium gas. She'd been feeding it 87. The manual says this is okay, but this is before they started both oxygenating gas and adding 10-15% methanol to the pumps. So we go to the station that only does 10% instead of 15%, and with a tank of premium, her bike made more power (it was funny to watch her be surprised by the solid "thunk" of a shift from time to time today), doesn't smoke under wide open throttle, and generally runs better. Hell, at $6 a tank, and her getting 70mpg, it's not like we're breaking the bank.
- My bike's been overheating (or close to) in 395 traffic during rush hour. I picked up a bottle of Redline Water Wetter at the auto parts store, and I think I'm gonna execute a coolant flush. It being summer, I think I may just run 90/10 water and glycol (we do get down in the 40's sometimes, there's no telling if some demonic cold snap would bust open my block, so best to have at least a little glycol in there...). So, this is good.
If the Subaru needs an air filter or an oil change, I'm off to the dealer. Screw that noise. Cars... man, they're a fucking drag. I really, really wish the wife would let me sell the Subaru and get maybe a regular Rex or something like that. Maybe even a Mazdaspeed 3. Maybe I could get the Z all legal-like, and we'd have a car that was sort-of practical.
But, between buying her a BMW bike on the used market, my financing a new super-tourer, and wanting to do work on the Z, we're probably going to need to have equity in the stupid Subaru just to convince the bank they can steal our belongings if we default.
Choice quote of the day, by the way, from a neighbor, looking over the hammer
Say, how fast does that thing go?
Oh, about a hundred and eighty.
Really? Do you do that, like, on a track?
A track? Nah. 395. I've only ever had it up to 140 or so. It's like absolute power, you know, it corrupts absolutely. You just reach for a hand full of throttle, and at the top of fourth gear, you know the bike will take you all the way to the top of sixth, and you're going to be doing north of 170mph. It's absolutely irresistible. I can't stand cars anymore, and I've got two fast ones.
Guy then relates to me that he used to own a Harley, but that when it got to 120 or so, it hunted and didn't feel stable. I noted that the hammer has all kinds of fairings to keep it safe in the wind, and that the front suspensions on our respective bikes (inverted forks on mine, for example), are way, way, different. Poor guy. I just don't understand the Harley crowd.
Trips
$wife doesn't want to do the Yukon, even if I get her an R1150GS or R1300GS. Instead, I am planning a trip from DC to Seattle via Reno and the Gerlach, then back down through Portland to Santa Cruz, Monterey, hopefully with a stay at the Hyatt Highlands Inn, and then back west through Yosemite, and maybe back up through Montana or Yellowstone on the way home. I want to put 10-12,000 miles on the bike, and I want her to, also, so her skills get honed to the point where I feel safe taking her anywhere. Any bike she gets will likely be a boxer or a KLR, and have big old crashbars up front, protecting boxer cylinder heads, feet, and the bike in general. I'm real worried about taking her out on a shaft-drive bike, because if the shaft breaks, you're done. Chains and sprockets on the other hand, you're okay with. It's not fun to fix, but it's something you don't have to weld, and you can even do it in the field if you have a spare, which you can actually carry on a bike. (original credit for the shaft photo Doug Grosjean)Me, I love my ZX-7. The hammer is a splendiferous bike. But, it lacks any carriage other than what I can net to it, and I am not going to ride ten thousand frickin miles with everything on my back. For my part, I've been looking at Hayabusas (because they can have hard bags), Concours 14's, and that new BMW K1300GT (aforementioned shaft-drive problem acknowledged).
So here they are:
The Suzuki Hayabusa:
Although I really think I like some of the other color schemes a bit more.
All this really puts a cramp in my style, because ever since I heard it running around the Ascari track, I've wanted a Katoom. Behold, the RC8:
The good news is, Charley Boorman, minor celebrity, but a man of like-mind, and someone who believes people should get out and see the world (showing that people are all just people, and this whole business of killing eachother over which bit of brush you own is just outdated, retarded, and pointless), has a site for planning trips at Big Earth. Google Earth, for all its directions, satellite imagery, overlays, and all that, is not real good at multi-destination maps.
The important thing is that we're able to carry laptops with us (probably the Air and a PC), a GPS, paper maps, and some token clothing. GPS-wise, I am thinking of a Nokia N96 or N97, and of course $wife's BMW will come with its own GPS.
Then there's that whole "taking six weeks off work" thing. I can't believe that someone would fire me for wanting to do that, especially if I'd proven that I was a worthwhile member of the team. My wife's employer is not real thrilled by the idea of her being gone that long either. But, really, why would employers begrudge their employees the opportunity, no, necessity, of getting out, figuring out what you're made of, and seeing the world? Why consign us all to these concrete caves, cubicles, office chairs, and shackle us to computers? The computers will all be there when we get back, along with the chairs and offices, and even if I'm gone for six weeks, your company isn't going to fall apart. And furthermore, it's going to take six fucking weeks to hire somebody to replace either of us.
No, our respective employers would fire us out of spite, for having the audacity to think that we could go be human somewhere, instead of wage slaves. This, I surmise, and I may be proven wrong, but I doubt it.
24 May, 2009
And I'm done for the day.
I had a lot of stuff saved up; my comcast modem caught fire or something over the weekend, and instead of working on Puppet, I was offline reading CJ Cherryh. Worse things have happened, to be sure.
It is worth noting that the guy that came out was really solid, and when I told him I was under no circumstances rebooting any of my machines, he let me be a prick about it. Good show.
Fucking harleys. Bring on the rain.
(note: we have business service, so they'll come see us on a sunday. fat chance if you're just a residential puke)
It is worth noting that the guy that came out was really solid, and when I told him I was under no circumstances rebooting any of my machines, he let me be a prick about it. Good show.
Fucking harleys. Bring on the rain.
(note: we have business service, so they'll come see us on a sunday. fat chance if you're just a residential puke)
Rolling Thunder
Guys, you make me fucking sick. You don't wave. You don't wear proper gear. Some of you asshats aren't even wearing helmets. You parked your goddamn trailers all up and down my street, most of which were emblazoned with "ROLLING THUNDER" and variations of eagles and flags on the side. You think it's real cool to roll full-throttle onto first with your flatulent pipes at 0300 on route 1, which has a population of over 12,000 people. You come out here, to washington dc, from your dentist jobs, with the harleys you leave on a trickle charger for the rest of a year, and under a cover at that, and you bring your wife with you. But, heaven forbid she actually know how to ride a fucking motorcycle. You're all riding these enormous fucking harleys with big, cushy rear seats she's sitting in, waiting to get killed when you can't make a fucking u-turn in three lanes and you topple over.
You're impolite. You're here because it's a fucking circus. You're here congesting the roads and making lives for the residents unpleasant. And you haven't even got the decency to be respectful of the locals.
I washed my fucking bike today. I took it out and I lubed the chain, I checked the tires, and I got me a tank of gas. I figured I'd go out and see some of you harley fuckers, and maybe some of you wouldn't be the absolute pricks you were last year. But, indeed you were. To a one. Every single one of you bastards ignored me. I rode up behind a pair of you riding two-wide in a lane, and pulled into the lane to the right and looked over, and I guess I wasn't fucking good enough for you to even fucking acknowledge me.
Consider, if I'd passed you on one wheel and given you the finger, you'd have said something like "fucking ricer squid," and yet, when I ride, polite as can be, next to you, look over and smile, you're saying the same fucking thing.
Go retire, you decrepit fucks. Sell your harleys and buy RV's. Go tour the grand canyon. But you're not here for the Armed Forces, and I really, really, don't care if you fucking served. I work with people who serve presently and they concur you're a bunch of douchebags.
Get the fuck out of DC, or shape the fuck up. You all make me sick. I hope these thunder clouds break open and cover your cute little trailered million-pound choppers with the ugliest, coldest rain there is. I hope you all cower in the fucking hilton for the duration of your stay here. Me, I'd go out, rain or shine, just to enjoy being on two wheels, but I can't abide being near your unfortunate asses another second.
You disgust me.
You're impolite. You're here because it's a fucking circus. You're here congesting the roads and making lives for the residents unpleasant. And you haven't even got the decency to be respectful of the locals.
I washed my fucking bike today. I took it out and I lubed the chain, I checked the tires, and I got me a tank of gas. I figured I'd go out and see some of you harley fuckers, and maybe some of you wouldn't be the absolute pricks you were last year. But, indeed you were. To a one. Every single one of you bastards ignored me. I rode up behind a pair of you riding two-wide in a lane, and pulled into the lane to the right and looked over, and I guess I wasn't fucking good enough for you to even fucking acknowledge me.
Consider, if I'd passed you on one wheel and given you the finger, you'd have said something like "fucking ricer squid," and yet, when I ride, polite as can be, next to you, look over and smile, you're saying the same fucking thing.
Go retire, you decrepit fucks. Sell your harleys and buy RV's. Go tour the grand canyon. But you're not here for the Armed Forces, and I really, really, don't care if you fucking served. I work with people who serve presently and they concur you're a bunch of douchebags.
Get the fuck out of DC, or shape the fuck up. You all make me sick. I hope these thunder clouds break open and cover your cute little trailered million-pound choppers with the ugliest, coldest rain there is. I hope you all cower in the fucking hilton for the duration of your stay here. Me, I'd go out, rain or shine, just to enjoy being on two wheels, but I can't abide being near your unfortunate asses another second.
You disgust me.
Tell us about system load, Alex
Let's first look at this script:
Many people don't understand the difference between system load, and processor load. Earlier today, when I was running a Windows VM, a Solaris VM, handbrake, Visualhub, two instances of iTunes, deleting 65,000 files, and using Windows to unrar two 12-gb files segmented into 100-rar-file elements, I had the load well above twenty. And yet, if we look at the actual idle percentage of the CPU, we see that it's not doing a whole lot.
The reason for this is that you're waiting for data to move across the network (even if it is a loopback, host-only network, in the case of the VMs), you're waiting on disks to cough up their data and their reads/writes to finish, and, sadly, you're waiting for data to be pushed across RAM. Let me show you one of Apple's favorite documents:
When we look at the Intel design, we see one that is slightly more mature, in that there are fewer bottlenecks between devices, processors, memory, buses, and so on, but they have made the same mistakes. You're still going to struggle to get data from one side of the computer to the other.
This is what we call "iowait." Sometimes (like on Solaris), it's called iow, FreeBSD calls it something similar, and it is the enemy of computing. Because you paid a heck of a lot of money for those processors, and if they're sitting at 48% idle, why on earth did you spend all that money?
Sun's solution with the SPARC and with the newer Niagara processors and architecture has been to make machines so massively parallel and multi-linked (note: IBM is doing this too) and having such enormous bandwidth between components (Sun owns technology they purchased, jointly, with SGI, from Cray, which Cray at the time called CrayLink — it's sort of like God tapping you on the shoulder and saying "Oh, hai there, I got ur bits..."). But, even a Sun or an IBM SMP will choke on IO, and you'll wind up with a system that is fantastically loaded, and yet, at the same time, it is indeed idle.
The other demon that plagues us as systems administrators, when we're trying to figure out why our systems reporting software is saying "ZOMG THE LOAD IS EIGHT JIGGOWATTS" is "sys". Sys is a little more insidious. Let's say that it's the kernel's job to overcome all the failings of a given architecture, such as those indicated above (and let me be fair to Apple; those are very nice architectures, and for desktop- or commodity-level machines, it's pretty goddamn good to be "like an Octane." Anyone who can guess the price delta between an R12k Octane2 and a PowerMac of the same day gets extra credit). You're running (as we've previously talked about in the virtualization soliloquy), say, sixty webservers, and each of those webservers has maybe four thousand connections. It's easy to do the math here, and realize that you've got GOBS AND GOBS of open connections. The kernel, because it handles things like TCP (which, sadly, your web serving daemon does not), is frantically trying to assemble TCP packets, your firewall (also part of sys) is probably trying to either drop or reassemble fragmented packets, and it's trying to maintain state when you have a jillion open, half-open, talking, and closing connections.
Let me show you how this looks on my, fairly idle, desktop:
Now, my kernel is trying to maintain state on all of these connections, to make sure that people get the bits they want, that people who don't want to talk to us get their sockets closed, and that people who do want to talk to us get a new socket to listen on.
This takes up a lot of time. And unfortunately, there's nothing we can do about it, because we use computers to talk to people. The kernel spends an enormous amount of time just making sure we can do precisely that. So if you start experimenting with commands like iostat(1), vmstat(1), or sysctl on the mac, you're going to notice that a vast majority of the time, the system isn't actually crunching your transcode from matroshka to mp4. Instead, it's desperately waiting for that giant USB array you have attached to your computer to feed it the bits it needs to actually do that transcoding. Even if you have a really, really, fast processor.
Other things that impact sys are logical volume management (although with lufs, this is sort of changing), sar and other accounting processes, interprocess communication, and managing multiprocessor systems (maintaining things like processor affinity and process "nice"ness).
You can sort of reduce the load your system is going to spend on TCP by getting what's called a TCP Offload Engine or TOE, which is usually built into your NIC. Most companies aren't shipping these, but Alienware and other boutique vendors will ship you a "Extreme Gaming Network Card" that incorporates features of a TOE. If you're living in a cluster, you can get NICs with RDMA, and of course using ultrafast filesystems like myrinet and infiniband will also allow you to use your processor, rather than waiting around for your stupid disks to spin up, hit the right sector, or whatever.
So next time you're looking at every newbie's favorite command, uptime, and you see OMG THE LOAD IS SO HIGH, have a little deeper look, and figure out what your computer is doing, rather than assuming that you need something faster. An Ultra 80 with four UltraSPARC 450mhz cpu's is still a badass monster of a machine if you stuff it full of ram and 15krpm SCSI 320 drives. Take this to heart. Don't just go buy the fastest proc you can. Get the fastest disks you can, and attach them internally, or on a PCI card, and make sure they have the highest rotational speed you can find. Get the highest speed ram you can fit in your machine, and get it with fancy cooling fins so your bits move faster through the ram.
Your video card, trust me, is unimportant, unless you have a tesla, and if you have a tesla, you probably didn't even bother to read this far.
I'm real tired, though, of people worrying about system loads. I've seen system loads over a thousand. No shit, the load on a six-processor E3500 was creeping up from 970 to briefly over a thousand before it settled down. It was having a bad FCAL day. But those processors were idle, folks.
Computers are complicated, and that stuff you read on Ars and Toms Hardware, and BYTE Magazine and Accelerate Your Mac — it's all bullshit. Really. You need to learn you some engineering if you're going to tell me that your system is overloaded.
#!/bin/bash
# Developed on Darwin. This won't work on Solaris, and probably won't work on Linux, either.
echo -n "number of cores available: " ; \
sysctl hw.ncpu | cut -d: -f 2
echo -n "users on system: " ; \
ps xawu | cut -f 1 -d ' ' | sort -u | wc -l ;
echo -n "system load averages: " ; \
sysctl vm.loadavg | tr -d '{}' | cut -d: -f 2 ;
echo -n "distinct processes on system: "; \
ps xa | wc -l ;
echo -n "actual cpu idle percentage: " ; \
iostat 1 | head -3 | tail -1 | tr -s ' ' | cut -d ' ' -f 11
# aja // vim: ts=2 noet
We get an output that looks something like this:number of cores available: 8 users on system: 16 system load averages: 7.44 8.72 8.20 distinct processes on system: 136 actual cpu idle percentage: 48
Many people don't understand the difference between system load, and processor load. Earlier today, when I was running a Windows VM, a Solaris VM, handbrake, Visualhub, two instances of iTunes, deleting 65,000 files, and using Windows to unrar two 12-gb files segmented into 100-rar-file elements, I had the load well above twenty. And yet, if we look at the actual idle percentage of the CPU, we see that it's not doing a whole lot.
The reason for this is that you're waiting for data to move across the network (even if it is a loopback, host-only network, in the case of the VMs), you're waiting on disks to cough up their data and their reads/writes to finish, and, sadly, you're waiting for data to be pushed across RAM. Let me show you one of Apple's favorite documents:
And their earlier, much more SGI Octane-like G5,
We very quickly start to see the problems with this. With the G5 for starters, we see the disks and all the external buses, like 1394a/b, USB, Bluetooth, and even wireless, living off on their own controller. This is, incidentally, what caused so much heat on my Macbook that it self-immolated and burned traces off the board, melting the case, and resulting in my getting an Air (which, let's face it, doesn't really have an architecture so much as magic smoke that just works).
When we look at the Intel design, we see one that is slightly more mature, in that there are fewer bottlenecks between devices, processors, memory, buses, and so on, but they have made the same mistakes. You're still going to struggle to get data from one side of the computer to the other.
This is what we call "iowait." Sometimes (like on Solaris), it's called iow, FreeBSD calls it something similar, and it is the enemy of computing. Because you paid a heck of a lot of money for those processors, and if they're sitting at 48% idle, why on earth did you spend all that money?
Sun's solution with the SPARC and with the newer Niagara processors and architecture has been to make machines so massively parallel and multi-linked (note: IBM is doing this too) and having such enormous bandwidth between components (Sun owns technology they purchased, jointly, with SGI, from Cray, which Cray at the time called CrayLink — it's sort of like God tapping you on the shoulder and saying "Oh, hai there, I got ur bits..."). But, even a Sun or an IBM SMP will choke on IO, and you'll wind up with a system that is fantastically loaded, and yet, at the same time, it is indeed idle.
The other demon that plagues us as systems administrators, when we're trying to figure out why our systems reporting software is saying "ZOMG THE LOAD IS EIGHT JIGGOWATTS" is "sys". Sys is a little more insidious. Let's say that it's the kernel's job to overcome all the failings of a given architecture, such as those indicated above (and let me be fair to Apple; those are very nice architectures, and for desktop- or commodity-level machines, it's pretty goddamn good to be "like an Octane." Anyone who can guess the price delta between an R12k Octane2 and a PowerMac of the same day gets extra credit). You're running (as we've previously talked about in the virtualization soliloquy), say, sixty webservers, and each of those webservers has maybe four thousand connections. It's easy to do the math here, and realize that you've got GOBS AND GOBS of open connections. The kernel, because it handles things like TCP (which, sadly, your web serving daemon does not), is frantically trying to assemble TCP packets, your firewall (also part of sys) is probably trying to either drop or reassemble fragmented packets, and it's trying to maintain state when you have a jillion open, half-open, talking, and closing connections.
Let me show you how this looks on my, fairly idle, desktop:
thunder% netstat -an | grep -vi listen | grep -v dgram | grep -v stream Active Internet connections (including servers) Proto Recv-Q Send-Q Local Address Foreign Address (state) tcp4 0 124 10.1.10.140.49344 24.86.166.100.31224 ESTABLISHED tcp4 0 0 10.1.10.140.49343 74.125.47.19.443 ESTABLISHED tcp4 0 0 10.1.10.140.49341 78.180.207.90.6348 ESTABLISHED tcp4 0 26 10.1.10.140.49335 88.192.26.203.61845 ESTABLISHED tcp4 0 0 10.1.10.140.49332 116.49.123.10.11881 ESTABLISHED tcp4 0 0 10.1.10.140.49319 194.109.248.110.36921 ESTABLISHED tcp4 0 0 10.1.10.140.49316 82.161.104.88.46031 ESTABLISHED tcp4 0 19250 10.1.10.140.49314 60.241.73.188.25000 ESTABLISHED tcp4 0 36656 10.1.10.140.49312 190.212.10.170.64283 ESTABLISHED tcp4 0 0 10.1.10.140.49310 58.70.117.71.16499 ESTABLISHED tcp4 0 0 10.1.10.140.49309 194.109.248.110.36921 ESTABLISHED tcp4 0 733 10.1.10.140.49308 85.228.205.19.36917 ESTABLISHED tcp4 0 0 10.1.10.140.49296 58.70.117.71.16499 ESTABLISHED tcp4 0 0 10.1.10.140.49291 96.21.108.102.30965 ESTABLISHED tcp4 0 0 10.1.10.140.49287 85.186.177.129.27194 ESTABLISHED tcp4 0 19964 10.1.10.140.49272 83.8.19.193.27797 ESTABLISHED tcp4 0 0 10.1.10.140.49269 60.241.73.188.25000 ESTABLISHED tcp4 0 0 10.1.10.140.49268 65.23.200.241.24890 ESTABLISHED tcp4 0 0 10.1.10.140.49260 151.53.4.153.57150 ESTABLISHED tcp4 0 0 10.1.10.140.49255 151.53.4.153.57150 ESTABLISHED tcp4 0 291 10.1.10.140.49247 194.109.248.110.36921 LAST_ACK tcp4 0 0 10.1.10.140.49245 173.128.165.180.3724 ESTABLISHED tcp4 0 0 10.1.10.140.49244 81.70.142.212.55986 ESTABLISHED tcp4 0 0 10.1.10.140.49242 85.83.1.95.8900 ESTABLISHED tcp4 0 0 10.1.10.140.49240 24.68.36.126.39271 ESTABLISHED tcp4 0 0 10.1.10.140.49239 80.99.34.174.6881 ESTABLISHED tcp4 0 0 10.1.10.140.49228 86.198.85.136.40695 ESTABLISHED tcp4 0 51691 10.1.10.140.49226 85.225.232.241.60167 ESTABLISHED tcp4 0 0 10.1.10.140.49225 85.241.164.146.53034 ESTABLISHED tcp4 0 8 10.1.10.140.49224 79.136.62.72.64299 ESTABLISHED tcp4 0 0 10.1.10.140.49205 92.133.223.226.59472 LAST_ACK tcp4 0 0 10.1.10.140.49204 88.192.26.203.61845 ESTABLISHED tcp4 0 0 10.1.10.140.49203 79.17.234.126.45930 ESTABLISHED tcp4 0 0 10.1.10.140.49193 151.53.4.153.57150 ESTABLISHED tcp4 0 0 10.1.10.140.65530 65.23.200.241.24890 ESTABLISHED tcp4 0 0 10.1.10.140.65518 88.192.26.203.61845 ESTABLISHED tcp4 0 739 10.1.10.140.65501 78.180.218.159.6348 ESTABLISHED tcp4 0 299 10.1.10.140.65494 78.180.218.159.6348 ESTABLISHED tcp4 0 39204 10.1.10.140.65485 60.241.130.179.6501 ESTABLISHED tcp4 0 0 10.1.10.140.65478 200.75.235.203.46125 ESTABLISHED tcp4 0 385 10.1.10.140.65469 78.180.218.159.6348 ESTABLISHED tcp4 0 0 10.1.10.140.65439 82.161.104.88.46031 ESTABLISHED tcp4 0 0 10.1.10.140.65438 81.217.78.140.7777 ESTABLISHED tcp4 0 0 10.1.10.140.65432 79.17.234.126.45930 ESTABLISHED tcp4 0 0 10.1.10.140.65430 78.105.187.201.25321 ESTABLISHED tcp4 0 0 10.1.10.140.65425 74.125.93.191.80 ESTABLISHED tcp4 0 0 10.1.10.140.65422 67.172.251.35.62523 ESTABLISHED tcp4 0 0 10.1.10.140.65324 70.83.32.117.48156 ESTABLISHED tcp4 0 0 10.1.10.140.65305 78.105.187.201.25321 ESTABLISHED tcp4 0 12291 10.1.10.140.65254 60.241.73.188.25000 FIN_WAIT_1 tcp4 0 0 10.1.10.140.65172 88.192.26.203.61845 ESTABLISHED tcp4 0 65605 10.1.10.140.65154 91.153.113.50.56319 ESTABLISHED tcp4 0 0 10.1.10.140.65145 88.90.235.18.49153 ESTABLISHED tcp4 0 39116 10.1.10.140.65136 79.172.92.36.37531 ESTABLISHED tcp4 0 0 10.1.10.140.65132 60.241.73.188.25000 CLOSING tcp4 0 13477 10.1.10.140.65122 69.242.186.238.45696 ESTABLISHED tcp4 0 34912 10.1.10.140.65119 79.172.92.36.37531 ESTABLISHED tcp4 0 0 10.1.10.140.65053 85.134.20.25.47537 ESTABLISHED tcp4 0 0 10.1.10.140.65025 60.241.130.179.6501 ESTABLISHED tcp4 0 0 10.1.10.140.64921 58.182.194.24.31822 ESTABLISHED tcp4 0 31980 10.1.10.140.64874 92.133.223.226.59472 FIN_WAIT_1 tcp4 0 66608 10.1.10.140.3689 10.1.10.40.57528 ESTABLISHED tcp4 0 10486 10.1.10.140.64747 97.104.232.8.35450 ESTABLISHED tcp4 0 53 10.1.10.140.64708 190.212.10.170.64283 FIN_WAIT_1 tcp4 0 0 10.1.10.140.64700 85.241.164.146.53034 ESTABLISHED tcp4 0 0 10.1.10.140.64648 78.105.187.201.25321 ESTABLISHED tcp4 0 0 10.1.10.140.64594 82.161.104.88.46031 ESTABLISHED tcp4 0 0 10.1.10.140.64593 96.49.23.203.34162 ESTABLISHED tcp4 0 0 10.1.10.140.64587 77.41.89.217.60847 ESTABLISHED tcp4 0 6144 10.1.10.140.64583 98.14.25.190.24852 ESTABLISHED tcp4 0 0 10.1.10.140.64467 24.87.89.150.32828 ESTABLISHED tcp4 0 0 10.1.10.140.64431 173.128.165.180.3724 ESTABLISHED tcp4 0 0 10.1.10.140.64374 190.207.230.140.41352 ESTABLISHED tcp4 0 30489 10.1.10.140.64320 94.21.56.185.58172 ESTABLISHED tcp4 0 11406 10.1.10.140.64315 93.6.97.49.25583 ESTABLISHED tcp4 0 49208 10.1.10.140.64273 93.6.97.49.25583 ESTABLISHED tcp4 0 0 10.1.10.140.64112 94.66.9.233.63929 ESTABLISHED tcp4 0 40942 10.1.10.140.64034 83.57.55.155.52328 ESTABLISHED tcp4 0 0 10.1.10.140.63965 82.131.25.19.45060 ESTABLISHED tcp4 0 0 127.0.0.1.63827 127.0.0.1.63829 ESTABLISHED tcp4 0 0 127.0.0.1.63829 127.0.0.1.63827 ESTABLISHED tcp4 0 0 10.1.10.140.63650 213.93.230.16.51002 ESTABLISHED tcp4 0 32794 10.1.10.140.63354 220.233.193.86.31914 ESTABLISHED tcp4 0 61210 10.1.10.140.63334 173.48.24.137.52734 ESTABLISHED tcp4 0 52175 10.1.10.140.63327 173.48.24.137.52734 ESTABLISHED tcp4 0 27686 10.1.10.140.63004 95.222.204.238.55253 ESTABLISHED tcp4 0 64268 10.1.10.140.62840 58.70.117.71.16499 ESTABLISHED tcp4 0 43379 10.1.10.140.62805 76.10.186.223.34335 ESTABLISHED tcp4 0 31077 10.1.10.140.62644 85.166.87.17.37397 ESTABLISHED tcp4 0 49005 10.1.10.140.62396 85.166.87.17.37397 ESTABLISHED tcp4 0 20365 10.1.10.140.62256 208.107.112.237.65000 ESTABLISHED tcp4 0 14144 10.1.10.140.62246 118.166.46.152.27888 ESTABLISHED tcp4 0 66608 10.1.10.140.61869 72.229.126.193.51413 ESTABLISHED tcp4 0 8 10.1.10.140.61853 24.68.36.126.39271 FIN_WAIT_1 tcp4 0 15707 10.1.10.140.61834 78.115.126.26.58474 ESTABLISHED tcp4 0 63133 10.1.10.140.61718 72.198.3.164.10051 ESTABLISHED tcp4 0 13870 10.1.10.140.61674 84.29.95.34.51978 ESTABLISHED tcp4 0 34791 10.1.10.140.61557 85.222.85.23.61687 ESTABLISHED tcp4 0 11113 10.1.10.140.61360 69.249.68.41.13708 ESTABLISHED tcp4 0 32794 10.1.10.140.61229 85.142.54.17.19999 ESTABLISHED tcp4 0 51745 10.1.10.140.60973 60.241.130.179.6501 ESTABLISHED tcp4 0 66240 10.1.10.140.60969 90.178.189.57.18596 ESTABLISHED tcp4 0 1766 10.1.10.140.60913 85.186.35.54.55933 ESTABLISHED tcp4 0 20744 10.1.10.140.60690 91.140.99.154.41854 ESTABLISHED tcp4 0 65894 10.1.10.140.60664 58.70.117.71.16499 ESTABLISHED tcp4 0 55729 10.1.10.140.60619 58.70.117.71.16499 ESTABLISHED tcp4 0 30477 10.1.10.140.60552 79.172.92.36.37531 ESTABLISHED tcp4 0 65894 10.1.10.140.60449 116.49.123.10.11881 ESTABLISHED tcp4 0 61362 10.1.10.140.60193 83.247.44.159.59045 ESTABLISHED tcp4 0 0 10.1.10.140.60131 24.87.89.150.32828 ESTABLISHED tcp4 0 23400 10.1.10.140.60126 77.106.172.230.51897 ESTABLISHED tcp4 0 2810 10.1.10.140.60051 24.86.166.100.31224 ESTABLISHED tcp4 0 654 10.1.10.140.59486 84.29.95.34.51978 ESTABLISHED tcp4 0 0 127.0.0.1.389 127.0.0.1.59452 ESTABLISHED tcp4 0 0 127.0.0.1.59452 127.0.0.1.389 ESTABLISHED tcp4 0 64985 10.1.10.140.59451 72.81.236.231.15206 ESTABLISHED tcp4 0 0 10.1.10.140.59191 24.86.166.100.31224 ESTABLISHED tcp4 0 0 10.1.10.140.59190 24.86.166.100.31224 ESTABLISHED tcp4 0 32754 10.1.10.140.59172 76.65.69.240.46795 ESTABLISHED tcp4 0 35133 10.1.10.140.59142 76.10.186.223.34335 ESTABLISHED tcp4 0 0 10.1.10.140.58987 79.186.250.63.43609 ESTABLISHED tcp4 0 29150 10.1.10.140.58893 76.65.69.240.46795 ESTABLISHED tcp4 0 0 10.1.10.140.58858 88.192.26.203.61845 ESTABLISHED tcp4 0 7527 10.1.10.140.58783 60.241.73.188.25000 ESTABLISHED tcp4 0 25266 10.1.10.140.58781 213.46.222.187.45896 ESTABLISHED tcp4 0 29472 10.1.10.140.58704 76.190.208.111.57017 ESTABLISHED tcp4 0 16045 10.1.10.140.58558 60.241.130.179.6501 ESTABLISHED tcp4 0 0 10.1.10.140.4111 10.1.10.140.58528 ESTABLISHED tcp4 0 0 10.1.10.140.58528 10.1.10.140.4111 ESTABLISHED tcp4 0 0 173.8.15.237.4111 173.8.15.237.49176 ESTABLISHED tcp4 0 0 127.0.0.1.5347 127.0.0.1.49175 ESTABLISHED tcp4 0 0 127.0.0.1.49175 127.0.0.1.5347 ESTABLISHED tcp4 0 0 127.0.0.1.5347 127.0.0.1.49166 ESTABLISHED tcp4 0 0 127.0.0.1.49166 127.0.0.1.5347 ESTABLISHED tcp4 0 0 127.0.0.1.5347 127.0.0.1.49165 ESTABLISHED tcp4 0 0 127.0.0.1.49165 127.0.0.1.5347 ESTABLISHED tcp4 0 0 127.0.0.1.5347 127.0.0.1.49164 ESTABLISHED tcp4 0 0 127.0.0.1.49164 127.0.0.1.5347 ESTABLISHED tcp4 0 0 127.0.0.1.5347 127.0.0.1.49163 ESTABLISHED tcp4 0 0 127.0.0.1.49163 127.0.0.1.5347 ESTABLISHED tcp4 0 0 127.0.0.1.5347 127.0.0.1.49162 ESTABLISHED tcp4 0 0 127.0.0.1.49162 127.0.0.1.5347 ESTABLISHED tcp4 0 0 127.0.0.1.5347 127.0.0.1.49161 ESTABLISHED tcp4 0 0 127.0.0.1.49161 127.0.0.1.5347 ESTABLISHED tcp4 0 0 127.0.0.1.5347 127.0.0.1.49160 ESTABLISHED tcp4 0 0 127.0.0.1.49160 127.0.0.1.5347 ESTABLISHED tcp4 0 0 10.1.10.140.65516 24.175.70.220.56601 FIN_WAIT_2 tcp4 0 0 10.1.10.140.49337 10.1.10.140.139 TIME_WAIT tcp4 0 0 10.1.10.140.49238 74.125.47.19.443 TIME_WAIT
Now, my kernel is trying to maintain state on all of these connections, to make sure that people get the bits they want, that people who don't want to talk to us get their sockets closed, and that people who do want to talk to us get a new socket to listen on.
This takes up a lot of time. And unfortunately, there's nothing we can do about it, because we use computers to talk to people. The kernel spends an enormous amount of time just making sure we can do precisely that. So if you start experimenting with commands like iostat(1), vmstat(1), or sysctl on the mac, you're going to notice that a vast majority of the time, the system isn't actually crunching your transcode from matroshka to mp4. Instead, it's desperately waiting for that giant USB array you have attached to your computer to feed it the bits it needs to actually do that transcoding. Even if you have a really, really, fast processor.
Other things that impact sys are logical volume management (although with lufs, this is sort of changing), sar and other accounting processes, interprocess communication, and managing multiprocessor systems (maintaining things like processor affinity and process "nice"ness).
You can sort of reduce the load your system is going to spend on TCP by getting what's called a TCP Offload Engine or TOE, which is usually built into your NIC. Most companies aren't shipping these, but Alienware and other boutique vendors will ship you a "Extreme Gaming Network Card" that incorporates features of a TOE. If you're living in a cluster, you can get NICs with RDMA, and of course using ultrafast filesystems like myrinet and infiniband will also allow you to use your processor, rather than waiting around for your stupid disks to spin up, hit the right sector, or whatever.
So next time you're looking at every newbie's favorite command, uptime, and you see OMG THE LOAD IS SO HIGH, have a little deeper look, and figure out what your computer is doing, rather than assuming that you need something faster. An Ultra 80 with four UltraSPARC 450mhz cpu's is still a badass monster of a machine if you stuff it full of ram and 15krpm SCSI 320 drives. Take this to heart. Don't just go buy the fastest proc you can. Get the fastest disks you can, and attach them internally, or on a PCI card, and make sure they have the highest rotational speed you can find. Get the highest speed ram you can fit in your machine, and get it with fancy cooling fins so your bits move faster through the ram.
Your video card, trust me, is unimportant, unless you have a tesla, and if you have a tesla, you probably didn't even bother to read this far.
I'm real tired, though, of people worrying about system loads. I've seen system loads over a thousand. No shit, the load on a six-processor E3500 was creeping up from 970 to briefly over a thousand before it settled down. It was having a bad FCAL day. But those processors were idle, folks.
Computers are complicated, and that stuff you read on Ars and Toms Hardware, and BYTE Magazine and Accelerate Your Mac — it's all bullshit. Really. You need to learn you some engineering if you're going to tell me that your system is overloaded.
running nginx on OSX
I run Leopard Server, and I have no need for Apache 1.3, Apache 2.x, Apple's wikid, their calendar server, and a bunch of other things that are all fixated on Apache and its configurations in launchd. Right now would be a good time for you to familiarize yourself with the launchd and launchctl man pages, although the referenced ("See Also") command, launch, does not exist.
I've written an nginx launchd configuration file which happily starts and stops on command, starts on boot of course, and has the required configurations. nginx, in general, is a simpler, better written, faster httpd than apache is (which is, after all, a patchy web server). It follows:
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>Label</key>
<string>nginx</string>
<key>Program</key>
<string>/usr/local/sbin/nginx</string>
<key>KeepAlive</key>
<true/>
<key>NetworkState</key>
<true/>
<key>StandardErrorPath</key>
<string>/var/log/system.log</string>
<key>LaunchOnlyOnce</key>
<true/>
</dict>
</plist>
You will want to place this in /System/Library/LaunchDaemons/nginx.plist, and it is helpful to issue "launchctl load -F /System/Library/LaunchDaemons/nginx.plist", after which it should behave correctly. Of course, the corresponding unload command is simply "unload", with or without the -F. Note that this launches nginx on boot, rather than when you choose it to. It also tells it to log stderr to syslog, which may not be to your preference (nginx likes to log to /usr/local/log/nginx.err). But I think you get the idea.
I've written an nginx launchd configuration file which happily starts and stops on command, starts on boot of course, and has the required configurations. nginx, in general, is a simpler, better written, faster httpd than apache is (which is, after all, a patchy web server). It follows:
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>Label</key>
<string>nginx</string>
<key>Program</key>
<string>/usr/local/sbin/nginx</string>
<key>KeepAlive</key>
<true/>
<key>NetworkState</key>
<true/>
<key>StandardErrorPath</key>
<string>/var/log/system.log</string>
<key>LaunchOnlyOnce</key>
<true/>
</dict>
</plist>
You will want to place this in /System/Library/LaunchDaemons/nginx.plist, and it is helpful to issue "launchctl load -F /System/Library/LaunchDaemons/nginx.plist", after which it should behave correctly. Of course, the corresponding unload command is simply "unload", with or without the -F. Note that this launches nginx on boot, rather than when you choose it to. It also tells it to log stderr to syslog, which may not be to your preference (nginx likes to log to /usr/local/log/nginx.err). But I think you get the idea.