A patch run on one of our production clusters resulted in one of the applications refusing to start. After much head-scratching and incrementation of logging verbosity, it was discovered that the data files for this software had expired on Feb 14th. It appears the software in question only checks the expiry dates when it starts, so the reboot after the patching caught us out. Calls to the software vendor revealed that they'd been sending us old copies of the data and they'd have to snail-mail us a new CD with the latest files to get us up and running again, leaving us without a fairly critical piece of infrastructure for a few days.

Yeah, not on my watch. Running the software through 'truss' (the Solaris 'strace' equivalent, if you're familiar with Linux) showed a number of 'time()' calls at startup, so a plan was hatched to unashamedly lie to the software.

A fake 'time()' function was thrown together to return the Unix-time for 2011/02/13-23:59:59 (a second before the data expiry) regardless of the current time:

int time() {
        return 1297641599;
};

$ cc -G -o time.so time.c

$ LD_PRELOAD=./time.so ./server-binary

Bingo. Surprisingly, the software doesn't seem to mind too much that time's always 1 second to midnight on Feb 13th. The datestamps in the application logs are royally messed up, but we're back up and running.

Good Time(s).

All of the interviewers had a master list of scenarios and questions (which I managed to get a fairly good look at during one of the face-to-face interviews - my Special Skill is reading upside-down), but I'm not going to reveal the details of any of the questions. I have a feeling the list doesn't change all that often.

Here we go.

If you're not already on LinkedIn, get yourself on there and add some colleagues. Join the various groups that are relevant to your mad skills. In my case, I'm fairly sure I was picked out of a 'Unix Professionals in the UK' type group. LinkedIn seems to be the One and Only place that recruiters use to find people for decent Unix jobs at the moment - I'm averaging about 2 recruiters a week looking for Unix people, even in the currently horrific job market.

If you're lucky, a recruiter from Google will send you a LinkedIn message which you should reply to with an only slightly restrained email explaining that in fact this is the best thing that has ever happened to you, and that you'd absolutely love to get in on the action.
And so it begins.

There will be a phone screen where your recruiter will ask you some simple, closed, technical questions. As far as I can tell, the recruiters don't have the answers to these questions, so there's no point in haggling or second guessing. Don't sweat it too much - I know I got a few details wrong, but still got through. I think they're basically weeding out the idiots and mental cases at this point - if you're not an idiot or mental, you should be OK.

After the phone screen, you should get an email asking you to rate your afore mentioned mad skills in 11 different fields:

• TCP/IP Networking (OSI stack, DNS, etc.)
• Unix/Linux System Administration tasks
• Unix/Linux internals
• Algorithms & Data Structures
• SQL and / or Database administration
• C
• C++
• Java
• Perl
• Shell Scripting (sh, bash, ksk, csh)
• Python

So begins the phone interviews. My first interview involved a couple of a couple of fairly simple Unix admin problem scenarios, both of which were answered with a series of commands to run, and the reasons for using those commands. If you've been doing any sort of Unix admin for any length of time, these shouldn't be a problem. There was also a more involved discussion about a standard piece of software and how it works. I wasn't particularly familiar with the way this software works, but I got through by suggesting ways which I thought the software might operate, and the sorts of circumstances under which the software might have problems, assuming it worked the way I thought it worked.

The second phone interview involved some network knowledge, and a discussion on another piece of standard Unix software. Again, I wasn't entirely sure of the finer details of how this software works, but again some educated guesses and explanations of why I thought the software works the way it works was enough. There was also a scripting exercise which on the face of it looked simple, but which was a bit tricky to figure out on-the-fly. During this part, you'll be typing into a Google Docs word document so the interviewer can watch you type. This is actually quite cool - the interviewer gets to watch as you assemble a solution, so even if the resulting script isn't quite right they can see what you're thinking as you hack out an answer.

If at any point during the phone interviews you feel like running 'apropos', that's up to you. Type quietly, though.

So far so good. If you've made the cut, Google will invite you round to theirs for lunch.
Lunch, and 5 one-on-one interviews.

I'm fairly sure I signed an NDA when I arrived at the Chocolate Factory, so I can't really tell you much about the on-site stuff. A few pointers, though:
1: If you don't know your shit cold, you aren't getting through.
2: Learn every scripting trick out there; you're going to need them.
3: Have the pizza for lunch. It's good.
Contrary to popular belief there aren't any questions about how many golf balls fit in a bus, or how many piano tuners there are in Chicago, but you may have to pull estimates out of your arse involving very large amounts of data and very large numbers of users, so these sorts of questions are more than slightly relevant. Apart from that, it's all technical. No soft-skills or HR bullshit. No messing about.

Sadly, I royally ballsed up one of the interviews, so The Panel (quite rightly) deemed me unworthy of a position.

Arse.

Should be enough to put 4 idiots into respiratory arrest.

You'll need:

• 80g ginger
• 6 green chillies
• 3 bird-eye chillies
• 4 fat cloves of garlic
• A tin of peeled plum tomatoes
• A fat onion
• 4 tbsp ground nut oil
• 2 tsp cumin
• 1 tsp salt
• 1/2 tsp chilli powder
• 1/2 tsp cayenne pepper
• 2 tsp turmeric
• 2 tsp garam masala
• 1 1/2 tbsp tomato puree

Shove the ginger, all the chillies (seeds included - no messing about) and garlic in a blender and blend to a paste. Transfer that dangerous shit into a bowl before it dissolves the blender blades.
Drain the tinned tomatoes and stick them in the blender. Give the button a few hits - you want a bit of texture left. Put the resulting slop in the bowl with the previously blended stuff.
Chop your onion.
Grab a pot and throw in the chopped onion and oil. Fry those tasty bits of onion fairly slowly 'til they're soft. Chuck the cumin in and stir, then give it a bit more frying time.
Add the chillies/tomato/etc to the onion and stir until it's venting noxious fumes. Keep stirring and add the salt, chilli powder, cayenne, turmeric and garam masala, then the tomato puree.
Stir some more.
Turn the heat down and keep the concoction simmering until you get bored or the anti-terror squad come round looking for a chemical weapons factory - the longer it simmers the better it gets. Add water if/when it dries out.
Turn the heat off and let that nonsense chill out for a while. Once it's safe(r) to handle, spoon it into freezer bags and freeze it. You can use it straight away if you want, but it's maybe 20% more panic-inducing after it's been frozen. So do that.

To unleash the fear, defrost the sauce and chuck in some chunks of chicken breast. Simmer for 15 minutes or so. If you're feeling particularly hardcore, finely slice another couple of bird-eye chillies and throw those bad boys in there.
Serve with rice and a well rehearsed safety briefing.

...

Being largely skint, I grabbed myself the cheapest netbook going. A Dell Mini 10v. It's awesome. The screen's razor sharp, the keyboard's satisfyingly clacky, and it's really well put together.
Being of the RedHat persuasion, the installed Ubuntu OS survived exactly one boot before being replaced by the mighty Fedora.

• Get the binary WiFi drivers:
  $ su -
  # rpm -Uvh http://download1.rpmfusion.org/free/fedora/rpmfusion-free-release-stable.noarch.rpm http://download1.rpmfusion.org/nonfree/fedora/rpmfusion-nonfree-release-stable.noarch.rpm
  # yum -y update
  # yum -y install kmod-wl broadcom-wl
• Tweak PulseAudio so it doesn't use so much CPU:
  $ su -
  # sed -i 's/; resample-method = speex-float-3/resample-method = speex-float-0/' /etc/pulse/daemon.conf

That's about it, really. Install Flash at your leisure and enjoy full-screen YouTube, iPlayer et al.

There's only one thing wrong with this netbook, and that's the touchpad. The buttons are under the pad, but the areas over the buttons are touch-sensitive, so you can't drag and drop without the pointer flying off in all directions. This isn't a problem that's particular to Linux - Windows 7 users (XP as well, I'd imagine) are reporting the same problem. If you run Linux, though, you get to talk to the developers and raise bug reports. So that's what I did. We'll see how it pans out.

Edit: The trackpad button problem was fixed somewhere around Fedora 15/16. Nice.

I'm in the middle of putting an LDAP infrastructure together, and rolling it out to 40-odd Solaris boxes for user authentication, NFS auto mounts, sudo and all that good stuff. Our consultant/contractor/architect worships at the alter of the Sun gods, so OpenDS 1.2.0 was his choice of LDAP server. Not a bad choice, to be honest - it's a piece of piss to set up and get multiple servers multi-master replicating, and it's Free.
All was tickety-boo until a week ago.

First up, all 4 replicating LDAP servers hung overnight. This locked out the admin team from half of our machines until I got into work and poked them back into life. No errors in the logs, nothing weird going on, just hung Java processes and no logins. Brought down 2 of the instances and upgraded the other two to OpenDS 2.0 so hopefully that should be the end of that. Hopefully. Maybe.

Then today, I'm happily LDAPing away when I ran an ldapmodify to change the UID of the user that runs our monitoring software. LDIF imported, no problem, except I'd run the ldapmodify binding as the user I was modifying, not the Directory Manager. Alarm bells went off in my head - users shouldn't be able to modify their own UID numbers. I tried it again, using 0 as the UID, and sure enough it turns out that anyone that could authenticate to LDAP could also change their UID so that they were running as root. Fuck me very hard indeed. A very swift google tipped up an ACL to add to stop this, as well as a whole load of other possible nightmares.

Next, while messing with the ACLs for the UID problem, it occurred to me that the 'proxy' user could see everyone's passwords in the directory. This is by design, since we're using proxy authentication on our Solaris hosts (which means user passwords aren't sent over the network in plaintext), but it also means that anyone logged into any LDAP-enabled Sun box can search for and list everyone in the directory's passwords with a simple 'ldaplist -l passwd'. The passwords are encrypted, but the old-skool Unix 'crypt' isn't exactly what you'd call military grade protection. Shit.
To get round this, I'm currently beating my head against the brick wall of TLS/SSL so we can remove the need for a proxy user. The tools supplied with OpenDS work fine, but the native Solaris stuff won't go near the self-signed certificates we're using (either that or I'm Doing It Wrong, I'm not entirely sure yet). I've given up for the night, but it looks like I'm going to have to generate CA certificates, then regenerate and sign all the certs for the LDAP servers, then import them, then import the CA and server certs into the LDAP config on the clients, then see if the native Solaris stuff works with them, then re-run ldapclient on all the clients. Arse.

I don't get paid enough for this.

Do it. Do it now. OM NOM NOM NOM.

And don't forget to tip your delivery guy.

Things of note:
1) When your dentist goes "Hmmm." during a procedure, and invites one of his colleagues to join the ongoing mouth-party, this is not, I am told, cause for alarm. Contrary to the prevailing advice, I chose to become entirely (albeit very quietly) alarmed.
2) Your Dental Specialist may hide the various implements of healing behind his or her back until just before they are to be used. These implements may or may not include:
Large syringes
Well-worn pliars
Blood-filled suction tubes
I'm not entirely certain that this behaviour is normal, however. The furtive yet swift nature in which my dentists work leads me to believe that I've stumbled on a clique of ninja health professionals. I'm expecting a re-enactment of the Shimabara rebellion at the next check-up.
3) Rinsing the resulting casm in your jaw with salt-water according to instruction is entirely ineffective; expect to poke about in there with a toothpick to retrieve horrifying globs of matter, lest your breath become the subject of future Germanic fairy tales.

That's my experience anyway. Your milage may, as ever, vary.

While we're on the subject, check this out. Awesome.

* - Yeah, I'm fucking hilarious.

I don't know what they're made of, but they won't burn without continuous provocation. Sinister.

Having trouble? Can't see your storage? Does your 'cfgadm -al' output look like this:

c11     fc-fabric    connected    unconfigured   unknown

Yeah, me too. Turns out the Emulex cards are a bit shy. You can poke them into life with the following:

luxadm -e forcelip /dev/cfg/c11

Change the c11 to whatever the device ID is for your card.

Tread carefully fellow admins - don't do this on an HBA that's already handling mounted storage - it resets all the ports on the loop, so you might lose sight of your storage for a second or two.

Never had to do any of this with QLogic cards...

FEMALE PREFERRED -
SHOULD BE TALL AND
VERY STRONG AS YOU
WILL HAVE TO CARRY
A SIZEABLE CASE - BUT
THAT IS 50% OF WHAT
I'M PAYING YOU FOR.

What's going on here? What's the other 50% of what (s)he's paying you for? What's in the case? Guesses in the comments, if you would.

Now, I fucking hate flying. Sitting in an aluminium tube (1mm thick! No, I'm not joking!), 6 miles up, doing 500mph, features prominently on my list of Things I Don't Like Doing. It's really pretty high on that list. Just above 'flying' on the list, though, is 'sitting in a GNER shithole-on-rails for 5 hours next to a cantankerous old twat that complains every time he hears the slightest "tsss" from my headphones', so I'll risk the potential explosive-decompression 30,000 foot free-fall inferno death, thanks.

Anyway, all the motherboard connections have changed since the Athlon XP days, so I'm having to replace pretty much everything. Here's the run-down:
Intel Core 2 Duo E7400 Processor
Asus P5QL SE S775 Motherboard
2GB Kingston RAM
NVidia 9500GT 1GB Video Card
250GB Seagate Barracuda Hard Drive
Akasa Ultra Quiet 460w Power Supply
I'm keeping my Lian Li case, the two CD/DVD drives, and my trusty Trinifuckinghugetron Sun monitor (1600x1200 of glorious ).

Total: 300 quid delivered. None too shabby. Can't really afford it, but still, none too shabby. Yes, the video card's weak and 2GB RAM isn't that much these days, but Unreal 2004 is about the most graphics intensive game I play, and Linux is pretty light on RAM. Hell, the old machine only had 1/2GB RAM and a GeForce 5200, fortheloveofgod.

Exactly a year ago, when I first moved in, my heavy-as-fuck blinds came off their hilariously flimsy mounts and attempted to remove my right arm on the way down. That time my shoulder popped out and back in again, resulting in only temporary pain and resentment.
On Saturday night, exactly the same thing happened, but the pain and resentment has persisted. A mortal blow was also delivered to my long-suffering dragon tree, and my equally long-suffering speakers have received injuries that may require plastic surgery.

Balls to the tree and speakers, though. My shoulder's gubbed, so I'm heading to the venerable family GP to demand satisfaction. Or whatever he recommends. The mouse-to-keyboard-and-back arm-rotation is now accompanied by a painful 'clunk', so I'm slightly concerned that this might end up being a real problem what with my profession and all. I can see it putting a dent in my sex life anyway, if you know what I mean. And I think you do.

It's probably worth noting that both of my shoulders are what I believe is termed unstable, so this was going to happen sooner or later. Actually, my knee used to do the same pop-out-and-back routine as well so I'm guessing it's congenital, and therefore my parents' fault more than my flat's. (If you want to experience pain and panic in equal and very large doses, I can recommend trying to straighten out a partially dislocated knee every time - popped shoulders are a definitely piece of piss in comparison.)

On a lighter note, the Sub was all kinds of excellent on Saturday (before the attempted dismemberment) - Domenic's back from Barcelona for good, and he's chucking around some fine quality House.