ILikeJam

Interesting weekend.

A patch run on one of our production clusters resulted in one of the applications refusing to start. After much head-scratching and incrementation of logging verbosity, it was discovered that the data files for this software had expired on Feb 14th. It appears the software in question only checks the expiry dates when it starts, so the reboot after the patching caught us out. Calls to the software vendor revealed that they'd been sending us old copies of the data and they'd have to snail-mail us a new CD with the latest files to get us up and running again, leaving us without a fairly critical piece of infrastructure for a few days.

Yeah, not on my watch. Running the software through 'truss' (the Solaris 'strace' equivalent, if you're familiar with Linux) showed a number of 'time()' calls at startup, so a plan was hatched to unashamedly lie to the software.

A fake 'time()' function was thrown together to return the Unix-time for 2011/02/13-23:59:59 (a second before the data expiry) regardless of the current time:

int time() {
        return 1297641599;
};

$ cc -G -o time.so time.c

$ LD_PRELOAD=./time.so ./server-binary

Bingo. Surprisingly, the software doesn't seem to mind too much that time's always 1 second to midnight on Feb 13th. The datestamps in the application logs are royally messed up, but we're back up and running.

Good Time(s).

As you may or may not be aware, Google recently reached out and touched me in a special way. They asked if I might be interested in having a chat with some engineers to see if I pack the gear to serve in their beloved corps. Now, in my line of work when Google calls you pick up the damn phone, and if, like me, you'd really quite like to work for Google, here's how it works. This was for a Unix/Application admin (SRE / Site Reliability Engineer) position - the process is probably different for Software Engineering or management.

All of the interviewers had a master list of scenarios and questions (which I managed to get a fairly good look at during one of the face-to-face interviews - my Special Skill is reading upside-down), but I'm not going to reveal the details of any of the questions. I have a feeling the list doesn't change all that often.

Here we go.

If you're not already on LinkedIn, get yourself on there and add some colleagues. Join the various groups that are relevant to your mad skills. In my case, I'm fairly sure I was picked out of a 'Unix Professionals in the UK' type group. LinkedIn seems to be the One and Only place that recruiters use to find people for decent Unix jobs at the moment - I'm averaging about 2 recruiters a week looking for Unix people, even in the currently horrific job market.

If you're lucky, a recruiter from Google will send you a LinkedIn message which you should reply to with an only slightly restrained email explaining that in fact this is the best thing that has ever happened to you, and that you'd absolutely love to get in on the action.
And so it begins.

There will be a phone screen where your recruiter will ask you some simple, closed, technical questions. As far as I can tell, the recruiters don't have the answers to these questions, so there's no point in haggling or second guessing. Don't sweat it too much - I know I got a few details wrong, but still got through. I think they're basically weeding out the idiots and mental cases at this point - if you're not an idiot or mental, you should be OK.

After the phone screen, you should get an email asking you to rate your afore mentioned mad skills in 11 different fields:

• TCP/IP Networking (OSI stack, DNS, etc.)
• Unix/Linux System Administration tasks
• Unix/Linux internals
• Algorithms & Data Structures
• SQL and / or Database administration
• C
• C++
• Java
• Perl
• Shell Scripting (sh, bash, ksk, csh)
• Python

So begins the phone interviews. My first interview involved a couple of a couple of fairly simple Unix admin problem scenarios, both of which were answered with a series of commands to run, and the reasons for using those commands. If you've been doing any sort of Unix admin for any length of time, these shouldn't be a problem. There was also a more involved discussion about a standard piece of software and how it works. I wasn't particularly familiar with the way this software works, but I got through by suggesting ways which I thought the software might operate, and the sorts of circumstances under which the software might have problems, assuming it worked the way I thought it worked.

The second phone interview involved some network knowledge, and a discussion on another piece of standard Unix software. Again, I wasn't entirely sure of the finer details of how this software works, but again some educated guesses and explanations of why I thought the software works the way it works was enough. There was also a scripting exercise which on the face of it looked simple, but which was a bit tricky to figure out on-the-fly. During this part, you'll be typing into a Google Docs word document so the interviewer can watch you type. This is actually quite cool - the interviewer gets to watch as you assemble a solution, so even if the resulting script isn't quite right they can see what you're thinking as you hack out an answer.

If at any point during the phone interviews you feel like running 'apropos', that's up to you. Type quietly, though.

So far so good. If you've made the cut, Google will invite you round to theirs for lunch.
Lunch, and 5 one-on-one interviews.

I'm fairly sure I signed an NDA when I arrived at the Chocolate Factory, so I can't really tell you much about the on-site stuff. A few pointers, though:
1: If you don't know your shit cold, you aren't getting through.
2: Learn every scripting trick out there; you're going to need them.
3: Have the pizza for lunch. It's good.
Contrary to popular belief there aren't any questions about how many golf balls fit in a bus, or how many piano tuners there are in Chicago, but you may have to pull estimates out of your arse involving very large amounts of data and very large numbers of users, so these sorts of questions are more than slightly relevant. Apart from that, it's all technical. No soft-skills or HR bullshit. No messing about.

Sadly, I royally ballsed up one of the interviews, so The Panel (quite rightly) deemed me unworthy of a position.

Arse.

Here's the recipe for the awesome, tasty, blows your head clean off ILikeJam curry sauce.

Should be enough to put 4 idiots into respiratory arrest.

You'll need:

• 80g ginger
• 6 green chillies
• 3 bird-eye chillies
• 4 fat cloves of garlic
• A tin of peeled plum tomatoes
• A fat onion
• 4 tbsp ground nut oil
• 2 tsp cumin
• 1 tsp salt
• 1/2 tsp chilli powder
• 1/2 tsp cayenne pepper
• 2 tsp turmeric
• 2 tsp garam masala
• 1 1/2 tbsp tomato puree

Shove the ginger, all the chillies (seeds included - no messing about) and garlic in a blender and blend to a paste. Transfer that dangerous shit into a bowl before it dissolves the blender blades.
Drain the tinned tomatoes and stick them in the blender. Give the button a few hits - you want a bit of texture left. Put the resulting slop in the bowl with the previously blended stuff.
Chop your onion.
Grab a pot and throw in the chopped onion and oil. Fry those tasty bits of onion fairly slowly 'til they're soft. Chuck the cumin in and stir, then give it a bit more frying time.
Add the chillies/tomato/etc to the onion and stir until it's venting noxious fumes. Keep stirring and add the salt, chilli powder, cayenne, turmeric and garam masala, then the tomato puree.
Stir some more.
Turn the heat down and keep the concoction simmering until you get bored or the anti-terror squad come round looking for a chemical weapons factory - the longer it simmers the better it gets. Add water if/when it dries out.
Turn the heat off and let that nonsense chill out for a while. Once it's safe(r) to handle, spoon it into freezer bags and freeze it. You can use it straight away if you want, but it's maybe 20% more panic-inducing after it's been frozen. So do that.

To unleash the fear, defrost the sauce and chuck in some chunks of chicken breast. Simmer for 15 minutes or so. If you're feeling particularly hardcore, finely slice another couple of bird-eye chillies and throw those bad boys in there.
Serve with rice and a well rehearsed safety briefing.

My ancient Thinkpad died a few days ago. A moment's silence, if you please.

...

Being largely skint, I grabbed myself the cheapest netbook going. A Dell Mini 10v. It's awesome. The screen's razor sharp, the keyboard's satisfyingly clacky, and it's really well put together.
Being of the RedHat persuasion, the installed Ubuntu OS survived exactly one boot before being replaced by the mighty Fedora.

• Get the binary WiFi drivers:
  $ su -
  # rpm -Uvh http://download1.rpmfusion.org/free/fedora/rpmfusion-free-release-stable.noarch.rpm http://download1.rpmfusion.org/nonfree/fedora/rpmfusion-nonfree-release-stable.noarch.rpm
  # yum -y update
  # yum -y install kmod-wl broadcom-wl
• Tweak PulseAudio so it doesn't use so much CPU:
  $ su -
  # sed -i 's/; resample-method = speex-float-3/resample-method = speex-float-0/' /etc/pulse/daemon.conf

That's about it, really. Install Flash at your leisure and enjoy full-screen YouTube, iPlayer et al.

There's only one thing wrong with this netbook, and that's the touchpad. The buttons are under the pad, but the areas over the buttons are touch-sensitive, so you can't drag and drop without the pointer flying off in all directions. This isn't a problem that's particular to Linux - Windows 7 users (XP as well, I'd imagine) are reporting the same problem. If you run Linux, though, you get to talk to the developers and raise bug reports. So that's what I did. We'll see how it pans out.

http://petitions.number10.gov.uk/dontdisconnectus/
Go on. You know you want to.

LDAP: Largely Dangerous And Painful. Let me describe why. (Normal people can give this one a miss. Fellow sysadmins can revel in the ridiculousness).

I'm in the middle of putting an LDAP infrastructure together, and rolling it out to 40-odd Solaris boxes for user authentication, NFS auto mounts, sudo and all that good stuff. Our consultant/contractor/architect worships at the alter of the Sun gods, so OpenDS 1.2.0 was his choice of LDAP server. Not a bad choice, to be honest - it's a piece of piss to set up and get multiple servers multi-master replicating, and it's Free.
All was tickety-boo until a week ago.

First up, all 4 replicating LDAP servers hung overnight. This locked out the admin team from half of our machines until I got into work and poked them back into life. No errors in the logs, nothing weird going on, just hung Java processes and no logins. Brought down 2 of the instances and upgraded the other two to OpenDS 2.0 so hopefully that should be the end of that. Hopefully. Maybe.

Then today, I'm happily LDAPing away when I ran an ldapmodify to change the UID of the user that runs out monitoring software. LDIF imported, no problem, except I'd run the ldapmodify binding as the user I was modifying, not the Directory Manager. Alarm bells went off in my head - users shouldn't be able to modify their own UID numbers. I tried it again, using 0 as the UID, and sure enough it turns out that anyone that could authenticate to LDAP could also change their UID so that they were running as root. Fuck me very hard indeed. A very swift google tipped up an ACL to add to stop this, as well as a whole load of other possible nightmares.

Next, while messing with the ACLs for the UID problem, it occurred to me that the 'proxy' user could see everyone's passwords in the directory. This is by design, since we're using proxy authentication on our Solaris hosts (which means user passwords aren't sent over the network in plaintext), but it also means that anyone logged into any LDAP-enabled Sun box can search for and list everyone in the directory's passwords with a simple 'ldaplist -l passwd'. The passwords are encrypted, but the old-skool Unix 'crypt' isn't exactly what you'd call military grade protection. Shit.
To get round this, I'm currently beating my head against the brick wall of TLS/SSL so we can remove the need for a proxy user. The tools supplied with OpenDS work fine, but the native Solaris stuff won't go near the self-signed certificates we're using (either that or I'm Doing It Wrong, I'm not entirely sure yet). I've given up for the night, but it looks like I'm going to have to generate CA certificates, then regenerate and sign all the certs for the LDAP servers, then import them, then import the CA and server certs into the LDAP config on the clients, then see if the native Solaris stuff works with them, then re-run ldapclient on all the clients. Arse.

I don't get paid enough for this.

Get yo' ass to dominos.co.uk, and construct yo' ass one of these:
- Regular base
- Pizza sauce
- Domino's Herbs
- Green Peppers
- Olives
- Roquito Sweet Chilli Peppers
- Tandoori Chicken
- Jalapeno Peppers
- Onions
- Pepperoni

Do it. Do it now. OM NOM NOM NOM.

And don't forget to tip your delivery guy.

I had a tooth removed recently. Contrary to received wisdom*, the whole procedure was completely painless and largely unremarkable.

Things of note:
1) When your dentist goes "Hmmm." during a procedure, and invites one of his colleagues to join the ongoing mouth-party, this is not, I am told, cause for alarm. Contrary to the prevailing advice, I chose to become entirely (albeit very quietly) alarmed.
2) Your Dental Specialist may hide the various implements of healing behind his or her back until just before they are to be used. These implements may or may not include:
Large syringes
Well-worn pliars
Blood-filled suction tubes
I'm not entirely certain that this behaviour is normal, however. The furtive yet swift nature in which my dentists work leads me to believe that I've stumbled on a clique of ninja health professionals. I'm expecting a re-enactment of the Shimabara rebellion at the next check-up.
3) Rinsing the resulting casm in your jaw with salt-water according to instruction is entirely ineffective; expect to poke about in there with a toothpick to retrieve horrifying globs of matter, lest your breath become the subject of future Germanic fairy tales.

That's my experience anyway. Your milage may, as ever, vary.

While we're on the subject, check this out. Awesome.

* - Yeah, I'm fucking hilarious.

PROTIP:
Credit cards are self-extinguishing.

I don't know what they're made of, but they won't burn without continuous provocation. Sinister.

Let's say you've just added an Emulex FC HBA or two to your Solaris box. Let's also say that you booted the machine before you hooked up the fibre, zoned the switch and presented some storage.

Having trouble? Can't see your storage? Does your 'cfgadm -al' output look like this:

c11     fc-fabric    connected    unconfigured   unknown

Yeah, me too. Turns out the Emulex cards are a bit shy. You can poke them into life with the following:

luxadm -e forcelip /dev/cfg/c11

Change the c11 to whatever the device ID is for your card.

Tread carefully fellow admins - don't do this on an HBA that's already handling mounted storage - it resets all the ports on the loop, so you might lose sight of your storage for a second or two.

Never had to do any of this with QLogic cards...