Unix: April 2008 Archives

Solaris patching rant

| | Comments (2)

It's customary for Solaris admins to slag off the patch system, so I guess I'll have a go. For the sake of full disclosure: I used to be part of the 1st / 2nd line support team that does global support for smpatch / Sun Update Connection / Update Connection Enterprise. If you raised a Sun case on these products between November 2005 and April 2007, you very probably spoke to me at some point. That said, most of what I have to say is based on my experience working with Solaris, rather than supporting it...

smpatch is broken, but you already know that.
The dependency tree required for a working smpatch install is vast - so vast that Sun won't (didn't?) support anything less than a full end-user (SUNWCuser) install cluster for smpatch. Put together a minimal install for a boundary system and want to patch it? Get yo' Recommended Cluster on, dawg.
Any kind of slow-ish or intermittent Internet connection, and you'll have to supervise patch downloads, lest the 'Error: null' beast be awoken from its slumber (just re-run the 'smpatch download' - you'll usually get at least one more patch down before it errors out again). This is typical of the error messages from smpatch - largely useless, mostly misleading, occasionally deceitful.
Circular dependency in your smpatch database (recent kernel/zones patches, for example)? Ha ha haaaa, sucks to be you. smpatch can't figure it out, so you'll just get a load of patchadd failures during the smpatch run and subsequent shutdown. Re-running smpatch won't work, so don't bother. Time to hit SunSolve and work backwards down the dependency tree manually. You'd better hope SunSolve's feeling perky (gateway timeout anyone? Yeah, thought so), 'cause you're going to be there for a long time.

sconadm is as broken as smpatch, but with the added advantage of weird database issues at the Sun side which may stop you registering anything on your support contract. "Registration failed!" it will challenge. "What the fuck?" you will retort. Do not attempt to debug sconadm issues - your sanity is more valuable than your pride. Call Sun and get hold of SWUP_SUPPORT - they'll give you a script to run that will detail all the packages and patches missing from your system, including a Java update you can't install because your Oracle instance depends on a particular JDK version. But maybe that's just me.

Update Connection Proxy? Don't put yourself through the agony.

The enlightened (with similarly enlightened management) among us use PCA and avoid the smpatch/sconadm brain damage. I heartily recommend that you do too.
Giggle as your patches download without error. Rejoice in the HTML patch list output, with links to the READMEs for each patch. Bask in the glory of the caching PCA proxy.
Unfortunately though, the horror runs much deeper than just the automated patch tools...

The way patches are rolled is insane. Any one patch can update several packages, all-but-one of which may not be installed on your host at the time of patching. What happens if you install one of those not-installed packages after patching? That's right. The patch tools see all your applied patches, so won't recommend any of them again despite the just-installed, unpatched package on your system. Cue headaches trying to work out why you just got pwn3d by 1337 |-|4x0rz despite apparently being patched up to the eyeballs.

On no other Unix (that I've ever worked on at least) does a kernel patch clobber your sendmail config. This is a result of the way the patches are rolled, as above. You were told during the patchadd operation that the sendmail config had been moved, but you weren't expecting what was allegedly a driver update to affect userland apps, and you don't have the time to review 1000 lines of smpatch output for each of 30 odd machines at the end of a 12 hour shift caused by smpatch failing. to. download. every. second. patch, so you missed it. Say goodbye to the free space in /var as the mail spool fills up over the next few days. Say goodbye to your email-based system monitoring.

Ah well, I suppose it's a living. Say hi to SWUP_SUPPORT for me next time you raise an smpatch case. There's also a pretty good blog by one of the PST team at Sun which goes some way to explaining the madness: Patch Corner. Not pretty, is it?