Interesting weekend.
A patch run on one of our production clusters resulted in one of the applications refusing to start. After much head-scratching and incrementation of logging verbosity, it was discovered that the data files for this software had expired on Feb 14th. It appears the software in question only checks the expiry dates when it starts, so the reboot after the patching caught us out. Calls to the software vendor revealed that they'd been sending us old copies of the data and they'd have to snail-mail us a new CD with the latest files to get us up and running again, leaving us without a fairly critical piece of infrastructure for a few days.
Yeah, not on my watch. Running the software through 'truss' (the Solaris 'strace' equivalent, if you're familiar with Linux) showed a number of 'time()' calls at startup, so a plan was hatched to unashamedly lie to the software.
A fake 'time()' function was thrown together to return the Unix-time for 2011/02/13-23:59:59 (a second before the data expiry) regardless of the current time:
int time() {
return 1297641599;
};
That was compiled as a shared object:
$ cc -G -o time.so time.cAnd the shared object was put into LD_PRELOAD so that the fake 'time()' function was called instead of the normal libC 'time()':
$ LD_PRELOAD=./time.so ./server-binary
Bingo. Surprisingly, the software doesn't seem to mind too much that time's always 1 second to midnight on Feb 13th. The datestamps in the application logs are royally messed up, but we're back up and running.
Good Time(s).
