Recovering a Debian System after running rm /*
The Oxford English Dictionary (I cannot believe that this is actually defined in the OED. I assumed it was a fake site or something. I also cannot believe that I actually used “The OED defines […]”. I didn’t even use that the best man speech at my brother’s wedding.) defines an ohnosecond as:
a moment in which one realizes that one has made an error, typically by pressing the wrong button.
It’s more commonly referred to in Operations Management parlance as:
OHGODFUARGHFLKJAFWHATIDIDIDOARGHNO
It is unfortunately something that will happen to everyone during their systems administration career, and the variations are almost endless, some notable occurrences include:-
- Copying SSL libraries over from a Debian host to a RHEL host
- Setting a new root password and immediately losing it
- Copying over an out of date backup CMS to a production system
- Running one of the many variations of ‘rm’ at the wrong level
Unfortunately in a recent scenario, a poor hypothetical sysadmin managed to issue:
rm /*
instead of:
rm ./*
This removed every non directory at the / level. The impact of this varies between operating systems, and even between Linux distributions. We’re lucky that in this scenario, there was no ‘-rf’ specified – or it’d be ‘recover from backup’ time, however this situation did (hypothetically) pose an interesting conundrum.
The Problem
In Debian x86_64 systems, /lib64 is a symlink to /lib, and you’ll find most applications (for instance, ‘ldd’) are linked to libraries in /lib64:
linux-vdso.so.1 => (0x00007fff251ff000) libc.so.6 => /lib/libc.so.6 (0x00007f278eb38000) /lib64/ld-linux-x86-64.so.2 (0x00007f278eea2000)
In the event of /lib64 not being accessible, most applications will fail to run because they’ll be missing a myriad of dependencies and won’t be able to find them. After a bit of brief investigation and some furious attempts to revive it with frankly disappointing results, including:
- Using a statically compiled symlinker such as sln (Available by default on CentOS and RHEL, not on the affected Debian box)
- Copying over sln via netcat and writing it out (proved surprisingly difficult)
- Trying to copy over a symlink via rsync (couldn’t rsync/scp/sftp as they need to exec another process – which they can’t because of missing libraries)
- Using BusyBox (Needs dynamic linking)
- Writing a linker in C, compiling it, getting it over there via a mixture of cat, echo \x{..}\x{..}, and other incantations (I lost the will to live around this point)
The Epiphany
I eventually remembered a slideshow – chmod -x chmod – which was surprisingly relevant. You see, those more eagle eyed may have noticed we would end up missing one important dependency: ld-linux-x86-64.so.2.
ld-linux and ld-linux-x86-64 find and load shared libraries used by other applications – preparing programs to run, and then actually executing them too. Most Linux binaries require dynamic linking, meaning at runtime the libraries that the application depends on are loaded in from a shared source rather than compiled into the executable, unless the -static option was used during compilation. As this is quite unlikely (with most modern distributions), this means if you cannot access ld-linux.so, you’re in trouble. Luckily, you can still use ld-linux.so to execute arbitrary commands, and it’ll resolve the dependencies relative to your LD_LIBRARY_PATH at that point. A simple:
/lib/ld-2.11.1.so /bin/ln -s /lib/ /lib64/
Restored the symlink and allowed normal execution of binaries again, leaving our hypothetical sysadmin off the hook, except for having to write a mildly humiliating email to the rest of the operations team who, understandably, responded a bit like this.
