Recovering a Debian System after running rm /*

Flying - felixtsao

The Oxford English Dictionary (I cannot believe that this is actually defined in the OED. I assumed it was a fake site or something. I also cannot believe that I actually used “The OED defines […]”. I didn’t even use that the best man speech at my brother’s wedding.) defines an ohnosecond as:

a moment in which one realizes that one has made an error, typically by pressing the wrong button.

It’s more commonly referred to in Operations Management parlance as:

OHGODFUARGHFLKJAFWHATIDIDIDOARGHNO

It is unfortunately something that will happen to everyone during their systems administration career, and the variations are almost endless, some notable occurrences include:-

  • Copying SSL libraries over from a Debian host to a RHEL host
  • Setting a new root password and immediately losing it
  • Copying over an out of date backup CMS to a production system
  • Running one of the many variations of ‘rm’ at the wrong level

Unfortunately in a recent scenario, a poor hypothetical sysadmin managed to issue:

rm /*

instead of:

 rm ./*

This removed every non directory at the / level. The impact of this varies between operating systems, and even between Linux distributions. We’re lucky that in this scenario, there was no ‘-rf’ specified – or it’d be ‘recover from backup’ time, however this situation did (hypothetically) pose an interesting conundrum.

The Problem

In Debian x86_64 systems, /lib64 is a symlink to /lib, and you’ll find most applications (for instance, ‘ldd’) are linked to libraries in /lib64:

linux-vdso.so.1 => (0x00007fff251ff000)
libc.so.6 => /lib/libc.so.6 (0x00007f278eb38000)
/lib64/ld-linux-x86-64.so.2 (0x00007f278eea2000)

In the event of /lib64 not being accessible, most applications will fail to run because they’ll be missing a myriad of dependencies and won’t be able to find them. After a bit of brief investigation and some furious attempts to revive it with frankly disappointing results, including:

  • Using a statically compiled symlinker such as sln (Available by default on CentOS and RHEL, not on the affected Debian box)
  • Copying over sln via netcat and writing it out (proved surprisingly difficult)
  • Trying to copy over a symlink via rsync (couldn’t rsync/scp/sftp as they need to exec another process – which they can’t because of missing libraries)
  • Using BusyBox (Needs dynamic linking)
  • Writing a linker in C, compiling it, getting it over there via a mixture of cat, echo \x{..}\x{..}, and other incantations (I lost the will to live around this point)

The Epiphany

I eventually remembered a slideshow – chmod -x chmod – which was surprisingly relevant. You see, those more eagle eyed may have noticed we would end up missing one important dependency: ld-linux-x86-64.so.2.

ld-linux and ld-linux-x86-64 find and load shared libraries used by other applications – preparing programs to run, and then actually executing them too. Most Linux binaries require dynamic linking, meaning at runtime the libraries that the application depends on are loaded in from a shared source rather than compiled into the executable, unless the -static option was used during compilation. As this is quite unlikely (with most modern distributions), this means if you cannot access ld-linux.so, you’re in trouble. Luckily, you can still use ld-linux.so to execute arbitrary commands, and it’ll resolve the dependencies relative to your LD_LIBRARY_PATH at that point. A simple:

/lib/ld-2.11.1.so /bin/ln -s /lib/ /lib64/

Restored the symlink and allowed normal execution of binaries again, leaving our hypothetical sysadmin off the hook, except for having to write a mildly humiliating email to the rest of the operations team who, understandably, responded a bit like this.

Photo Flying by felixtsao (CC)

Leave a Reply

Your email address will not be published. Required fields are marked *