Ubuntu 12.04.1 LTS: Recovering from a Broken libc

Note: Reasonably heavy UNIX-geekery ahead. Mostly Linux-specific, somewhat Debian-specific and a little bit Ubuntu-specific. Skip if that isn’t your cup of ichor.

I recently did something incredibly ill-considered while logged in (as root, natch) to my Ubuntu server box at home. In essence, I told the package manager to uninstall libc.

On a scale from good to bad, this is bad.

Now, I could have booted up from a rescue disk and fixed it. I could have re-installed easily enough (using the trick where you just keep your existing partitions and don’t format them — though this would have inevitably led to some fallout as various config files and customizations got clobbered). Heck, I even had a reasonably recent backup at hand. None of these sounded especially fun, mostly because the machine was in a place where it’s a hassle to stick a head on it. I had three things working in my favor: an open root-privileged shell prompt, Internet connectivity and my native cunning.

Read on after the jump for the full tale.

So, the first and most obvious problem (other than “If you didn’t want to do that, then why did you type the command?”) was that, while I had a shell prompt, most of the usual commands are dynamically linked and therefore busted. I had the stuff in /sbin, /usr/sbin and the shell builtins. OK, so I can use “echo *” instead of “ls”. But there’s no static apt-get or dpkg (or wget or tar or ftp or cat or dd).

Fortunately for me, the Ubuntu gnomes had kindly provided a static-sh command. This is a busybox with all kinds of treats built in, and not a dynamic library in sight. All the basic command line stuff seemed to be there, as well as such things as tar and bzip2 and wget. (Aside: Though I linked to the page for the full-blown GNU wget, the version in busybox is a work-vaguely-alike stripped down to the bare essentials.)

So, first problem: How do I get my system back to a somewhat-working state?

1. On another system, get the appropriate libc6 package (.deb) from packages.ubuntu.com. In my case, it was libc6-amd64_2.15-0ubuntu10.3_i386.deb.
2. Still on the other system, unpack the contents into a subdirectory (preferrably, without adgering that second system in the process):
   

   mkdir foo
   cd foo
   dpkg-deb -x ../libc6-amd64_2.15-0ubuntu10.3_i386.deb

3. Yet again on the other system, archive up the extracted files (and remove the unpacked version):
   

   tar cvf – . | bzip2 -v9 >../libc6.tar.bz2
   cd ..
   rm -rf foo

4. Put the archive thus created on a handy Internet-accessible web server.
5. On my broken system, filled with brokenness, which I broke, launch static-sh.
6. On the broken system, use wget to fetch the archive and extract in the root directory:
   

   cd /
   wget http://mywebserver.example.org/some/path/libc6.tar.bz2
   bunzip2 -c libc6.tar.bz2 | tar xvf –

7. Exit static-sh and bask in the power of a vaguely usable system.

That was obviously an improvement. All the usual shell commands worked again. I could ssh in from outside and sudo. ldconfig -v did something reasonable. I could even reboot the system and it would come back up and work.

The remaining problem was that the package manager was not best pleased. As far as it could tell, libc6 was uninstalled and deselected, and every dependency known to man was broken. Everything is ruined forever!

Okay, whatever.

cd /var/cache/apt/archives
dpkg -i libc6-amd64_2.15-0ubuntu10.3_i386.deb

would fix it, right? Right?

Not so fast.

A copy of the C library was found in an unexpected directory:
  '/lib/x86_64-linux-gnu/libc-2.15.so'
It is not safe to upgrade the C library in this situation;
please remove that copy of the C library or get it out of
'/lib/x86_64-linux-gnu' and try again.

Update: As of 24 April, 2014, the libc6-amd64_2.15-0ubuntu10.3_i386.deb package appears to have an updated preinstall script which does not contain the check that produces the error quoted above. If you have this problem, please re-download the package and try installing again, before you bother trying the more complex method outlined below.

Now, I get what the package maintainer was thinking here. Screwing this up could turn a working (or even partly-working) system into a doorstop. But I had already completely removed libc and lived to tell the tale. What if I really truly did want to JFDI and accept the consequences? (Not having a working package manager is not a viable long-term solution. If I couldn’t fix this, I’d have to re-install anyway…)

I tried various promising things that a reasonable and prudent person might think would work. Here is a partial list:

• dpkg -i –force-all
• dpkg –unpack
• performing the installation in a chroot-ed jail
• actually following the suggestion in the error text and removing the offending file
• making a temp directory somewhere, adding it to /etc/ld.so.conf, and moving offending files there one at a time, running ldconfig -v after each

(Though it is perhaps worth noting that the second-to-last of these put the system back in the original broken state. Yay!)

Welp. Brute force and ignorance got me into this mess in the first place… perhaps they can get me out.

cd /var/cache/apt/archives
mkdir libc6-amd64_2.15-0ubuntu10.3_i386
cd libc6-amd64_2.15-0ubuntu10.3_i386
dpkg-deb -R ../libc6-amd64_2.15-0ubuntu10.3_i386.deb .
# edit DEBIAN/preinst to remove exit 1 following error quote above
# and also to replace error text with vulgarity
cd ..
mv libc6-amd64_2.15-0ubuntu10.3_i386.deb libc6-amd64_2.15-0ubuntu10.3_i386.deb.real
dpkg-deb -b libc6-amd64_2.15-0ubuntu10.3_i386
rm -rf libc6-amd64_2.15-0ubuntu10.3_i386
dpkg -i libc6-amd64_2.15-0ubuntu10.3_i386.deb

Against all odds, that seemed to work.

Updated 2012-Feb-01 by DGH: The dpkg-deb command has no -B option. Corrected to -b. Thanks, Scott.