Converting a 32-bit x86 Gentoo Linux installation to 64-bit

In mid-2013, I converted my 32-bit Gentoo Linux x86 installation to 64-bit using an in-place rebuild (emerge -e world), in spite of Gentoo's advice to perform a fresh install. This file documents the steps I followed to perform the conversion, as well as the problems I ran into along the way.

Note (March 2019): Kimmo Martimo reports that the /lib -> /lib64 symlink configuration has been deprecated as of the Gentoo 17.1 profile, and additional dependencies in several packages require changes to the instructions. I've reproduced his notes with permission below.

Note (May 2014): Gentoo is in the process of deprecating the setup described below, in which /lib is a symlink to /lib64 (and similarly for /usr/lib). Similar instructions should work for non-symlinked systems, though more care may be needed when replacing files in /lib. For details, see bug 506276.

Note that most operations will need to be performed as root. As the sudo command may fail during the course of the rebuild, I recommend using a root shell with either su or sudo bash (or logging in directly as root), depending on your system and preference.

0. Back everything up

Before doing anything else, make sure to back up your entire system. If something goes wrong, you may become unable to even boot, much less perform any recovery operations like copying files around.

1. Install an x86-64 cross-compiler

In order to build an x86-64 kernel on a 32-bit system, you'll need to install an x86-64 cross compiler. This can be done by emerging the sys-devel/crossdev package and running crossdev -s1 -t x86_64 (optionally add -S to use the stable versions of the toolchain).

2. Build an x86-64 kernel

The Linux kernel now treats 32-bit and 64-bit builds as two variations of the same architecture, so configuring a 64-bit kernel is mostly as easy as setting the "64-bit kernel" option. (If you don't have an existing .config file in your kernel source directory, copy it from the appropriately-versioned file in /boot or extract it from the running kernel with zcat /proc/config.gz >.config.) If you want to take a closer look at options specific to x86-64, you can instead change these two lines: # CONFIG_64BIT is not defined CONFIG_X86_32=y to: CONFIG_64BIT=y # CONFIG_X86_32 is not defined and then run make oldconfig.

Important: Make sure the CONFIG_IA32_EMULATION option ("IA32 Emulation" under "Executable file formats / emulations") is enabled, or you won't be able to boot the system with the new kernel! You don't need to enable CONFIG_X86_X32 ("x32 ABI for 64-bit mode"), however.

Once the kernel is configured, you'll need to build it with the x86-64 cross-compiler you installed in step 1. Run the command: make bzImage install modules modules_install CROSS_COMPILE=x86_64-pc-linux-gnu- (don't forget the hyphen at the end of the line). For a faster build, you can pass the option -jN (where N is the number of jobs to run at once; typically, one more than the number of logical CPU cores is a good choice) on the make command line, but you may need to run the targets one at a time (make -jN bzImage && make -jN install && ...).

3. Boot into the new kernel

Since you enabled CONFIG_IA32_EMULATION in the new kernel, it will be able to run your existing 32-bit system just fine (unless you have any out-of-tree modules, such as app-emulation/vmware-modules or x11-drivers/nvidia-drivers). Reboot into the new kernel and verify that it works.

4. Set up a recovery shell

As insurance against something going wrong, set up some sort of recovery shell that doesn't depend on any live libraries in the system. This can either be a statically linked all-in-one shell binary like busybox (sys-apps/busybox) or a chrooted shell in your system backup. The latter is safer in that you'll also have all of your old binaries and libraries available, but make sure the backup is mounted readonly so you don't accidentally modify it. Also make sure to bind your filesystem root so that it's available inside the chroot: mount --bind / /chroot/mnt (assuming your chroot environment is located in /chroot and you want to make the root filesystem visible under /mnt inside the chroot).

5. Copy existing 32-bit libraries to lib32 directories

IMPORTANT: Ensure that your recovery shell is available from this point forward. If you use the chroot method, start up the chroot shell now – if you wait until something actually breaks, you may find that you can't run the chroot command to get into your shell!

As the rebuild proceeds, your existing 32-bit libraries will be overwritten by 64-bit ones. Since this would break any existing (not-yet-rebuilt) programs depending on them, these libraries should be saved away to a separate location so they'll still be available during the rebuild. Fortunately, Gentoo x86-64 already uses the /lib32 and /usr/lib32 directories for 32-bit libraries, so you can reuse the same paths: mkdir -m0755 /lib32 /usr/lib32 cp -a /lib/*.so* /lib32/ cp -a /usr/lib/* /usr/lib32/ Make sure to use the -a option to cp in order to copy symbolic links as links rather than as regular files. (It may not be necessary to copy directories and other non-library files from /usr/lib, but I did so out of an excess of caution.)

In addition to copying the files themselves, you'll also need to configure the dynamic linker to find the files in those directories. Edit /etc/env.d/99local (or create it if it does not exist) and add a line reading: LDPATH="/lib32:/lib64:/usr/lib32:/usr/lib64:${LDPATH}" and then run the command env-update. (The lib64 directories don't exist yet, but the added entries will be important in the next step.)

6. Rename /lib and /usr/lib to lib64 and install lib -> lib64 symlinks

DANGER: Making an error in this step can leave your system unusable!

Similarly to the lib32 directories, Gentoo x86-64 places native 64-bit libraries in lib64 directories instead of the unnumbered lib directories used by 32-bit systems. To avoid ending up with multiple copies of files or other problems later, we'll rename the existing lib directories now: mv /lib /lib64 && ln -s lib64 /lib mv /usr/lib /usr/lib64 && ln -s lib64 /usr/lib

But wait! Since you can't atomically replace a directory with a symlink, the naive method shown above will break your system when /lib disappears. You could use your recovery shell or an interpreter such as Python, or you can run the ln -s command by invoking the loader directly: /lib64/ld-linux.so.2 /bin/ln -s lib64 /lib Note that the latter method relies on having /lib64 in the dynamic linker's search path, as configured in the previous step. (It may also work to set LD_LIBRARY_PATH=/lib64 for this one command, but I added /lib64 to the search path out of an excess of caution.)

7. Change /etc/portage/make.conf and /etc/portage/make.profile to use the x86-64 architecture

Since we're about to start installing x86-64 software, the Portage configuration data needs to be updated to use the x86-64 architecture. Edit make.conf to change any x86-32 references; for example, if you set CHOST=i686-pc-linux-gnu, change it to CHOST=x86_64-pc-linux-gnu. Then add the line: ABI_X86="32 64" which will cause Portage to build both 32-bit and 64-bit versions of libraries. (If you don't plan on running any prebuilt 32-bit binaries, it might be possible to leave this line out or explicitly set ABI_X86="64".)

You'll also need to update your make.profile link to point to an x86-64 profile instead of a 32-bit x86 profile. For example, if the link currently points to /usr/portage/profiles/default/linux/x86, update it with: rm /etc/portage/make.profile; ln -s /usr/portage/profiles/default/linux/amd64 /etc/portage/make.profile Don't try to use ln -fs to overwrite the link, since that will instead put a link inside the existing profile – which is not what you want!

8. Copy gcc, binutils, and glibc packages from an existing x86-64 Gentoo system

DANGER: Making an error in this step can leave your system unusable!

There seem to be circular dependencies in some of the basic system tools libraries: attempting to build glibc, for example, fails because it assumes the existence of system header files from a previous install of (64-bit) glibc, while the cross-compiler used earlier to compile the kernel seems to have trouble working with a native binutils install. The surest method seems to be to just copy the needed packages out of an existing x86-64 system, such as the Gentoo live DVD.

Important: The packages copied from the 64-bit system (especially libraries) should be the same versions as those installed on the 32-bit system, to avoid confusion from having separate 32-bit and 64-bit files installed in the same location.

8.1. First, clean out the existing x86-64 cross compiler: crossdev -C x86_64 (You may need to set ARCH=x86 in /etc/portage/make.conf to force crossdev to actually remove the packages. If so, remove it again after crossdev completes.)

8.2. Next, on an existing x86-64 system, create a tarball of the files contained in the sys-devel/gcc, sys-devel/binutils, and sys-libs/glibc packages. You'll also need the library dependencies for GCC: dev-libs/gmp, dev-libs/mpc, and dev-libs/mpfr. You can do this with approximately: for p in sys-devel/gcc sys-devel/binutils sys-libs/glibc dev-libs/gmp dev-libs/mpc dev-libs/mpfr; do     cat /var/db/pkg/$p-[0-9]*/CONTENTS done | tar cTf - /tmp/install.tar

8.3. Copy that tarball to the system you're rebuilding, then extract it in a temporary directory: mkdir /tmp/install && tar Cxf /tmp/install /tmp/install.tar

8.4. Remove 32-bit libraries from the extracted files: rm -r /tmp/install/{lib32,usr/lib32,lib64/ld-linux.so.2} In theory, these files should be identical (or effectively so) to the ones already on your system, but since they're not needed for running 64-bit binaries, it's safer to just avoid installing them.

8.5. Install the files using mv: cd /tmp/install && find * -type f | while read f; do mkdir -pm755 "/$(dirname "$f")"; mv "$f" "/$f"; done It is absolutely critical that you use mv and not cp, tar, or other commands! If (as I accidentally did) you use a simple cp -a, the process will crash with a "Bus error" signal (how often do you see those anymore?) when it tries to overwrite /lib/libc-*.so, and you'll be left with a broken libc library that confuses even the 32-bit ld-linux.so.2 and prevents any programs from running at all.

9. Configure gcc and binutils

Gentoo's gcc and binutils wrappers need to be reconfigured to find the newly installed programs. Run gcc-config -l (resp. binutils-config -l) to find the option number for the x86_64 version, then run gcc-config NUMBER (resp. binutils-config NUMBER) to make it active.

It's possible the configuration scripts won't be able to locate the new programs; in that case, look in /etc/env.d/gcc (resp. /etc/env.d/binutils) and follow the format of the existing files to create an entry for the new x86-64 programs.

Once this is done, you should be able to compile and run a simple "hello world" program.

10. Emerge the sys-apps/coreutils package
Both as a check that the compiler toolchain is correctly installed and as a prerequisite for rebuilding glibc, emerge the sys-apps/coreutils package: emerge -1 sys-apps/coreutils

If you don't do this, attempting to emerge glibc will fail because, as a check of the validity of the newly-built glibc, the ebuild tries to execute /bin/date (which is currently a 32-bit executable) with the newly-built loader and libraries (which use the 64-bit architecture).

11. Emerge the packages you overwrote in step 8

Now you can emerge gcc, binutils, and glibc to bring Portage's state back in line with what's installed on the system: FEATURES=-collision-protect emerge -1 sys-devel/binutils dev-libs/gmp dev-libs/mpc dev-libs/mpfr sys-devel/gcc sys-libs/glibc

FEATURES=-collision-protect is required to avoid merge errors for two reasons. First, the new installs will (deliberately) overwrite some 64-bit-specific files you installed earlier. Second (and this will also be true for many packages you emerge later), the packages will overwrite the old 32-bit libraries you saved in /lib32 and /usr/lib32 with equivalent but newly-compiled versions.

12. Emerge Xwindows and drivers (optional)

If you use Xwindows with a non-kernel driver (such as the x11-drivers/nvidia-drivers package), you'll need to rebuild it for the new kernel before you can start X. Since this will produce a 64-bit X module, you'll also need to rebuild the X server itself and any other drivers you use with it (such as xf86-input-keyboard or xf86-input-mouse). To find out what libraries you'll need to build a 64-bit binary of the X server, run ldd /usr/bin/Xorg and work out which package is responsible for each library (by, for example, using the equery program from the app-portage/gentoolkit package: equery b /usr/lib/FILENAME).

13. Emerge the world

At this point, your system is ready to rebuild all installed packages, so run: FEATURES=-collision-protect emerge -e world

Optionally, you can first run emerge -pve world and check that there are no blockers or unexpected new packages. (However, at least as of August 2013, if you have any packages that depend on the emul-linux 32-bit library sets, you may see several new packages because those packages build libraries which were originally distributed as binaries in the emul-linux packages. Gentoo seems to be working on phasing out the emul-linux packages entirely, so this may not be a problem for too long.)

As the rebuild proceeds, programs are likely to break; for example, if a library with no multi-ABI support is rebuilt, any programs which depend on it will stop working until they are also rebuilt. It's generally possible to continue using your system during the rebuild, but if you see any program behaving unexpectedly, it's probably best to stop using that program until it has been rebuilt. (In my case, the Seamonkey browser crashed early on in the build and subsequently refused to start up due to library architecture incompatibilities. I also got "error -2146697208" (0x800C0008) in a Wine-hosted Windows program trying to connect to an SSL website, which was resolved when Wine was rebuilt for the x86-64 architecture.)

14. Remove old libraries

Once all packages have been successfully merged, you will probably have some libraries in /lib32 and /usr/lib32 which no longer belong to any packages. Such libraries may be obsolete, but especially if you have software on your system not managed by Portage (such as programs you've written or downloaded yourself), you should first check that each such library (and any corresponding symlink) is not referenced by anything on your system. You can generate a full list of library dependencies using something like: find / -xdev -type f -executable | while read f; do ldd "$f" 2>/dev/null | sed "s|^|$f:|"; done You can write that output to a file and then grep the file for each library or symlink to ensure that the library is unused. Note that a match doesn't necessarily mean the library cannot be deleted; if the reference is itself from an obsolete library, then both can be deleted.

Note that a file whose timestamp is older than when you started the rebuild might still belong to a package! Binary library files installed via the emul-linux-x86-* packages will be timestamped with the date the file was originally packaged, not the date it was installed. You can check whether each file belongs to a package with a command like "equery b PATHNAME".

15. Reboot

At this point, your system probably has a mixture of 32-bit and 64-bit code running. This isn't a problem in and of itself, but it may hide problems with your newly-installed 64-bit packages, so it's worth rebooting once to ensure that the 64-bit packages are all working as expected.


Kimmo Martimo says:

Regarding the May 2014 note: it seems that it's finally happening soon with the experimental 17.1 profile. Unfortunately I didn't use it though, I used a 17.0 profile.

--

8. Copy gcc, binutils, and glibc packages from an existing x86-64 Gentoo system

The latest Gentoo LiveDVD release is from 2016, so it probably can't be used as a donor anymore. Also I managed to hose the system once by forgetting that while a 2010 Core 2 and a 2018 Ryzen are technically both amd64 processors, they don't support the same set of x86 extensions. A quick recompile with -march=core2 on the donor system of course solved that.

It also seems that GCC also has picked up some additional dependencies. Unfortunately I started this process a day before GCC 8 was stabilized so these apply to GCC 7. I needed to also tar up sys-libs/zlib, sys-apps/acl (and sys-apps/attr as a dependency for sys-apps/acl). The funny thing here is that installing zlib hoses emerge (can be worked around) and SSH (so any further tarballs need to be conveyed via sneakernet) and acl hoses among others mv. This being the case I would recommend to do the whole 64-bit bootstrap package installation using the chroot recovery shell.

Emerge also complained about sys-apps/sandbox, but copying that over didn't seem to help. Thankfully portage seems to work fine without it.

It would be nice to have some command to generate the list of packages to transfer over, but it seems that there is no existing solution for getting just the runtime dependencies of a package except making a Python script using gentoolkit Python classes. I couldn't be bothered, so instead I used trial and error.

--

8.2 Next, on an existing x86-64 system, create a tarball

The "approximately for p in ... done | tar" can be done exactly with this: equery -C files --filter=obj,sym sys-devel/gcc sys-devel/binutils \ sys-libs/glibc dev-libs/gmp dev-libs/mpc dev-libs/mpfr sys-libs/zlib \ sys-apps/acl sys-apps/attr | tar cTf - /tmp/install.tar

--

9. Configure gcc and binutils

gcc-config seemed to work, but produced messages like: * double insanity with c++ and //usr/x86_64-pc-linux-gnu/gcc-bin/7.3.0/c++ The result was a broken compiler, of course.

This was caused by missing symlinks. I had to also install the symlinks with this in step 8.5: (in /tmp/install) find * -type l | while read l; \ do mkdir -pm755 "/$(dirname"$l")"; ln -sn "$(readlink "$l")" "/$l"; done Directories containing only symlinks probably don't exist but I was cautious. After this gcc-config worked normally.

binutils had to be manually massaged like in the guide.

--

10. Emerge the sys-apps/coreutils package

emerge -1 sys-apps/coreutils (Python stack trace) ImportError: libz.so.1: wrong ELF class: ELFCLASS64 This is the aforementioned zlib emerge hosage. LD_LIBRARY_PATH="/lib32:/usr/lib32" emerge -1 sys-apps/coreutils worked around that.

There was another zlib-related error spewed by Python during the merge phase, but despite that the package was merged successfully, so I just ignored it.

--

11. Emerge the packages you overwrote in step 8

This required me to have ABI_X86="32 64" in make.conf

Also I had to do LD_LIBRARY_PATH="/lib32:/usr/lib32" emerge -1 sys-libs/glibc first by itself, since the other packages would not compile with the following error: /usr/lib/gcc/x86_64-pc-linux-gnu/7.3.0/../../../../x86_64-pc-linux-gnu/bin/ld: skipping incompatible /lib/libc.so.6 when searching for /lib/libc.so.6 /usr/lib/gcc/x86_64-pc-linux-gnu/7.3.0/../../../../x86_64-pc-linux-gnu/bin/ld: cannot find /lib/libc.so.6 /usr/lib/gcc/x86_64-pc-linux-gnu/7.3.0/../../../../x86_64-pc-linux-gnu/bin/ld: skipping incompatible /usr/lib/libc_nonshared.a when searching for /usr/lib/libc_nonshared.a /usr/lib/gcc/x86_64-pc-linux-gnu/7.3.0/../../../../x86_64-pc-linux-gnu/bin/ld: cannot find /usr/lib/libc_nonshared.a

However, the emerge will fail because these days they are executing more than /bin/date for testing the new glibc. Specifically they test the following binaries: cal, date, env, free, ls, true, uname, uptime (cal is from sys-apps/util-linux, free and uptime from sys-process/procps, others from sys-apps/coreutils)

The problem is, both sys-apps/util-linux and sys-process/procps want ncurses by default. ncurses is something I didn't particularly want to deal with, so I fixed sys-apps/procps with just: FEATURES="-collision-protect" USE="-ncurses" emerge -1 sys-apps/procps but this didn't work for sys-apps/util-linux for some reason.

At this point my patience was running thin, and since I don't consider cal to be a mission critical application, I worked around the problem with: cp /bin/date /usr/bin/cal

After this I could emerge glibc and after that the other packages just fine.


Andrew Church
Last modified: 2019/3/5