I recently (early 2008) converted two of my personal Linux systems from a custom setup using a self-built packaging tool to Gentoo Linux. (For the curious, the tool can be found here.) Since I didn't want to go through the error-prone hassle of saving and restoring all of the customizations I'd made to the systems, I decided to convert the systems in place. As I was unable to find any guides on how to do this, I ended up having to work out the process through trial and error. This file documents the steps I followed to perform the conversion, as well as the problems I ran into along the way.
Before doing anything else, make sure to back up all critical system directories, including (at a minimum) /etc and /lib. It goes without saying that you should back up your entire system before performing an operation of this magnitude, but keep copies of these directories somewhere convenient as you perform the conversion, so you can restore any files that get inadvertently modified.
In order to make use of Gentoo, one first needs to install the Portage system used by Gentoo for package management. Portage itself is written in Python, and its files can simply be copied into the system; however, the sandboxing utility is distributed as a separate package, which must be compiled and installed as with most C-language programs. Portage also expects a Gentoo-style system layout, provided by Gentoo's sys-apps/baselayout package; however, blindly installing the files from this package can destroy the existing system's configuration, so care is needed when selecting the files to install. Finally, the Portage tree needs to be installed on the system so package information is available.
The scripts I used with my packaging system for installing Portage can be found in the pkg/packager/packages subdirectory of the packager tarball from the above-linked page. The primary package, portage, depends on the three Portage sub-packages; those packages install only to their package-private directories*, and the portage package then copies the necessary files out of those directories into the system itself.
*My packaging system installed all packages to their own directories, placing symbolic links to the files there into /usr/bin, /usr/lib, and so forth. I still prefer this style of filesystem management, as it helps keep each package's files separate (thus reducing the danger of accidental file collisions) and appeals to my own sense of aesthetics.
The Portage configuration files, /etc/make.conf and /etc/portage/*, can be configured as usual for Gentoo. (As of June 2013, make.conf is now also located under the /etc/portage directory.)
Create the directory /etc/env.d if it does not exist, and
save all library directories listed in /etc/ld.so.conf in the file
/etc/env.d/99local. The file should contain one line, of the
(using colons to separate directories as usual).
Initially I did not take this step, and I was quite startled when commands started saying "error while loading shared libraries: libc.so.6: cannot open shared object file: No such file or directory" after reinstalling Portage. It turns out that Portage builds the ld.so.conf file on its own from files in the /etc/env.d directory; since the Gentoo packages for the system libraries are not yet installed, this extra file is needed to help things along until the conversion is complete.
After Portage has been installed, the next step is to use it to
reinstall itself. This both ensures that Portage is functioning properly
and primes the package database for future package operations. Since
Portage's view of the system is currently empty, the --nodeps
(-O: the letter O, not the number 0) flag
is needed to prevent Portage from installing other packages, which would
overwrite those already on the system. For example:
emerge -O1 portage
emerge -O1 sandbox
emerge -O1 baselayout
Additionally, make sure that collision-protect is not listed in the FEATURES variable in /etc/make.conf, since the packages will overwrite the files installed in step 1 above. (You may want to enable collision-protect for the baselayout package, just to ensure it does not overwrite any of your system files; if collisions are reported, you can verify that they are harmless or move any affected files out of the way.)
Now the remaining system packages can be installed. However, at this point Gentoo's packages will begin overwriting files already present on the system. The safest way to proceed is one package at a time: With the collision-protect feature enabled, install (with emerge -1) a single package from the "system" set; if any collisions are reported, check that the collisions are harmless, and rerun the installation with collision-protect disabled. To avoid a single package pulling in other dependencies, you can use the output from emerge -pvD system, installing packages in the order listed.
On the second system I converted, I created a backup directory tree, then moved the files belonging to each package into the backup tree immediately before merging the corresponding Gentoo package, e.g. /bin/cat to /backup/bin/cat. I had intended to ensure that the files continued to be accessible via $PATH, but it seems that some part of the Portage installation process resets the $PATH variable to a default value. I ended up copying necessary programs like cat from the backup tree to /usr/local/bin for use during the installation of the corresponding packages. (For packages like glibc that had to remain in place, I disabled the collision-protect feature after attempting the merge once to verify the list of files to be overwritten.) This method still resulted in problems for a few packages: