Skip to content

Migration between Linux distributions within the same family

There may be circumstances where it's desirable to migrate from one Linux distribution to another. An example would be systems that are currently deployed with CentOS Stream 8, wishing to migrate to AlmaLinux 8 or Red Hat Enterprise Linux 8.

Migration between families of the same version (eg: '8' or '9') is possible and mostly entails changing repositories and upgrading (or downgrading) packages.

Please be aware that migration may entail package downgrades, which could mean that some packages will be installed with older versions and may require some configuration changes.

Please find below the instructions for your particular setup:

Migration for unmanaged, standalone or locmap-managed machines

As root, run the following commands on the machine to migrate:

The first step is to install the CERN version of the 'release' RPM for the new distribution. The CERN version ensures that your system will utilise the CERN repository infrastructure.

Important: Please take a moment to confirm which command to run (depending on your current distribution, and the distribution that you wish to migrate to

Migrate from Migrate to Command to run
CentOS Stream Red Hat Enterprise Linux dnf --repofrompath='tmpcern,https://linuxsoft.cern.ch/cern/rhel/$releasever/CERN/$basearch/' swap centos-stream-release redhat-release --nogpgcheck
CentOS Stream AlmaLinux dnf --repofrompath='tmpcern,https://linuxsoft.cern.ch/cern/alma/$releasever/CERN/$basearch/' swap centos-stream-release almalinux-release --nogpgcheck
Rocky Linux Red Hat Enterprise Linux dnf --repofrompath='tmpcern,https://linuxsoft.cern.ch/cern/alma/$releasever/CERN/$basearch/' swap rocky-release redhat-release --nogpgcheck
Rocky Linux AlmaLinux dnf --repofrompath='tmpcern,https://linuxsoft.cern.ch/cern/alma/$releasever/CERN/$basearch/' swap rocky-release almalinux-release --nogpgcheck

The commands above use --nogpgcheck. This is to avoid a chicken/egg problem if the host you are migrating from does not already have "CERN" gpgkeys installed

Once the 'new' release RPM has been installed from the above command, it's time to run 'distro-sync' to apply the new distribution content to your system.

Please note that after this initial step, some repositories (such as the openafs ones, if used) may be inaccessible due to a missing DNF variable, which is why those repositories need to be disabled in the subsequent step below (this means you may need to disable other repos if necessary).

Step 1 below will change some repository configuration files, so the second distro-sync in step 2 is necessary to upgrade packages coming from those repositories as well.

The reboot is necessary to use the new kernel and it's associated modules.

Please follow these commands:

  1. dnf distro-sync --disablerepo=openafs*
  2. dnf distro-sync
  3. reboot

If you encounter a 'transaction error' - please refer to the Troubleshooting section below

Migration for CERN IT Puppet-managed machines (VMs or Ironic Bare Metal)

Before starting, it is strongly recommended that you create a new machine with a the desired target distribution image in your hostgroup. This will allow you to verify that your Puppet catalog compiles correctly, that all your dependencies are met and that all functional tests for your service pass. Doing this will significantly improve your chances of a smooth migration.

Once that is complete, you have two migration options:

Create or Rebuild machines

You can use ai-bs to create new machines in your hostgroup:

$ ai-bs --rhel9 -g foohostgroup fooname.cern.ch

Or you can use ai-rebuild to rebuild your existing machines:

$ ai-rebuild --rhel9 fooname.cern.ch

In-place migration

Migration with no reinstall for CERN IT Puppet-managed machines was enabled by the changes in CRM-4443. Here are the steps to follow:

  1. Remove your machine(s) from production, you will be rebooting and your service will be impacted.
  2. In Hiera, set base::migrate::target to either rhel or almalinux for the machine/hostgroup you wish to upgrade.
  3. Run puppet agent -t on your machine(s) twice. This could be done with mco like this:
    $ mco puppet runonce --batch 100 --batch-sleep 60 -t 5 -T lxsoft -F hostgroup_0=lxsoft
    $ mco puppet status  --batch 100 --batch-sleep 60 -t 5 -T lxsoft -F hostgroup_0=lxsoft (repeat until all machines are "Currently idling")
    $ mco puppet runonce --batch 100 --batch-sleep 60 -t 5 -T lxsoft -F hostgroup_0=lxsoft
    
  4. Run systemctl start --wait dnf-distro-sync.service. With mco, this would be:
    $ mco shell run "systemctl start --wait dnf-distro-sync.service" --batch 100 --batch-sleep 60 -t 5 -T lxsoft -F hostgroup_0=lxsoft
    
  5. Reboot.
  6. Once you're happy, add your machine(s) back into production.

The first Puppet run is needed so Facter reports this machine as the 'new' operating system to the Puppet servers, the second Puppet run is the one that actually configures the new repositories. Both of them must complete before running distro-sync.

Migration for non-IT Puppet-managed machines

Administrators of non-IT Puppet managed machines can look at the changes introduced in CRM-4443 for inspiration on what they need to do. In short, here is what is needed:

Puppet module changes

You will need to change your Puppet modules to account for the new distribution and new sets of repos. Detailed instructions are outside the scope of this guide, but you can use the following merge requests as inspiration:

Client changes

On the machines to upgrade:

  1. Replace the contents of /etc/redhat-release with AlmaLinux Release 9.1 (Lime Lynx) or Red Hat Enterprise Linux release 9.1 (Plow) This will "trick" Facter into reporting that machine as a the new target operating system.
  2. Run puppet agent -t twice.
  3. Run dnf swap (refer to the table listed at the top of this page), otherwise the distro-sync won't be able to succeed.
  4. Run dnf distro-sync
  5. Reboot.

Troubleshooting a failed migration

In some circumstances the migration process from 'old' to 'new' may 'fail' due to package conflicts.

If this occurs you should analyse what package has the conflict, remove the package and then reattempt the distro-sync command.

A current example of this is if you are migrating from CentOS Stream 9 to either AlmaLinux or Red Hat Enterprise Linux and have python3-perf installed.

In this case, the RPM transaction test will fail similar to:

Error: Transaction test error:
  file /usr/lib64/python3.9/site-packages/perf-0.1-py3.9.egg-info from install of python3-perf-5.14.0-162.6.1.el9_1.x86_64 conflicts with file from package python3-perf-5.14.0-229.el9.x86_64

The solution in this example would be to:

  1. dnf remove python3-perf
  2. dnf distro-sync