Add RHEL8 nodes to OpenShift deployments

This blogpost is going to show how to automatically enroll RHEL 8 (realtime) to Openshift 4.1 deployments, using UPI method.
This assumes that you will setup an OpenShift cluster using UPI, following the according documentation: https://docs.openshift.com/container-platform/4.1/installing/installing_bare_metal
The procedure on how to spin up this cluster in a semi-automated way is also shown at https://github.com/redhat-nfvpe/upi-rt . This article will assume that you have this UPI cluster up and running.

Enroll RHEL 8 nodes as workers

By default, all nodes added into an OpenShift cluster are based on RHCOS. But there are use caes where you may need RHEL nodes. This is the case of RT (real time) nodes, where you need an specific kernel.
This can be achieved with the help of kickstart, and some specific configuration of PXE kernel args (in this case achieved with matchbox).
We are going to use RHEL8 images, booted with PXE, but we are going to add some specific configuration to allow them to join an existing OCP cluster, using a kickstart file.

RHEL8 pxe images and ISO preparation

The first step before starting the install will be to download the installation source. In order to do it in a disconnected way, you need first to download the RHEL 8 DVD ISO that can be found at https://access.redhat.com/downloads/content/479/ver=/rhel---8/8.0/x86_64/product-software (Binary DVD one) .
Once downloaded, you need to mount the ISO and copy the content to the http server that is going to be used (/var/lib/matchbox/assets in our case):

mkdir /tmp/mnt_rhel8/
mount -o loop /tmp/rhel8.iso /tmp/mnt_rhel8/
mkdir /var/lib/matchbox/assets/rhel8
cp -ar /tmp/mnt_rhel8/. /var/lib/matchbox/assets/rhel8/
chmod -R 755 /var/lib/matchbox/assets/rhel8

The pxe images (initrd and vmlinuz) can be found at /var/lib/matchbox/assets/rhel8/images/pxeboot.
This will be useful later for configuring the kickstart file and matchbox profiles.

Kickstart file generation

Kickstart file generation can be easily achieved by using the helper scripts on: https://github.com/redhat-nfvpe/upi-rt/tree/master/kickstart . This directory contain several helper scripts for different distros, and specific realtime configs. We are going to focus on https://github.com/redhat-nfvpe/upi-rt/blob/master/kickstart/add_kickstart_for_rhel8_rt.sh , that generates a kickstart file to allow a RHEL8 RT node to join an existing cluster.

Before running this script, we need to know the following settings. The scripts relies on those vars being present at a $HOME/settings_upi.env file, in order to source it. So we will create this file with the following content:

• CLUSTER_NAME: name of the OCP cluster
• CLUSTER_DOMAIN: domain for the cluster
• PULL_SECRET: pull secret that can be extracted from https://cloud.redhat.com/openshift/install/metal/user-provisioned
• KUBECONFIG_PATH: path to the kubeconfig file that has been generated. When using UPI, it is auth/kubeconfig in the directory where you generated your ignition files.
• ROOT_PASSWORD: a root password in case you need to login to your server by console
• RH_USERNAME, RH_PASSWORD, RH_POOL: username/pass and pool id matching your Red Hat subscription. You need to have permissions to grab RHEL 8 repositories with that pool.
• RHEL_INSTALL_ENDPOINT: path for your uncompressed DVD ISO (if using matchbox it will be http://${PROVISIONING_IP}:8080/assets/rhel8

With this file on place, you can just run the add_kickstart_for_rhel_rt.sh script, and it will generate a rhel8-rt-worker-kickstart.cfg , that will perform the automated install. This file needs to be copied to /var/lib/matchbox/assets directory.

What is kickstart doing?

The generated kickstart is executing a typical RHEL8 unattended install, but also performing those extra tasks:

• write /etc/profile.env with the subscription data, to be used later for registering the system and subscribing to the right repos
• create a core user , with root permissions, and adding your pubkey (in $HOME/.ssh/id_rsa.pub) into authorized keys.
• write the pull secret (that has been passed in settings_upi.env), into a temporary file, to be used later.
• write the kubeconfig file (that has been passed in settings_upi.env) to /root/.kube/config file, to be used later.
• write the ignition endpoint (hardcoded to http://api.$CLUSTER_NAME.$CLUSTER_DOMAIN:2263/config/worker) in a temporary file, to be used later.
• subscribe the system with the credentials provided, and subscribe to rhel-8-for-x86_64-baseos-rpms, rhel-8-for-x86_64-appstream-rpms and rhocp-4.1-for-rhel-8-x86_64-rpms repos.
• Install needed packages, dependencies to work as an OpenShift node . These dependencies include packages like cri-o, hyperkube, openshift-clients, etc...
• Do system adjustments: disabling swap, enabling cri-o, enable ip forwarding, manage selinux cgroups, etc...
• Grab the content from the previously written ignition endpoint, and store it on a temporary file (/tmp/bootstrap.ign), to be used later.
• Create a runignition.service, that will be run just one time, that will perform the enrollment of the node.

In the case of real time, enable rhel-8-for-x86_64-rt-rpms repo, and install the RT bits.
After that, reboot the system to allow runignition service to run.

What is runignition service doing?

This service needs to be executed outside kickstart, because it needs to run podman, so it cannot be run inside kickstart chroot. It performs those steps:

• Gets the version of the cluster performing an oc get clusterversion, using the /root/.kube/config credentials.
• With this cluster version, downloads the image of machine-config-daemon that matches
• Uses podman to run this image with machine-config-daemon, passing the content downloaded before (/tmp/bootstrap.ign). This contains all the tasks that a worker needs to perform in order to enroll a cluster.
• After that is completed, reboot the node

Next time that the node starts, it will join the cluster, and you will have RHEL 8 worker nodes.

How to configure PXE boot for the worker.

This repository is using terraform and matchbox in order to perform the automation of pxe booting. But really any other pxe servers will work. The important thing to be considered are the kernel parameters and pxe images. So, in order to make it work you will need to pass those parameters on PXE:

• kernel: assets/rhel8_vmlinuz (or full url if needed)
• initrd: assets/rhel8_initrd.img (or full url if needed)
• console=tty0 (console=ttyS0,115200n8 , or more if needed)
• rd.neednet=1
• inst.ks=http://provisioning_url/assets/rhel8-rt-worker-kickstart.cfg

When using those parameters and booting from PXE, the server will start with RHEL8 PXE images, and will perform an automated installation, with the extra bits described in the previous section.


After it has completed, you may need to approve certs as explained on https://docs.openshift.com/container-platform/4.1/installing/installing_bare_metal/installing-bare-metal.html#installation-approve-csrs_installing-bare-metal . You may also need to perform any additional configuration on your cluster, according to the documentation.