Skip to main content

Add RHEL8 nodes to OpenShift deployments

This blogpost is going to show how to automatically enroll RHEL 8 (realtime) to Openshift 4.1 deployments, using UPI method.
This assumes that you will setup an OpenShift cluster using UPI, following the according documentation:
The procedure on how to spin up this cluster in a semi-automated way is also shown at . This article will assume that you have this UPI cluster up and running.

Enroll RHEL 8 nodes as workers

By default, all nodes added into an OpenShift cluster are based on RHCOS. But there are use caes where you may need RHEL nodes. This is the case of RT (real time) nodes, where you need an specific kernel.
This can be achieved with the help of kickstart, and some specific configuration of PXE kernel args (in this case achieved with matchbox).
We are going to use RHEL8 images, booted with PXE, but we are going to add some specific configuration to allow them to join an existing OCP cluster, using a kickstart file.

RHEL8 pxe images and ISO preparation

The first step before starting the install will be to download the installation source. In order to do it in a disconnected way, you need first to download the RHEL 8 DVD ISO that can be found at (Binary DVD one) .
Once downloaded, you need to mount the ISO and copy the content to the http server that is going to be used (/var/lib/matchbox/assets in our case):

mkdir /tmp/mnt_rhel8/
mount -o loop /tmp/rhel8.iso /tmp/mnt_rhel8/
mkdir /var/lib/matchbox/assets/rhel8
cp -ar /tmp/mnt_rhel8/. /var/lib/matchbox/assets/rhel8/
chmod -R 755 /var/lib/matchbox/assets/rhel8

The pxe images (initrd and vmlinuz) can be found at /var/lib/matchbox/assets/rhel8/images/pxeboot.
This will be useful later for configuring the kickstart file and matchbox profiles.

Kickstart file generation

Kickstart file generation can be easily achieved by using the helper scripts on: . This directory contain several helper scripts for different distros, and specific realtime configs. We are going to focus on , that generates a kickstart file to allow a RHEL8 RT node to join an existing cluster.

Before running this script, we need to know the following settings. The scripts relies on those vars being present at a $HOME/settings_upi.env file, in order to source it. So we will create this file with the following content:
  • CLUSTER_NAME: name of the OCP cluster
  • CLUSTER_DOMAIN: domain for the cluster
  • PULL_SECRET: pull secret that can be extracted from
  • KUBECONFIG_PATH: path to the kubeconfig file that has been generated. When using UPI, it is auth/kubeconfig in the directory where you generated your ignition files.
  • ROOT_PASSWORD: a root password in case you need to login to your server by console
  • RH_USERNAME, RH_PASSWORD, RH_POOL: username/pass and pool id matching your Red Hat subscription. You need to have permissions to grab RHEL 8 repositories with that pool.
  • RHEL_INSTALL_ENDPOINT: path for your uncompressed DVD ISO (if using matchbox it will be http://${PROVISIONING_IP}:8080/assets/rhel8
With this file on place, you can just run the script, and it will generate a rhel8-rt-worker-kickstart.cfg , that will perform the automated install. This file needs to be copied to /var/lib/matchbox/assets directory.

What is kickstart doing?

The generated kickstart is executing a typical RHEL8 unattended install, but also performing those extra tasks:
  • write /etc/profile.env with the subscription data, to be used later for registering the system and subscribing to the right repos
  • create a core user , with root permissions, and adding your pubkey (in $HOME/.ssh/ into authorized keys.
  • write the pull secret (that has been passed in settings_upi.env), into a temporary file, to be used later.
  • write the kubeconfig file (that has been passed in settings_upi.env) to /root/.kube/config file, to be used later.
  • write the ignition endpoint (hardcoded to http://api.$CLUSTER_NAME.$CLUSTER_DOMAIN:2263/config/worker) in a temporary file, to be used later.
  • subscribe the system with the credentials provided, and subscribe to rhel-8-for-x86_64-baseos-rpms, rhel-8-for-x86_64-appstream-rpms and rhocp-4.1-for-rhel-8-x86_64-rpms repos.
  • Install needed packages, dependencies to work as an OpenShift node . These dependencies include packages like cri-o, hyperkube, openshift-clients, etc...
  • Do system adjustments: disabling swap, enabling cri-o, enable ip forwarding, manage selinux cgroups, etc...
  • Grab the content from the previously written ignition endpoint, and store it on a temporary file (/tmp/bootstrap.ign), to be used later.
  • Create a runignition.service, that will be run just one time, that will perform the enrollment of the node.
In the case of real time, enable rhel-8-for-x86_64-rt-rpms repo, and install the RT bits.
After that, reboot the system to allow runignition service to run.

What is runignition service doing?

This service needs to be executed outside kickstart, because it needs to run podman, so it cannot be run inside kickstart chroot. It performs those steps:
  • Gets the version of the cluster performing an oc get clusterversion, using the /root/.kube/config credentials.
  • With this cluster version, downloads the image of machine-config-daemon that matches
  • Uses podman to run this image with machine-config-daemon, passing the content downloaded before (/tmp/bootstrap.ign). This contains all the tasks that a worker needs to perform in order to enroll a cluster.
  • After that is completed, reboot the node
Next time that the node starts, it will join the cluster, and you will have RHEL 8 worker nodes.

How to configure PXE boot for the worker.

This repository is using terraform and matchbox in order to perform the automation of pxe booting. But really any other pxe servers will work. The important thing to be considered are the kernel parameters and pxe images. So, in order to make it work you will need to pass those parameters on PXE:
  • kernel: assets/rhel8_vmlinuz (or full url if needed)
  • initrd: assets/rhel8_initrd.img (or full url if needed)
  • console=tty0 (console=ttyS0,115200n8 , or more if needed)
  • rd.neednet=1
  • inst.ks=http://provisioning_url/assets/rhel8-rt-worker-kickstart.cfg
When using those parameters and booting from PXE, the server will start with RHEL8 PXE images, and will perform an automated installation, with the extra bits described in the previous section.

After it has completed, you may need to approve certs as explained on . You may also need to perform any additional configuration on your cluster, according to the documentation.


Post a Comment

Popular posts from this blog

Setup an NFS client provisioner in Kubernetes

Setup an NFS client provisioner in Kubernetes One of the most common needs when deploying Kubernetes is the ability to use shared storage. While there are several options available, one of the most commons and easier to setup is to use an NFS server.
This post will explain how to setup a dynamic NFS client provisioner on Kubernetes, relying on an existing NFS server on your systems.
Step 1. Setup an NFS server (sample for CentOS) First thing you will need, of course, is to have an NFS server. This can be easily achieved with some easy steps:

Install nfs package: yum install -y nfs-utils Enable and start nfs service and rpcbind:
systemctl enable rpcbind
systemctl enable nfs-server
systemctl start rpcbind
systemctl start nfs-server
Create the directory that will be shared by NFS, and change the permissions:
mkdir /var/nfsshare
chmod -R 755 /var/nfsshare
chown nfsnobody:nfsnobody /var/nfsshare
 Share the NFS directory over the network, creating the /etc/exports file:
vi /etc/exports
/var/nfsshare …

Create and restore external backups of virtual machines with libvirt

A common need for deployments in production, is to have the possibility of taking backups of your working virtual machines, and export them to some external storage.
Although libvirt offers the possibility of taking snapshots and restore them, those snapshots are intended to be managed locally, and are lost when you destroy your virtual machines.
There may be the need to just trash all your environment, and re-create the virtual machines from an external backup, so this article offers a procedure to achieve it.
First step, create an external snapshot So the first step will be taking an snapshot from your running vm. The best way to take an isolated backup is using blockcopy virsh command. So, how to proceed?

1. First you need to extract all the disks that your vm has. This can be achieved with domblklist command:
DISK_NAME=$(virsh domblklist {{domain}} --details | grep 'disk' | awk '{print $3}')

This will extract the name of the device that the vm is using (vda, hda, et…

Automating local mirrors creation in RHEL

Sometimes there is a need to consume RHEL mirrors locally, not using the Red Hat content delivery network. It may be needed to speed up some deployment, or due to network constraints.

I create an ansible playbook, rhel-local-mirrors (, that can help with that.
What does rhel-local-mirrors do? It is basically a tool that connects to the Red Hat CDN, and syncs the repositories locally, allowing to populate the desired mirrors, that can be accessed by other systems via HTTP.

The playbook is performing several tasks, that can be run together or independently:
register a system on the Red Hat Networkprepare the system to host mirrorscreate the specified mirrorsschedule automatic updates of the mirrors How to use it?It is an Ansible playbook, so start by installing it, in any prefered format. Then continue by cloning the playbook:
git clone playbook expects a group of servers called