Skip to main content

Build and use security hardened images with TripleO

Starting to apply since Pike

Concept of security hardened images

Normally the images used for overcloud deployment in TripleO are not security hardened. It means, the images lack all the extra security measures needed to accomplish with ANSSI requirements. These extra measures are needed to deploy TripleO in environments where security is an important feature.
The following recommendations are given to accomplish with security guidelines:
  • ensure that /tmp is mounted on a separate volume or partition, and that it is mounted with rw,nosuid,nodev,noexec,relatime flags
  • ensure that /var, /var/log and /var/log/audit are mounted on separates volumes or partitions, and that are mounted with rw,relatime flags.
  • ensure that /home is mounted on a separate partition or volume, and that it is mounted with rw,nodev,relatime flags.
  • include extra kernel boot flag to enable auditing: add audit=1 to GRUB_CMDLINE_LINUX setting
  • disable kernel support for USB via bootloader configuration: add nousb to GRUB_CMDLINE_LINUX setting
  • remove unsecure boot flags: remove crashkernel=auto from GRUB_CMDLINE_LINUX setting
  • blacklist insecure modules, preventing those to be loaded: usb-storage, cramfs, freevxfs, jffs2, hfs, hfsplus, squashfs, udf, vfat
  • remove unsecure packages from the image, as they are installed by default: kdump (installed by kexec-tools) and telnet
  • add new package needed for security: screen
To achieve these settings, a new image can be generated and built by TripleO, using what we call "Security hardened image". The produced image will be stored on a new qcow2 file, and can be used to provision all the overcloud servers for TripleO.

How to build the security hardened image

Traditionally, TripleO ships with pre-defined images already built. For the RHEL image, it is available on rhosp-director-images package. However, it is also possible to build your own image, to adapt to your own needs. That is documented on

How to build for CentOS 

Additional steps need to be performed in order to build the CentOS image:
  • Install the current-tripleo delorean repository and deps repository:
    sudo curl -L -o /etc/yum.repos.d/delorean.repo

    sudo curl -L -o /etc/yum.repos.d/delorean-current.repo

    sudo sed -i 's/\[delorean\]/\[delorean-current\]/' /etc/yum.repos.d/delorean-current.repo

    sudo /bin/bash -c "cat <<EOF>>/etc/yum.repos.d/delorean-current.repo includepkgs=diskimage-builder,instack,instack-undercloud,os-apply-config,os-collect-config,os-net-config,os-refresh-config,python-tripleoclient,openstack-tripleo-common*,openstack-tripleo-heat-templates,openstack-tripleo-image-elements,openstack-tripleo,openstack-tripleo-puppet-elements,openstack-puppet-modules,openstack-tripleo-ui,puppet-* EOF"

    sudo curl -L -o /etc/yum.repos.d/delorean-deps.repo
  • If Ceph needed:
    sudo yum -y install --enablerepo=extras centos-release-ceph-jewel

    sudo sed -i -e 's%gpgcheck=.*%gpgcheck=0%' /etc/yum.repos.d/CentOS-Ceph-Jewel.repo
  • Export the environment variables:
    export DIB_YUM_REPO_CONF="/etc/yum.repos.d/delorean*"
  • If Ceph needed:
    export DIB_YUM_REPO_CONF="$DIB_YUM_REPO_CONF /etc/yum.repos.d/CentOS-Ceph-Jewel.repo"

How to build for RHEL

Additional steps need to be performed in order to build the RHEL image:
  • Get and export the local image that is going to be used:
    export DIB_LOCAL_IMAGE=rhel-guest-image-7.4.x86_64.qcow2
  • Register into the system:
    export REG_METHOD=portal export REG_USER="[your username]"

    export REG_PASSWORD="[your password]"

    # Find this with `sudo subscription-manager list --available` export REG_POOL_ID="[pool id]"

    export REG_REPOS="rhel-7-server-rpms rhel-7-server-extras-rpms rhel-ha-for-rhel-7-server-rpms \ rhel-7-server-optional-rpms rhel-7-server-openstack-12.0-rpms \
    [rhel-7-server-rhceph-2-mon-rpms rhel-7-server-rhceph-2-osd-rpms \

Common steps

Basically you need to rely on openstack overcloud image build command, to build the image for your needs. In this case,  the security hardened images for the overcloud are contained in a config file called overcloud-hardened-images.yaml. After following all the process documented on OpenStack, you need to execute the command:
openstack overcloud image build --image-name overcloud-hardened-full --config-file /usr/share/openstack-tripleo-common/image-yaml/overcloud-hardened-images.yaml --config-file /usr/share/openstack-tripleo-common/image-yaml/overcloud-hardened-images-[centos7|rhel7].yaml

This will produce the overcloud-hardened-full.qcow2 image, that will contains all the features needed for security. Next you can upload the image to glance and start using it from TripleO.

How to customize the security hardened image

One of the main disadvantages of the security hardened images, is that the partitions for all the filesystems are pre-defined and hardcoded. The pre-defined sizes are:
  • /              -> 6g
  • /tmp           -> 1g
  • /var           -> 7g
  • /var/log       ->5g
  • /var/log/audit -> 900m
  • /home          -> 100m
 Composing an image of 20G . Although this may be ok for most of the deployments, it may be needed to alter this partitioning size, depending on the hardware or environmetn requirements. This is possible with two steps:
  • modify partitioning schema, exporting DIB_BLOCK_DEVICE_CONFIG
  • modify image global size, updating DIB_IMAGE_SIZE var

1. Modify partitioning schema

To modify the partitioning schema, either to alter the partitioning size or create/remove existing partitions, you need to execute: 

export DIB_BLOCK_DEVICE_CONFIG='<yaml_schema_with_partitions>'

Before executing the openstack overcloud image build command. The current YAML used to produce the security hardened image is the following, so you can reuse and update the sizes of the partitions as needed:

- local_loop:
    name: image0
- partitioning:
    base: image0
    label: mbr
      - name: root
        flags: [ boot,primary ]
        size: 6G
            type: xfs
            label: "img-rootfs"
                mount_point: /
                    options: "rw,relatime"
                    fck-passno: 1
      - name: tmp
        size: 1G
            type: xfs
                mount_point: /tmp
                    options: "rw,nosuid,nodev,noexec,relatime"
      - name: var
        size: 7G
            type: xfs
                mount_point: /var
                    options: "rw,relatime"
      - name: log
        size: 5G
            type: xfs
                mount_point: /var/log
                    options: "rw,relatime"
      - name: audit
        size: 900M
            type: xfs
                mount_point: /var/log/audit
                    options: "rw,relatime"
      - name: home
        size: 100M
            type: xfs
                mount_point: /home
                    options: "rw,nodev,relatime"


For a reference about the YAML schema, please visit

2. Update image size

Once you modify the partitioning schema, you may need to update the size of the generated image, because the global sum of partition sizes may exceed the one by default (20G). To modify the image size, you may need to update the config files generated to produce the image.
To achieve this , you need to make a copy of the /usr/share/openstack-tripleo-common/image-yaml/overcloud-hardened-images.yaml:

cp /usr/share/openstack-tripleo-common/image-yaml/overcloud-hardened-images.yaml /home/stack/overcloud-hardened-images-custom.yaml

Then you may need to edit the DIB_IMAGE_SIZE setting contained there, to give the right value to it:


DIB_MODPROBE_BLACKLIST: 'usb-storage cramfs freevxfs jffs2 hfs hfsplus squashfs udf vfat bluetooth'
DIB_BOOTLOADER_DEFAULT_CMDLINE: 'nofb nomodeset vga=normal console=tty0 console=ttyS0,115200 audit=1 nousb'

After creating that new file, you can execute the image build command, pointing to that new generated file:

openstack overcloud image build --image-name overcloud-hardened-full --config-file /home/stack/overcloud-hardened-images-custom.yaml --config-file /usr/share/openstack-tripleo-common/image-yaml/overcloud-hardened-images-[centos7|rhel7].yaml

It is important to define the right partition sizes for the image, as it will be very hard to resize after deployment (it will need interactions with parted and xfs_growfs). No automatic growth of the filesystem is performed, so the partitions will be fixed independently of disk space, leaving the remaining space of the disk (if existing) not used.

How to upload the security hardened image

The generated image will be a whole disk one. It means, it will not have initrd and vmlinuz files, just the qcow2 one. So once, you have the security hardened image generated, overwrite the original overcloud-full.qcow2 image you had, with the newly generated once.
Then, to upload it, you  need to pass a special flag to the command: 

openstack overcloud image upload --whole-disk --image-path /home/stack/images --update-existing

This will overwrite the original overcloud-full.qcow2 image you had, with the new security hardened image you just generated.

Starting at that point, you can continue with the TripleO deployment as usual, and the deployed overcloud nodes will boot with the partitioned images, that will accomplish with the ANSSI guidelines for security hardening.


Popular posts from this blog

Setup an NFS client provisioner in Kubernetes

Setup an NFS client provisioner in Kubernetes One of the most common needs when deploying Kubernetes is the ability to use shared storage. While there are several options available, one of the most commons and easier to setup is to use an NFS server.
This post will explain how to setup a dynamic NFS client provisioner on Kubernetes, relying on an existing NFS server on your systems.
Step 1. Setup an NFS server (sample for CentOS) First thing you will need, of course, is to have an NFS server. This can be easily achieved with some easy steps:

Install nfs package: yum install -y nfs-utils Enable and start nfs service and rpcbind:
systemctl enable rpcbind
systemctl enable nfs-server
systemctl start rpcbind
systemctl start nfs-server
Create the directory that will be shared by NFS, and change the permissions:
mkdir /var/nfsshare
chmod -R 755 /var/nfsshare
chown nfsnobody:nfsnobody /var/nfsshare
 Share the NFS directory over the network, creating the /etc/exports file:
vi /etc/exports
/var/nfsshare …

Create and restore external backups of virtual machines with libvirt

A common need for deployments in production, is to have the possibility of taking backups of your working virtual machines, and export them to some external storage.
Although libvirt offers the possibility of taking snapshots and restore them, those snapshots are intended to be managed locally, and are lost when you destroy your virtual machines.
There may be the need to just trash all your environment, and re-create the virtual machines from an external backup, so this article offers a procedure to achieve it.
First step, create an external snapshot So the first step will be taking an snapshot from your running vm. The best way to take an isolated backup is using blockcopy virsh command. So, how to proceed?

1. First you need to extract all the disks that your vm has. This can be achieved with domblklist command:
DISK_NAME=$(virsh domblklist {{domain}} --details | grep 'disk' | awk '{print $3}')

This will extract the name of the device that the vm is using (vda, hda, et…

Automating local mirrors creation in RHEL

Sometimes there is a need to consume RHEL mirrors locally, not using the Red Hat content delivery network. It may be needed to speed up some deployment, or due to network constraints.

I create an ansible playbook, rhel-local-mirrors (, that can help with that.
What does rhel-local-mirrors do? It is basically a tool that connects to the Red Hat CDN, and syncs the repositories locally, allowing to populate the desired mirrors, that can be accessed by other systems via HTTP.

The playbook is performing several tasks, that can be run together or independently:
register a system on the Red Hat Networkprepare the system to host mirrorscreate the specified mirrorsschedule automatic updates of the mirrors How to use it?It is an Ansible playbook, so start by installing it, in any prefered format. Then continue by cloning the playbook:
git clone playbook expects a group of servers called