Skip to main content

Build and use security hardened images with TripleO

Starting to apply since Pike

Concept of security hardened images

Normally the images used for overcloud deployment in TripleO are not security hardened. It means, the images lack all the extra security measures needed to accomplish with ANSSI requirements. These extra measures are needed to deploy TripleO in environments where security is an important feature.
The following recommendations are given to accomplish with security guidelines:
  • ensure that /tmp is mounted on a separate volume or partition, and that it is mounted with rw,nosuid,nodev,noexec,relatime flags
  • ensure that /var, /var/log and /var/log/audit are mounted on separates volumes or partitions, and that are mounted with rw,relatime flags.
  • ensure that /home is mounted on a separate partition or volume, and that it is mounted with rw,nodev,relatime flags.
  • include extra kernel boot flag to enable auditing: add audit=1 to GRUB_CMDLINE_LINUX setting
  • disable kernel support for USB via bootloader configuration: add nousb to GRUB_CMDLINE_LINUX setting
  • remove unsecure boot flags: remove crashkernel=auto from GRUB_CMDLINE_LINUX setting
  • blacklist insecure modules, preventing those to be loaded: usb-storage, cramfs, freevxfs, jffs2, hfs, hfsplus, squashfs, udf, vfat
  • remove unsecure packages from the image, as they are installed by default: kdump (installed by kexec-tools) and telnet
  • add new package needed for security: screen
To achieve these settings, a new image can be generated and built by TripleO, using what we call "Security hardened image". The produced image will be stored on a new qcow2 file, and can be used to provision all the overcloud servers for TripleO.

How to build the security hardened image

Traditionally, TripleO ships with pre-defined images already built. For the RHEL image, it is available on rhosp-director-images package. However, it is also possible to build your own image, to adapt to your own needs. That is documented on http://tripleo.org/install/basic_deployment/basic_deployment_cli.html

How to build for CentOS 

Additional steps need to be performed in order to build the CentOS image:
  • Install the current-tripleo delorean repository and deps repository:
    sudo curl -L -o /etc/yum.repos.d/delorean.repo https://trunk.rdoproject.org/centos7-master/current-passed-ci/delorean.repo

    sudo curl -L -o /etc/yum.repos.d/delorean-current.repo https://trunk.rdoproject.org/centos7/current/delorean.repo

    sudo sed -i 's/\[delorean\]/\[delorean-current\]/' /etc/yum.repos.d/delorean-current.repo

    sudo /bin/bash -c "cat <<EOF>>/etc/yum.repos.d/delorean-current.repo includepkgs=diskimage-builder,instack,instack-undercloud,os-apply-config,os-collect-config,os-net-config,os-refresh-config,python-tripleoclient,openstack-tripleo-common*,openstack-tripleo-heat-templates,openstack-tripleo-image-elements,openstack-tripleo,openstack-tripleo-puppet-elements,openstack-puppet-modules,openstack-tripleo-ui,puppet-* EOF"

    sudo curl -L -o /etc/yum.repos.d/delorean-deps.repo
    https://trunk.rdoproject.org/centos7/delorean-deps.repo
  • If Ceph needed:
    sudo yum -y install --enablerepo=extras centos-release-ceph-jewel

    sudo sed -i -e 's%gpgcheck=.*%gpgcheck=0%' /etc/yum.repos.d/CentOS-Ceph-Jewel.repo
  • Export the environment variables:
    export DIB_YUM_REPO_CONF="/etc/yum.repos.d/delorean*"
  • If Ceph needed:
    export DIB_YUM_REPO_CONF="$DIB_YUM_REPO_CONF /etc/yum.repos.d/CentOS-Ceph-Jewel.repo"

How to build for RHEL

Additional steps need to be performed in order to build the RHEL image:
  • Get and export the local image that is going to be used:
    export DIB_LOCAL_IMAGE=rhel-guest-image-7.4.x86_64.qcow2
  • Register into the system:
    export REG_METHOD=portal export REG_USER="[your username]"

    export REG_PASSWORD="[your password]"

    # Find this with `sudo subscription-manager list --available` export REG_POOL_ID="[pool id]"

    export REG_REPOS="rhel-7-server-rpms rhel-7-server-extras-rpms rhel-ha-for-rhel-7-server-rpms \ rhel-7-server-optional-rpms rhel-7-server-openstack-12.0-rpms \
    [rhel-7-server-rhceph-2-mon-rpms rhel-7-server-rhceph-2-osd-rpms \
    rhel-7-server-rhceph-2-tools-rpms]"

Common steps

Basically you need to rely on openstack overcloud image build command, to build the image for your needs. In this case,  the security hardened images for the overcloud are contained in a config file called overcloud-hardened-images.yaml. After following all the process documented on OpenStack, you need to execute the command:
openstack overcloud image build --image-name overcloud-hardened-full --config-file /usr/share/openstack-tripleo-common/image-yaml/overcloud-hardened-images.yaml --config-file /usr/share/openstack-tripleo-common/image-yaml/overcloud-hardened-images-[centos7|rhel7].yaml
This will produce the overcloud-hardened-full.qcow2 image, that will contains all the features needed for security. Next you can upload the image to glance and start using it from TripleO.

How to customize the security hardened image

One of the main disadvantages of the security hardened images, is that the partitions for all the filesystems are pre-defined and hardcoded. The pre-defined sizes are:
  • /              -> 6g
  • /tmp           -> 1g
  • /var           -> 7g
  • /var/log       ->5g
  • /var/log/audit -> 900m
  • /home          -> 100m
 Composing an image of 20G . Although this may be ok for most of the deployments, it may be needed to alter this partitioning size, depending on the hardware or environmetn requirements. This is possible with two steps:
  • modify partitioning schema, exporting DIB_BLOCK_DEVICE_CONFIG
  • modify image global size, updating DIB_IMAGE_SIZE var

1. Modify partitioning schema

To modify the partitioning schema, either to alter the partitioning size or create/remove existing partitions, you need to execute: 

export DIB_BLOCK_DEVICE_CONFIG='<yaml_schema_with_partitions>'

Before executing the openstack overcloud image build command. The current YAML used to produce the security hardened image is the following, so you can reuse and update the sizes of the partitions as needed:

export DIB_BLOCK_DEVICE_CONFIG='''
- local_loop:
    name: image0
- partitioning:
    base: image0
    label: mbr
    partitions:
      - name: root
        flags: [ boot,primary ]
        size: 6G
        mkfs:
            type: xfs
            label: "img-rootfs"
            mount:
                mount_point: /
                fstab:
                    options: "rw,relatime"
                    fck-passno: 1
      - name: tmp
        size: 1G
        mkfs:
            type: xfs
            mount:
                mount_point: /tmp
                fstab:
                    options: "rw,nosuid,nodev,noexec,relatime"
      - name: var
        size: 7G
        mkfs:
            type: xfs
            mount:
                mount_point: /var
                fstab:
                    options: "rw,relatime"
      - name: log
        size: 5G
        mkfs:
            type: xfs
            mount:
                mount_point: /var/log
                fstab:
                    options: "rw,relatime"
      - name: audit
        size: 900M
        mkfs:
            type: xfs
            mount:
                mount_point: /var/log/audit
                fstab:
                    options: "rw,relatime"
      - name: home
        size: 100M
        mkfs:
            type: xfs
            mount:
                mount_point: /home
                fstab:
                    options: "rw,nodev,relatime"
'''

For a reference about the YAML schema, please visit https://docs.openstack.org/developer/diskimage-builder/user_guide/building_an_image.html

2. Update image size

Once you modify the partitioning schema, you may need to update the size of the generated image, because the global sum of partition sizes may exceed the one by default (20G). To modify the image size, you may need to update the config files generated to produce the image.
To achieve this , you need to make a copy of the /usr/share/openstack-tripleo-common/image-yaml/overcloud-hardened-images.yaml:

cp /usr/share/openstack-tripleo-common/image-yaml/overcloud-hardened-images.yaml /home/stack/overcloud-hardened-images-custom.yaml

Then you may need to edit the DIB_IMAGE_SIZE setting contained there, to give the right value to it:

...

environment:
DIB_PYTHON_VERSION: '2'
DIB_MODPROBE_BLACKLIST: 'usb-storage cramfs freevxfs jffs2 hfs hfsplus squashfs udf vfat bluetooth'
DIB_BOOTLOADER_DEFAULT_CMDLINE: 'nofb nomodeset vga=normal console=tty0 console=ttyS0,115200 audit=1 nousb'
DIB_IMAGE_SIZE: '20'
COMPRESS_IMAGE: '1'

After creating that new file, you can execute the image build command, pointing to that new generated file:

 openstack overcloud image build --image-name overcloud-hardened-full --config-file /home/stack/overcloud-hardened-images-custom.yaml --config-file /usr/share/openstack-tripleo-common/image-yaml/overcloud-hardened-images-[centos7|rhel7].yaml

It is important to define the right partition sizes for the image, as it will be very hard to resize after deployment (it will need interactions with parted and xfs_growfs). No automatic growth of the filesystem is performed, so the partitions will be fixed independently of disk space, leaving the remaining space of the disk (if existing) not used.

How to upload the security hardened image

The generated image will be a whole disk one. It means, it will not have initrd and vmlinuz files, just the qcow2 one. So once, you have the security hardened image generated, overwrite the original overcloud-full.qcow2 image you had, with the newly generated once.
Then, to upload it, you  need to pass a special flag to the command: 

openstack overcloud image upload --whole-disk --image-path /home/stack/images --update-existing

This will overwrite the original overcloud-full.qcow2 image you had, with the new security hardened image you just generated.

Starting at that point, you can continue with the TripleO deployment as usual, and the deployed overcloud nodes will boot with the partitioned images, that will accomplish with the ANSSI guidelines for security hardening.
Labels:

Comments

Post a Comment

Popular posts from this blog

Enable UEFI PXE boot in Supermicro SYS-E200

When provisioning my Supermicro SYS-E200-8D machines (X10 motherboard), i had the need to enable UEFI boot mode, and provision through PXE. This may seem straightforward, but there is a set of BIOS settings that need to be changed in order to enable it. First thing is to enable EFI on LAN , and enable Network Stack. To do that, enter into BIOS > Advanced > PCIe/PCI/PnP configuration and check that your settings match the following: See that PCI-E have EFI firmware loaded. Same for Onboard LAN OPROM and Onboard Video OPROM. And UEFI Network stack is enabled , as well as IPv4 PXE/IPv6 PXE support. Next thing is to modify boot settings. The usual boot order for PXE is to first add hard disk and second PXE network . The PXE tools (for example Ironic) will set a temporary boot order for PXE (one time) to enable the boot from network, but then the reboot will be done from hard disk. So be sure that your boot order matches the following: See that the first order is hard d
1 comment
Read more

Test API endpoint with netcat

Do you need a simple way to validate that an API endpoint is responsive, but you don't want to use curl? There is a simple way to validate the endpoint with nc, producing an output that can be redirected to a logfile and parsed later: URL=$1 PORT=$2 while true; do     RESULT=$(nc -vz $URL $PORT 2>&1)     DATE=$(date)     echo $DATE $RESULT     sleep 1 done You can all this script with the API URL as first parameter, and API port as the second. netcat will be accessing to that endpoint and will report the results, detecting when the API is down. We also can output the date to have a reference when failures are detected. The produced output will be something like: vie jun 26 08:19:28 UTC 2020 Ncat: Version 7.70 ( https://nmap.org/ncat ) Ncat: Connected to 192.168.111.3:6443. Ncat: 0 bytes sent, 0 bytes received in 0.01 seconds. vie jun 26 08:19:29 UTC 2020 Ncat: Version 7.70 ( https://nmap.org/ncat ) Ncat: Connected to 192.168.111.3:6443. Ncat: 0 bytes sent, 0 bytes
1 comment
Read more

Create and restore external backups of virtual machines with libvirt

A common need for deployments in production, is to have the possibility of taking backups of your working virtual machines, and export them to some external storage. Although libvirt offers the possibility of taking snapshots and restore them, those snapshots are intended to be managed locally, and are lost when you destroy your virtual machines. There may be the need to just trash all your environment, and re-create the virtual machines from an external backup, so this article offers a procedure to achieve it. First step, create an external snapshot So the first step will be taking an snapshot from your running vm. The best way to take an isolated backup is using blockcopy virsh command. So, how to proceed? 1. First you need to extract all the disks that your vm has. This can be achieved with domblklist command:   DISK_NAME=$(virsh domblklist {{domain}} --details | grep 'disk' | awk '{print $3}') This will extract the name of the device that the vm is using
4 comments
Read more