In this article I will show KubeStack, a python daemon and command line tool to spin un dynamic Jenkins slaves using Kubernetes:
How to install itKubeStack is currently a POC, so it's not still published as a package. To install it, you can clone from the upper url. KubeStack project is divided on three directories:
- ansible: a set of scripts and documentation explaining how to setup a Kubernetes cluster on OpenStack
- images: Dockerfile and scripts used to generate Jenkins slaves
- app: KubeStack application to interact with Kubernetes and Jenkins
You can run the daemon using:
python kubestack/cmd/kubestackd.py (-d if you don't want as a daemon).
Also a CLI tool is available on kubestack/cmd/kubestackcmd.py
How to configureKubeStack relies on a configuration file, that lives on /etc/kubestack/config.yaml
The file has the following format:
- name: jenkins-queue
- name: zmq
- name: dummy-image
demand-listenersKubeStack has configurable listeners to be aware of the demand of slaves. Currently two types of listeners are available: jenkins_queue and gearman.
To configure jenkins_queue there is no extra configuration needed, it will use the settings defined in jenkins section to listen to jenkins queue, to get aware of the demand.
To configure gearman you need to provide host and port settings, that need to point to the Gearman server holding the demand of jobs.
destroy-listenersKubeStack has configurable listeners to be aware of job completion, to disconnect and destroy Jenkins slaves when a job is finished.
Currently only zmq listener is available, that will interact with Jenkins ZMQ plugin (https://github.com/openstack-infra/zmq-event-publisher) . This plugin publishes status of jobs to a ZMQ queue, and KubeStack can listen to this queue, to know about job completion and react according to it.
To configure zmq listener, only host and port of ZMQ need to be provided.
In the future, more destroy-listeners will be added.
jenkinsThis section allows to define the jenkins master settings, where all the jobs will be run, and kubernetes Jenkins slaves will be attached. Following settings need to be provided:
- external_url -> jenkins url, to be able to interact with jenkins api
- internal_url -> jenkins internal ip (if it's different from external), that jenkins slaves will be using to connect with jenkins masters
- user -> username for the jenkins user, to connect to Jenkins API
- password -> password for the jenkins user, to connect to jenkins API
kubernetesThis section allows to define the settings to interact with Kubernetes API. The url and the api key used to connect to the API are needed.
labelsIn this section we will define all the labels (image types) that will be used by our system. Jenkins allows to define labels where certain jobs can be run (for example images of different operating systems, different flavors...)
In this section we can define the same labels as needed by Jenkins. Each label is defined by:
- name -> name of the label, that needs to match with Jenkins label name
- image -> name of the docker image that is used by this label. This needs to be based on a jenkins swarm slave image
- cpu and memory (optional) -> if no resource constraint is defined, kubernetes will spin up pods without constraints, and this can affect performance. To define a jenkins slave, is better to define a minimum cpu and memory needed, that guarantees a proper performance on each slave.
How does it workKubeStack is a daemon that interacts with Kubernetes and Jenkins, listening to demand of jenkins slaves and generating this slaves using a Kubernetes cluster, attaching them to Jenkins.
So each time a new job is spinned up on Jenkins, a demand of an specific label is generated:
Labels can be associated with a given job, using NodeLabel plugin (https://wiki.jenkins-ci.org/display/JENKINS/NodeLabel+Parameter+Plugin):
It attaches to Jenkins using Swarm plugin (https://wiki.jenkins-ci.org/display/JENKINS/Swarm+Plugin) , connecting it as slave, and making it available to run jobs.
Once that Jenkins is aware of the given slave is only, it will execute the job that requested that slave.
Jenkins will publish the status of the job (started, finished...) on ZMQ queue. KubeStack will listen to that queue, and will disconnect the slave and destroy the POD once finished, giving room for more slaves.
KubeStack in the futureThis project is mostly a POC at the moment. More work in terms of configuration, new listeners and reliability need to happen.
Following ideas are in the ROADMAP:
- Scale: KubeStack now relies on a fixed size Kubernetes cluster. This makes it difficult to scale. The idea is to monitor Kubernetes cluster load, and spin up / remove minions depending on needs.
- Jenkins multi-master: currently only one Jenkins master is supported, you need to run different daemons for different masters. Adding multi-master support is a feature scheduled for the future
- Configure jenkins slaves connection: currently the only way to attach jenkins slaves is based on Swarm plugin. In the future, more ways of adding jenkins slaves, giving the ability to configure them flexibility, will be created.
- More demand and destroy listeners: currently we limit to a subset of demand listeners (gearman, jenkins queue) and destroy (zmq). Jenkins has a wide range of plugins that can provide demand and notification of jobs, so more listeners should be added to support it
- Not only jenkins... KubeStack relies on jenkins as the platform to spin up jobs and attach slaves to it. But there are more ways to execute jobs (Ansible, Travis CI... or any custom system you need). KubeStack should support these systems, and be flexible enough to add new systems on demand.