Every devops engineer knows the importance of continuous integration (CI) testing. It’s vital to prevent regressions, as well as maintain performance, security, and supportability. At Bitwise IO, we are experimenting with Kubernetes as an automated CI deployment tool. We like the simplicity of extending tests with deployments on Kubernetes. We think Kubernetes has compelling potential for use in the CI workflow.
Figure 1: The main tools in our CI workflow
This blog post explains how Kubernetes fits into our CI workflow for Hyperledger Sawtooth. We don’t go into detail, but we provide links so you can learn more about Kubernetes and the other tools we use.
Building Sawtooth
Hyperledger Sawtooth uses Jenkins to automate builds of around 20 GitHub repositories. Each new or merged pull request in the master branch initiates a build that contains project-specific tests. The next logical step is to deploy into a test environment.
We have two deployment methods: Debian packages and Docker images. We install the Debian packages inside the Docker deployment image to ensure both that the binaries are tested and that the packages are installable.
Using Docker’s multi-stage build capability, an intermediate build container makes the deployment image smaller and narrows its exposed attack surface (possible vulnerabilities).
Handing off Docker Images with Docker Registry
Jenkins is a great place for build artifacts, but Docker has its own way to easily retrieve images. Docker Registry allows you to push your newly created images and easily retrieve them with Docker, Kubernetes, or anything else that uses the Docker Registry model.
This example shows how to tag an image with the URL of an internal registry, then upload the image to that registry.
$ docker build -f Dockerfile -t registry.url/repo/image_name:${tag} .
$ docker push registry.url/repo/image_name:${tag}
We also use Portus, because Docker Registry does not provide user and access management on its own. The Portus project makes it simple to place an authentication layer over Docker Registry. Now, any authenticated user can pull and deploy the same images that are being deployed into the test environment.
Kubernetes: Simulating Scaled Deployments
Kubernetes excels at creating deployments within and across abstracted infrastructures. We have done our experiments on local (“on-prem”) hardware with a cluster of small Kubernetes nodes dedicated to Sawtooth deployments. Each deployment consists of several pods partitioned in a namespace, which allows us to run multiple networks based on the same deployment file. (Without namespaces, Kubernetes would think we are updating the deployment.) A pod represents a Sawtooth node and contains several containers, each running a specific Sawtooth component: validator, transaction processor, REST API, and so on. Each namespace can have independent quotas for resources such as CPU time, memory, and storage, which prevents a misbehaving network from impacting another network.
Figure 2: Containerized services grouped together in pods.
Because we use Kubernetes, these deployments are portable. We can use them on any cloud infrastructure that supports Kubernetes.
Kubernetes also allows us to scale the number of CI test deployments. With an elastic cloud infrastructure, Kubernetes provides effortless testing on a large number of virtual systems (limited only by the cost of cloud hosting). This solves the issue of limited hardware resources, where each additional deployment will stress existing deployments when they share a node’s resources.
Workload Generation: Deploying Changes Under Load
Deploying Sawtooth is the first step, but you need to give it something to do—better yet, lots to do. Sawtooth includes several workload generators and corresponding transaction processors. In our Kubernetes environment, we deploy intkey_workload and smallbank_workload at rates slightly above what we think the hardware can handle for shorter runs.
Modifying workload rates is as simple as editing the deployment file, changing the rate settings, and reapplying with kubectl. When Kubernetes detects that the pod’s configuration has changed, it terminates the existing workload pod and creates a new one with the changed settings.
This example shows a container definition for an intkey workload pod.
containers:
- name: sawtooth-intkey-workload
image: registry.url/repo/sawtooth-intkey-workload:latest
resources:
limits:
memory: "1Gi"
command:
- bash
args:
- -c
- |
intkey-workload \
--rate 10 \
--urls ...
Retaining Data with Kubernetes: Logging and Graphing
All this testing isn’t much use if you can’t troubleshoot issues when they arise. Kubernetes can streamline deployments, but it can also frustrate your attempts to gather logs and data embedded inside Docker containers after a pod has failed or stopped. Luckily, Sawtooth provides real-time metrics (which we view with Grafana) and remote logging through syslog. We actively collect logs and metrics from the Sawtooth networks, even down to syslog running on the hardware, then carefully match the logging and metrics artifacts to the testing instance. In the end, we can provide a comprehensive set of log data and system metrics for each code change.
Try It Out!
The Sawtooth documentation can help you get started: See Using Kubernetes for Your Development Environment and Kubernetes: Start a Multiple-node Sawtooth Network.
To configure Grafana, see Using Grafana to Display Sawtooth Metrics.
See these links for more information about each tool in our CI workflow:
About the Authors
Richard Berg is a Senior Systems Engineer at Bitwise IO. He has several years’ experience in sysadmin and devops roles. When not behind a terminal, Richard can be found in the woods with his snow-loving adventure cat.
Ben Betts is a Senior Systems Engineer at Bitwise IO. Ben has lots of experience with deploying, monitoring, coordinating, and supporting large systems and with writing long lists of experiences. He only uses Oxford commas because he has to.
