Gelsenkirchen-Erle, pipeline along the Emscher

How we’re building our Kubernetes pipeline in GitLab

Our deployment pipeline helps our developers build, test, and release Docker images to our production Kubernetes. Now we’re migrating the pipeline to GitLab and seizing the chance to refine our tools.

By Tyler Cipriani

Creating a Docker image for your service should be easy—cram your code and its dependencies into a container: boom. done.

But that’s never the whole story.

You have to build new images for each release, monitor them for vulnerabilities, and find a way to safely ship them to production.

You need a reliable process to create, test, and deploy images to Kubernetes. In short: you need a release pipeline.

Wikimedia’s service release pipeline 🚢

A “build and deployment expert” is an antipattern.

Jez Humble & David Farley, Continuous Delivery

Wikimedia has a little more than thirty microservices running atop our in-house Kubernetes infrastructure.

Back when we started moving to Kubernetes in 2017, we had a few aims:

  • Build trust – After you generate an image, build confidence through incremental testing and validation.
  • Streamlined image builds – Developer teams shouldn’t need to be experts to build an excellent image for their service.
  • Security – Build on known-good images, run as a non-root user, and monitor for common vulnerabilities and exposures (CVEs).

And we created two tools to help us achieve these goals:

  1. Blubber – This tool ensures our Docker images are lean, safe, and built from our blessed subset of known-good base images.
  2. PipelineLib – A Jenkins library that uses Blubber to produce, test, and promote images to our Docker registry after establishing trust.

But our migration from Jenkins to GitLab has required some changes to these tools.

Kokkuri: the pipeline from GitLab 🦊

Now we’re migrating to GitLab, we’re replacing Jenkins and PipelineLib with a shared GitLab repository called Kokkuri.

What PipelineLib was for Jenkins, Kokkuri is for GitLab. You can extend Kokkuri jobs in your GitLab project’s `.gitlab-ci.yml` to build streamlined and secure docker images for Wikimedia production, test them, and push them to our production registry.

We’re using this tooling today for two of our internal projects: Scap (our deployment tool for MediaWiki) and Blubber itself.

For now, Kokkuri is an internal tool for Wikimedia’s GitLab. Using it outside of our unique production environment wouldn’t make sense.

Blubber as a BuildKit Frontend 🐳

All of our Wikimedia production services use Blubber to build their Docker images. Blubber is an active, open project—for use both inside and outside Wikimedia 🎉 And as part of the migration to GitLab, we’ve made improvements.

Blubber used to generate opinionated Dockerfiles—now it’s a full-fledged BuildKit front-end. BuildKit is a project from Moby, the people who make Docker, and it’s now used by Docker itself to create images.

This means `docker build` can produce a docker image directly from a `blubber.yaml` file—no Dockerfile necessary 🤯

What’s missing and what’s next 🔜

As with all in-progress migrations: we’re still missing some things.

Here’s what we’re working on next for our GitLab move:

  • Dependency caching – tests will be slow if they need to fetch a lot of dependencies for every run, we’re working on a few solutions and you can follow along on Phabricator.
  • Visibility – we’re still missing all the nice integrations we have in our old systems
    • Links between our bug tracker (Phabricator) and GitLab
    • IRC and Slack notifications—yes, we use both 😅

But why “Kokkuri”? 🦝

An image of a tanuki, a racoon/dog/fox-type thing.
Tanukis: a crucial part of our pipeline.

Alright. Let’s unpack the name “kokkuri.”

Fun fact: the GitLab logo may look like a fox, but it’s a tanuki—a totally real racoon/dog/fox-type thing `{{citation-needed}}`.

Tanukis are the real-life inspiration for a mythical trickster known as a “kokkuri-san”—an animal spirit bringing mischief, magic, and luck.

And to summon a kokkuri-san: you’d use a kokkuri—which is kinda like a Japanese Ouija board.


To summon a mischievous and magical tanuki you use a kokkuri. And now you can summon our tricksy GitLab magic in the exact. same. way.

About this post

Featured image credit: File:Gelsenkirchen-Erle, leiding langs de Emscher IMG 8418 2018-09-01 12.05.jpg by Michielverbeek, licensed under Creative Commons Attribution-Share Alike 4.0 International license.

Tanuki image credit: File:Єнотовидний собака (Nyctereutes procyonoides).jpg by Ryzhkov Sergey, licensed under Creative Commons Attribution-Share Alike 4.0 International license.

Leave a Reply

Your email address will not be published. Required fields are marked *