gitlab-runner on docker

Use this template to start your own gitlab-runner on docker.

Setup

1. Configure environment variables (optional):

   cp env.example .env
   # Edit .env to set GITLAB_RUNNER_VERSION if needed
   

2. Configure the runner: Either use the included ./config/config.toml.example and copy it to ./config/config.toml or create your own. By default the container should create a new config.toml file in the ./config directory if it is not present.

3. Register the runner (if using your own config.toml or the default creation): Start the container and attach a shell in the container and run the gitlab-runner register --url https://gitlab.example.com --token your-secret-token command and follow the instructions.

4. Start the runner:

   docker compose up -d
   

If you use docker in docker, make sure to add /var/run/docker.sock:/var/run/docker.sock in your ./config/config.toml, e.g.

# ...
[[runners]]
  name = "runner on stefan-workstation"
  url = "https://gitlab.example.com"
  # ...
  executor = "docker"
  [runners.custom_build_dir]
  [runners.cache]
    MaxUploadedArchiveSize = 0
  [runners.docker]
    # ...
    volumes = ["/cache", "/var/run/docker.sock:/var/run/docker.sock"]
    # ...
  [runners.feature_flags]
    FF_NETWORK_PER_BUILD = true

Important: The FF_NETWORK_PER_BUILD = true feature flag is required when using Docker 29+ or newer GitLab Runner versions to avoid the deprecated container linking mechanism. This flag enables per-job Docker networks instead of using the deprecated --link feature. For more information, see the GitLab Runner Docker Executor documentation.

Troubleshooting

docker buildx bake fails with DNS errors inside BuildKit

Symptom in job logs:

lookup registry.gitlab.example.com on [::1]:53: read: connection refused

The job container can reach the registry (docker login succeeds), but the separate BuildKit container created by docker buildx create --driver docker-container cannot.

Cause: An orphaned buildx_buildkit_* container can keep an empty /etc/resolv.conf from an earlier Docker version or host DNS change. docker buildx rm multi-arch only removes the builder definition; it does not always delete the underlying container. The next docker buildx create may reuse that container with broken DNS.

Fix on the runner host:

# Remove stale builder and its container, then recreate (or re-run the CI job)
docker buildx rm multi-arch 2>/dev/null || true
docker rm -f buildx_buildkit_multi-arch0 2>/dev/null || true

Prevention: Pin DNS in /etc/docker/daemon.json so new containers always get a resolver even when copying the host resolv.conf fails:

{
  "storage-driver": "overlay2",
  "dns": ["10.10.0.1"]
}

Restart Docker after editing: systemctl restart docker, then restart the runner: docker compose up -d.

CI hardening (add before docker buildx create in .gitlab-ci.yml):

- docker buildx rm multi-arch 2>/dev/null || true
- docker rm -f buildx_buildkit_multi-arch0 2>/dev/null || true

failed to decode referrers index with Dependency Proxy (GitLab 17.10)

Symptom

Jobs fail while pulling the job image from the GitLab Dependency Proxy with an error similar to:

failed to decode referrers index: invalid character '<' looking for beginning of value

You may also reproduce it with a manual docker pull (if you are logged in to the registry). The image layers download successfully, and the pull fails at the end.

Why this happens

On fresh Docker Engine 29+ installations, Docker defaults to the containerd image store. This storage backend supports OCI artifacts like SBOMs / attestations and will query the registry for OCI referrers (the /v2/<name>/referrers/<digest> endpoint).

With an outdated GitLab (e.g. 17.10), the Container Registry / Dependency Proxy does not fully support all OCI “referrers” behaviors, and it may respond with a non-JSON error page (HTML, starting with <...). Docker expects an OCI index JSON document and fails parsing it, producing the invalid character '<' error.

The important detail is that this is not primarily an authentication issue: the pull can succeed for layers and still fail when Docker performs the additional referrers lookup.

Fix (keeps Dependency Proxy)

The most reliable workaround is to avoid the containerd image store on the runner host and switch back to the legacy overlay2 image store:

1. Create or edit /etc/docker/daemon.json:

   {
     "storage-driver": "overlay2"
   }
   

   If you also need pinned DNS (see BuildKit section above), combine both settings in the same file.

2. Restart Docker:

   sudo systemctl restart docker
   

3. Verify with:

   docker info | sed -n '1,120p'
   

   You should see Storage Driver: overlay2 (not overlayfs with driver-type: io.containerd.snapshotter.v1).

Background / upstream reports

• Docker docs: containerd image store
• OCI referrers concept and endpoint: ORAS docs “Listing Referrers”
• GitLab issue about Dependency Proxy failing on OCI indexes: https://gitlab.com/gitlab-org/gitlab/-/issues/385217
• GitLab support article showing OCI-index related pull failures and registry behavior: https://support.gitlab.com/hc/en-us/articles/20331100094364-Failed-to-pull-OCI-Image-from-GitLab-Container-Registry

Configuration Files

• compose.yml - Docker Compose configuration file (modern naming, no version field)
• env.example - Example environment variables file (copy to .env to customize)
• config/config.toml.example - Example GitLab Runner configuration