Host a local GitLab Runner for homelab Ansible deployments

Prerequisites

#

Before starting, make sure you have:

• Docker and Docker Compose installed on your runner host
• Network connectivity from the runner host to your internal infrastructure
• GitLab instance 15.0+ (registration flow varies; see Step 3 for version-specific guidance)
• SSH keys generated (typically Ed25519; store the private key securely before uploading to CI variables)

Why host a local runner?

#

A local runner is often the cleanest option for homelab automation.

• Private network reachability: Ansible can connect directly to your hosts without complex VPNs or port forwarding.
• No inbound exposure: your runner can access the LAN, but external users cannot access the runner or your hosts.
• Easier secret boundaries: keep vault passwords and SSH keys in GitLab CI variables, and they only exist on the runner during job execution.

Typical architecture

#

flowchart LR
    A[GitLab Repository] --> B[GitLab Pipeline]
    B --> C[Local GitLab Runner in Homelab]
    C --> D[Ansible Playbooks]
    D --> E[Proxmox or Docker Hosts]
    D --> F[NAS and Network Services]
    D --> G[VMs and Containers]

GitLab orchestrates the pipeline, and the local runner does the work on your LAN.

Runner deployment options

#

You can run GitLab Runner in a few ways:

• Docker container: easy upgrades, reproducible, ideal for homelabs.
• Native package on Linux: good if you want direct host integration.
• Kubernetes executor: useful for larger setups, usually overkill for a small homelab.

For Ansible jobs in a homelab, Docker is usually the best balance.

Step 1: Create a dedicated runner in GitLab

#

In GitLab:

1. Go to your project (or group) settings.
2. Open CI/CD then Runners.
3. Create a new runner with:
   • Type: project runner (or group runner if shared across repos).
   • Tags: for example homelab, ansible.
   • Run untagged jobs: disabled (recommended).
   • Locked to current project: enabled for least privilege.

GitLab will provide either a registration command or a token flow, depending on version. Keep the token secret.

Step 2: Run GitLab Runner locally with Docker Compose

#

Create a compose.yml on your runner host:

services:
  gitlab-runner:
    image: gitlab/gitlab-runner:alpine
    container_name: gitlab-runner
    restart: unless-stopped
    volumes:
      - ./config:/etc/gitlab-runner
      # WARNING: Mounting /var/run/docker.sock grants full Docker daemon access.
      # A compromised job can escape the container and gain root-equivalent access
      # to the runner host. This is the most significant security trade-off in this setup.
      #
      # Mitigation options (in order of preference):
      #   1. Rootless Docker (https://docs.docker.com/engine/security/rootless/)
      #      - Runs the Docker daemon as a non-root user
      #      - Limits blast radius of container breakout
      #   2. User namespace remapping
      #      - Maps container root to an unprivileged host UID
      #      - Configure in /etc/docker/daemon.json: {"userns-remap": "default"}
      #   3. Docker-in-Docker (DinD) with separate TLS
      #      - Full isolation but slower and more complex
      #      - See: https://docs.gitlab.com/runner/executors/docker.html#use-docker-in-docker
      #
      # Evaluate whether your threat model justifies the simpler socket mount.
      - /var/run/docker.sock:/var/run/docker.sock:ro
    networks:
      - gitlab-runner-homelab

networks:
  gitlab-runner-homelab:
    driver: bridge

Start it:

docker compose up -d

Step 3: Register the runner

#

Example registration command. Replace the URL and token:

docker exec -it gitlab-runner gitlab-runner register \
  --non-interactive \
  --url "https://gitlab.example.com" \
  --token "YOUR_RUNNER_TOKEN" \
  --executor "docker" \
  --description "homelab-ansible-runner" \
  --docker-image "python:3.12-alpine" \
  --tag-list "homelab,ansible" \
  --run-untagged="false" \
  --locked="true"

This command creates config.toml in ./config.

Step 4: Harden config.toml

#

After registration, review the generated runner config and keep it minimal. Example:

concurrent = 1
check_interval = 0

[[runners]]
  name = "homelab-ansible-runner"
  url = "https://gitlab.example.com"
  token = "REDACTED"
  executor = "docker"
  [runners.docker]
    tls_verify = true                     # REQUIRED for production
    image = "python:3.12-alpine"
    privileged = false
    disable_entrypoint_overwrite = false
    oom_kill_disable = false
    disable_cache = false
    # Option A: Bridge network (recommended for security)
    networks = ["gitlab-runner-homelab"]  # Create this bridge explicitly
                                          # Isolates runner containers from the host
                                          # network. Ansible reaches targets via the
                                          # Docker bridge > host LAN routing.

    # Option B: Host networking (only if justified)
    # network_mode = "host"               # Bypasses all Docker network isolation.
                                          # Job containers share the host's network
                                          # stack entirely. Use ONLY if bridge
                                          # networking prevents reaching your targets
                                          # and you accept full host network exposure.
                                          # Prefer fixing routing/DNS issues instead.
    volumes = ["/cache", "/var/run/docker.sock:/var/run/docker.sock:rw"]
    shm_size = 0                          # Uses default /dev/shm size. Set explicitly
                                          # (e.g., 67108864 for 64MB) only if jobs
                                          # require shared memory beyond the default.

If you choose Option A, the network is already defined in compose.yml from Step 2, so Docker Compose creates it on startup. Restart the runner to apply network changes:

docker compose restart gitlab-runner

Step 5: Prepare CI secrets for Ansible

#

At minimum, define these CI/CD variables in GitLab:

• ANSIBLE_SSH_PRIVATE_KEY (masked, protected)
• ANSIBLE_VAULT_PASSWORD (masked, protected) or a vault token if you use a secret manager
• KNOWN_HOSTS (optional but recommended to enforce host key checking)
• TARGET_HOSTS (comma-separated list of internal hostnames or IPs for ssh-keyscan)

Inventory and secret management

#

Keep the inventory structure in the repository, but encrypt all sensitive values:

inventories/
  prod/
    hosts.yml              # Hostnames, groups, connection vars (no secrets)
    group_vars/
      all/
        vault.yml          # Encrypted with ansible-vault
        vars.yml           # Non-sensitive defaults

Encrypt variable files with ansible-vault encrypt group_vars/all/vault.yml. This gives you:

• The inventory structure (hostnames, grouping) is version-controlled and reviewable.
• Secrets (passwords, API tokens) are vault-encrypted and only decrypted at runtime.
• Code reviews can verify structure without exposing secrets.

For dynamic or short-lived credentials, consider a secrets manager such as HashiCorp Vault or Infisical instead of a static vault password.

Step 6: Add a pipeline for Ansible

#

Example .gitlab-ci.yml:

stages:
  - validate
  - lint
  - deploy

default:
  tags:
    - homelab
    - ansible
  timeout: 45m  # Adjust based on playbook complexity; large deployments may need more

variables:
  ANSIBLE_HOST_KEY_CHECKING: "True"
  ANSIBLE_FORCE_COLOR: "True"
  ANSIBLE_RETRY_FILES_ENABLED: "False"
  ANSIBLE_STDOUT_CALLBACK: "debug"  # More verbose output for debugging

before_script:
  - apk add --no-cache openssh-client ansible
  - mkdir -p ~/.ssh
  - chmod 700 ~/.ssh
  - echo "$ANSIBLE_SSH_PRIVATE_KEY" > ~/.ssh/id_ed25519
  - chmod 600 ~/.ssh/id_ed25519

  # Dynamically fetch known_hosts to avoid stale entries
  # For multiple hosts, separate by comma in TARGET_HOSTS: "host1,host2,host3"
  - ssh-keyscan -H $(echo "$TARGET_HOSTS" | tr ',' '\n') >> ~/.ssh/known_hosts 2>/dev/null || echo "WARNING: Could not scan all hosts"
  - chmod 644 ~/.ssh/known_hosts

validate-infrastructure:
  stage: validate
  script:
    - ansible-vault view --vault-password-file <(echo "$ANSIBLE_VAULT_PASSWORD") inventories/prod/group_vars/all/vault.yml > /dev/null
    - ansible-inventory -i inventories/prod/hosts.yml --list > /dev/null
  rules:
    - if: '$CI_COMMIT_BRANCH == "main"'

ansible-lint:
  stage: lint
  script:
    - pip install ansible-lint==24.2.0  # Pin version
    - ansible-lint playbooks/site.yml --profile production
  rules:
    - if: '$CI_COMMIT_BRANCH == "main"'

deploy-dry-run:
  stage: deploy
  script:
    # Use process substitution to avoid writing vault password to disk
    - ansible-playbook playbooks/site.yml -i inventories/prod/hosts.yml --vault-password-file <(echo "$ANSIBLE_VAULT_PASSWORD") --check --diff
  rules:
    - if: '$CI_COMMIT_BRANCH == "main"'
  allow_failure: true  # Dry run shouldn't block the pipeline

deploy-homelab:
  stage: deploy
  resource_group: homelab-deploy  # Serialize deployments to prevent conflicts
  variables:
    RESOURCE_GROUP_TIMEOUT: 600
  script:
    - |
      # Set up cleanup trap to ensure secrets are removed even on failure
      cleanup() {
        rm -f ~/.ssh/id_ed25519
        # Kill any lingering process substitution file descriptors
        jobs -p | xargs -r kill 2>/dev/null || true
      }
      trap cleanup EXIT
    # Use process substitution consistently - vault password never touches disk
    - ansible-playbook playbooks/site.yml -i inventories/prod/hosts.yml --vault-password-file <(echo "$ANSIBLE_VAULT_PASSWORD")
  rules:
    - if: '$CI_COMMIT_BRANCH == "main"'

Key changes from a naive setup

#

• No sshpass: unnecessary when you use SSH keys, and removing it reduces the image’s attack surface.
• Consistent process substitution: the vault password is passed with <(...) in all jobs, so no .vault_pass file is written to disk.
• Increased timeout: 45 minutes gives large playbooks room to finish.
• Cleanup trap: private SSH keys are removed at job exit, and stray process-substitution processes are cleaned up.

The flow is straightforward:

1. Push to main branch.
2. Local runner picks the job.
3. Ansible executes directly against internal homelab hosts.

Monitoring and observability

#

Once the runner is live, visibility matters as much as deployment logic.

Runner health

#

Check runner status and connectivity:

# Verify the runner is online and registered
docker exec gitlab-runner gitlab-runner verify

# View live runner logs
docker compose logs -f gitlab-runner

# Check running jobs
docker exec gitlab-runner gitlab-runner list

In GitLab, navigate to Settings > CI/CD > Runners to see:

• Online/offline status
• Active job count
• Last contact time

Job alerting

#

Configure pipeline-failure notifications:

• Project-level: Settings > Integrations > Emails on push/pipeline events
• Slack/Mattermost integration: Settings > Integrations > Slack notifications, filtered to pipeline events
• Webhook-based alerts: pipe pipeline status to a webhook (for example, ntfy or Gotify) for push notifications to your phone

Example webhook notification script for failed pipelines:

# .gitlab-ci.yml addition
notify-failure:
  stage: .post
  rules:
    - if: '$CI_COMMIT_BRANCH == "main"'
      when: on_failure
  script:
    - |
      curl -X POST "https://ntfy.example.com/homelab-alerts" \
        -H "Title: Deployment Failed" \
        -H "Tags: warning,gitlab" \
        -d "Pipeline $CI_PIPELINE_URL failed on $CI_COMMIT_BRANCH"
  allow_failure: true

Log retention

#

GitLab keeps job logs by default, but it is worth planning retention:

• Setting a retention policy for old builds if storage is a concern.
• Forwarding runner container logs to a central log store (Loki, Elasticsearch) for long-term auditability.
• Ensuring logs do not accidentally capture secrets. Verify masked variables are properly configured and avoid set -x in scripts that handle credentials.

Backup and recovery

#

Your runner host is a critical automation node. Treat it like any other infrastructure service.

What to back up

#

Backing up config.toml

#

Because config.toml contains the runner token, back it up securely:

# Encrypt and copy to your NAS or backup destination
tar czf - -C ./config . | openssl enc -aes-256-cbc -salt -pbkdf2 -out /backups/gitlab-runner-config-$(date +%F).enc

Store the decryption passphrase in your password manager.

Recovery procedure

#

If the runner host is lost:

1. Provision a new host with Docker and Docker Compose.
2. Restore compose.yml from your Git repository or backup.
3. Restore config.toml from the encrypted backup:

openssl enc -d -aes-256-cbc -pbkdf2 -in /backups/gitlab-runner-config-LATEST.enc | tar xzf - -C ./config

4. Start the runner:

docker compose up -d

5. Verify connectivity:

docker exec gitlab-runner gitlab-runner verify

6. Run a test pipeline with a simple ansible –version job to confirm end-to-end functionality.

If config.toml is also lost and you have no backup, re-register the runner from Step 3. This generates a new token, so update any references in GitLab.

Token rotation

#

If the runner token is compromised:

1. In GitLab: Settings > CI/CD > Runners > select runner > Reset registration token.
2. Re-register the runner with the new token.
3. Update your encrypted backup of config.toml.
4. Audit recent pipeline logs for unauthorized job runs.

Scaling considerations

#

A single runner is fine for most homelabs. As automation grows:

• Multiple tagged runners: run a second runner with different tags, for example build and test, to parallelize non-deployment jobs. Keep the homelab/ansible runner dedicated to deployment.
• Concurrency tuning: increase concurrent in config.toml only if your runner host has the resources. Each concurrent job spawns a separate container.
• GitLab Runner autoscaling (Docker machine): usually overkill for a homelab, but useful to know if CI load grows.

Operational tips that matter in practice

#

• Use one runner for deployment jobs only; keep build/test jobs separate.
• Pin the Ansible version in the CI image to avoid surprise upgrades.
• Add resource_group in deploy jobs to serialize production changes and prevent concurrent deployments.
• Keep idempotency strict in playbooks so reruns are safe.
• Start with --check --diff in a manual stage before automatic rollout.
• Track runner host updates like any other critical infra node and patch within your change window.
• Rotate SSH keys quarterly and update CI variables accordingly. Document the rotation in a runbook.
• Test your cleanup trap (trap cleanup EXIT) in a dry-run job first; failed cleanup can leave secrets in logs.
• If you use multiple target hosts, test ssh-keyscan with your TARGET_HOSTS format before going to production.

Common pitfalls

#

• Runner is online but jobs are stuck: tags do not match job tags.
• SSH fails intermittently: missing host keys or unstable DNS for internal names.
• Random drift after updates: unpinned Ansible collections or roles.
• Token leaks in logs: avoid set -x and mask sensitive variables.
• tls_verify = true fails with private CA: ensure the CA certificate is trusted inside the runner’s Docker image.
• Jobs fail with OOM: check shm_size and container memory limits if playbooks process large inventories.
• Vault password errors in process substitution: ensure your shell supports <(...) (Bash and Zsh do; plain sh does not, so use #!/bin/bash or Alpine’s default Ash with care).

Final take

#

For homelab automation, local GitLab Runner plus Ansible works well. GitLab stays the control plane, and execution stays where your infrastructure actually lives.

You keep private hosts private, gain repeatable deployments, and move from ad-hoc shell sessions to auditable, versioned operations. With monitoring, backups, and solid secret hygiene, this setup scales from a few playbooks to a full infrastructure-as-code workflow.