The Ops Community ⚙️

Arseny Zinchenko
Arseny Zinchenko

Posted on • Originally published at rtfm.co.ua on

Nexus: Configuring Docker proxy repository, and ContainerD in Kubernetes

#devops #docker #kubernetes #tutorials

I wrote about launching Nexus in the Nexus: launch in Kubernetes, and PyPI caching repository configuration post, now I want to add Docker image caching to PyPI, especially since Docker Hub introduces new limits from April 1, 2025 — see Docker Hub usage and limits.

We’ll do it as usual: first run manually locally on a work laptop to see how it works in general, and then add a config for the Helm-chart, deploy it to Kubernetes, and see how to configure ContainerD to use this mirror.

Content

  • Running Sonatype Nexus locally with Docker
  • Creating a Docker cache repository
  • Enabling the Docker Security Realm
  • Checking the Docker mirror
  • ContainerD and registry mirrors
  • Nexus Helm chart values and Kubernetes
  • Ingress/ALB for Nexus Docker cache
  • Configuring ContainerD mirror on Kubernetes WorkerNodes
  • Karpenter and the config for ContainerD
  • Useful links

Running Sonatype Nexus locally with Docker

Create a local directory for Nexus data so that the data is saved when Docker restarts, change the user, because otherwise you’ll catch type errors:

...
mkdir: cannot create directory '../sonatype-work/nexus3/log': Permission denied
mkdir: cannot create directory '../sonatype-work/nexus3/tmp': Permission denied
...
Enter fullscreen mode Exit fullscreen mode

Run:

$ mkdir /home/setevoy/Temp/nexus3-data/
$ sudo chown -R 200:200 /home/setevoy/Temp/nexus3-data/
Enter fullscreen mode Exit fullscreen mode

Start Nexus with this directory, and add ports forwarding to access the Nexus instance itself, and to the Docker registry which we will create later:

$ docker run -p 8080:8081 -p 8092:8092 --name nexus3 --restart=always -v /home/setevoy/Temp/nexus3-data:/nexus-data sonatype/nexus3
Enter fullscreen mode Exit fullscreen mode

Wait for everything to start, and get the password for admin:

$ docker exec -ti nexus3 cat /nexus-data/admin.password d549658c-f57a-4339-a589-1f244d4dd21b
Enter fullscreen mode Exit fullscreen mode

Go to the http://localhost:8080 in a browser, and log in to the system:

You can skip the Setup wizard, or quickly click “Next” and set a new admin password.

Creating a Docker cache repository

Go to Administration > Repository > Repositories:

Click Create repository:

Select the docker (proxy) type:

Specify a name, an HTTP port on which connections from the Docker daemon will be accepted, and enable Anonymous Docker pulls:

Next, set the address from which we will pull original images — https://registry-1.docker.io, the rest of the parameters can be left unchanged for now:

You can also create an docker (group), where a single connector for several repositories in Nexus is configured. But I don't need it yet, although I might in the future.

See Grouping Docker Repositories and Using Nexus OSS as a proxy/cache for Docker images.

Enabling the Docker Security Realm

Although we don’t use authentication, we need to enable the realm.

Go to Security > Realms, add Docker Bearer Token Realm:

Checking the Docker mirror

Find an IP of the container running Nexus:

$ docker inspect nexus3
...
        "NetworkSettings": {
           ..
            "Networks": {
              ...
                    "IPAddress": "172.17.0.2",
...
Enter fullscreen mode Exit fullscreen mode

Previously, we’ve set the port for Docker cache in Nexus to 8092.

On the workstation, edit the /etc/docker/daemon.json file, set registry-mirrors and insecure-registries, since we don't have SSL:

{
        "insecure-registries": ["http://172.17.0.2:8092"],
        "registry-mirrors": ["http://172.17.0.2:8092"]
}
Enter fullscreen mode Exit fullscreen mode

Restart the local Docker service:

$ systemctl restart docker
Enter fullscreen mode Exit fullscreen mode

Run docker info to verify that the changes have been applied:

$ docker info
...
 Insecure Registries:
  172.17.0.2:8092
  ::1/128
  127.0.0.0/8
 Registry Mirrors:
  http://172.17.0.2:8092/
Enter fullscreen mode Exit fullscreen mode

Run docker pull nginx - the request should go through Nexus, and the data should be saved there:

If the data does not appear, there is most likely a problem with authentication.

To check, add debug=true to the /etc/docker/daemon.json:

{
        "insecure-registries": ["http://172.17.0.2:8092"],
        "registry-mirrors": ["http://172.17.0.2:8092"],
        "debug": true
}
Enter fullscreen mode Exit fullscreen mode

Restart the local Docker, run a docker pull and look at the logs with journalctl -u docker:

$ sudo journalctl -u docker --no-pager -f
...
level=debug msg="Trying to pull rabbitmq from http://172.17.0.2:8092/"
level=info msg="Attempting next endpoint for pull after error: Head "http://172.17.0.2:8092/v2/library/rabbitmq/manifests/latest": unauthorized: "
level=debug msg="Trying to pull rabbitmq from https://registry-1.docker.io"
...
Enter fullscreen mode Exit fullscreen mode

The first time I forgot to enable Docker Bearer Token Realm.

And the second time, I had a token for the https://index.docker.io saved in my local ~/.docker/config.json, and Docker tried to use it. In this case, you can simply delete/move the config.json and perform the pull again.

Okay.

What about ContainerD? Because we don’t have Docker in AWS Elastic Kubenretes Service.

ContainerD and registry mirrors

To be honest, containerd was more painful than Nexus. There is its TOML for configurations, and different versions of ContainerD itself and configurations, and deprecated parameters.

This format makes my eyes hurt:

[plugins]
  [plugins.'io.containerd.cri.v1.runtime']
    [plugins.'io.containerd.cri.v1.runtime'.containerd]
      [plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes]
        [plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes.runc]
          [plugins.'io.containerd.cri.v1.runtime'.containerd.runtimes.runc.options]
Enter fullscreen mode Exit fullscreen mode

Anyway, it worked, so here we go.

Let’s do it locally first, and then we’ll set it up in Kubernetes.

On Arch Linux, we install with pacman:

$ sudo pacman -S containerd crictl
Enter fullscreen mode Exit fullscreen mode

Generate a default configuration for the containerd:

$ sudo mkdir /etc/containerd/
$ containerd config default | sudo tee /etc/containerd/config.toml
Enter fullscreen mode Exit fullscreen mode

In the /etc/containerd/config.toml file, add the necessary parameters for mirrors:

...
[plugins]
  [plugins.'io.containerd.cri.v1.images']
...
    [plugins.'io.containerd.cri.v1.images'.registry]
      [plugins.'io.containerd.cri.v1.images'.registry.mirrors."docker.io"]
        endpoint = ["http://172.17.0.2:8092"]
...
Enter fullscreen mode Exit fullscreen mode

Here:

  • plugins.'io.containerd.cri.v1.images: parameters for image service, images management
  • registry: registry settings for images service
  • mirrors."docker.io": mirrors for the docker.io
  • endpoint: where to go when you need to pull an image from the docker.io

Restart containerd:

$ sudo systemctl restart containerd
Enter fullscreen mode Exit fullscreen mode

Check that the new settings have been applied:

$ containerd config dump | grep -A 1 mirrors
      [plugins.'io.containerd.cri.v1.images'.registry.mirrors]
        [plugins.'io.containerd.cri.v1.images'.registry.mirrors.'docker.io']
          endpoint = ['http://172.17.0.2:8092']
Enter fullscreen mode Exit fullscreen mode

Run crictl pull:

$ sudo crictl pull ubuntu
Enter fullscreen mode Exit fullscreen mode

And check the Nexus:

The Ubuntu image data appears.

Everything seems to be working here, so let’s try to configure the whole thing in Kubernetes.

Nexus Helm chart values and Kubernetes

In the Adding a repository to Nexus using Helm chart values part of my previous post, I wrote a bit about how and what values were added to run from the Nexus Helm chart in Kubernetes for the PyPI cache.

Let’s update them a bit:

  • add a separate blob store: connect a separate persistentVolume, because the default one is only 8 gigabytes, and if this is more or less enough for PyPI cache, it will not be enough for Docker images
  • add additionalPorts: here we set the port on which the Docker cache will service requests
  • enable Ingress

All values are values.yaml.

I deploy from Terraform when setting up a Kubernetes cluster.

All together with PyPI looks like this:

resource "helm_release" "nexus" {
  namespace = "ops-nexus-ns"
  create_namespace = true

  name = "nexus3"
  repository = "https://stevehipwell.github.io/helm-charts/"
  #repository_username = data.aws_ecrpublic_authorization_token.token.user_name
  #repository_password = data.aws_ecrpublic_authorization_token.token.password
  chart = "nexus3"
  version = "5.7.2"
  # also:
  # Environment:
  # INSTALL4J_ADD_VM_PARAMS: -Djava.util.prefs.userRoot=${NEXUS_DATA}/javaprefs -Xms1024m -Xmx1024m -XX:MaxDirectMemorySize=2048m
  values = [
    <<-EOT
      # use existing Kubernetes Secret with admin's password
      rootPassword:
        secret: nexus-root-password
        key: password

      # enable storage
      persistence:
        enabled: true
        storageClass: gp2-retain

      # create additional PersistentVolume to store Docker cached data
      extraVolumes:
        - name: nexus-docker-volume
          persistentVolumeClaim:
            claimName: nexus-docker-pvc

      # mount the PersistentVolume into the Nexus' Pod
      extraVolumeMounts:
        - name: nexus-docker-volume
          mountPath: /data/nexus/docker-cache

      resources:
        requests:
          cpu: 100m
          memory: 2000Mi
        limits:
          cpu: 500m
          memory: 3000Mi

      # enable to collect Nexus metrics to VictoriaMetrics/Prometheus
      metrics:
        enabled: true
        serviceMonitor:
          enabled: true

      # use dedicated ServiceAccount
      # still, EKS Pod Identity isn't working (yet?)
      serviceAccount:
        create: true
        name: nexus3
        automountToken: true

      # add additional TCP port for the Docker caching listener
      service:
        additionalPorts:
          - port: 8082
            name: docker-proxy
            containerPort: 8082
            hosts:
              - nexus-docker.ops.example.co

      # to be able to connect from Kubernetes WorkerNodes, we have to have a dedicated AWS LoadBalancer, not only Kubernetes Service with ClusterIP
      ingress:
        enabled: true
        annotations:
          alb.ingress.kubernetes.io/group.name: ops-1-30-internal-alb
          alb.ingress.kubernetes.io/target-type: ip
        ingressClassName: alb
        hosts:
        - nexus.ops.example.co

      # define the Nexus configuration
      config:
        enabled: true
        anonymous:
          enabled: true

        blobStores:

          # local EBS storage; 8 GB total default size ('persistence' config above)
          # is attached to a repository in the 'repos.pip-cache' below
          - name: default
            type: file
            path: /nexus-data/blobs/default
            softQuota:
              type: spaceRemainingQuota
              limit: 500

          # dedicated sorage for PyPi caching
          - name: PyPILocalStore
            type: file
            path: /nexus-data/blobs/pypi
            softQuota:
              type: spaceRemainingQuota
              limit: 500              

          # dedicated sorage for Docker caching
          - name: DockerCacheLocalStore
            type: file
            path: /data/nexus/docker-cache
            softQuota:
              type: spaceRemainingQuota
              limit: 500

        # enable Docker Bearer Token Realm
        realms:
          enabled: true
          values:
            - NexusAuthenticatingRealm
            - DockerToken

        # cleanup policies for Blob Storages
        # is attached to epositories below
        cleanup:
          - name: CleanupAll
            notes: "Cleanup content that hasn't been updated in 14 days downloaded in 28 days."
            format: ALL_FORMATS
            mode: delete
            criteria:
              isPrerelease:
              lastBlobUpdated: "1209600"
              lastDownloaded: "2419200"

        repos:

          - name: pip-cache
            format: pypi
            type: proxy
            online: true
            negativeCache:
              enabled: true
              timeToLive: 1440
            proxy:
              remoteUrl: https://pypi.org
              metadataMaxAge: 1440
              contentMaxAge: 1440
            httpClient:
              blocked: false
              autoBlock: true
              connection:
              retries: 0
              useTrustStore: false
            storage:
              blobStoreName: default
              strictContentTypeValidation: false
            cleanup:
              policyNames:
                - CleanupAll

          - name: docker-cache
            format: docker
            type: proxy
            online: true
            negativeCache:
              enabled: true
              timeToLive: 1440
            proxy:
              remoteUrl: https://registry-1.docker.io
              metadataMaxAge: 1440
              contentMaxAge: 1440
            httpClient:
              blocked: false
              autoBlock: true
              connection:
              retries: 0
              useTrustStore: false
            storage:
              blobStoreName: DockerCacheLocalStore
              strictContentTypeValidation: false
            cleanup:
              policyNames:
                - CleanupAll
            docker:
              v1Enabled: false
              forceBasicAuth: false
              httpPort: 8082
            dockerProxy:
              indexType: "REGISTRY"
              cacheForeignLayers: "true"
    EOT
  ]
}
Enter fullscreen mode Exit fullscreen mode

Deploy, open the port:

$ kk -n ops-nexus-ns port-forward svc/nexus3 8081
Enter fullscreen mode Exit fullscreen mode

Check the Realm:

Check the Docker repository itself:

Good.

Ingress/ALB for Nexus Docker cache

Since ContainerD on EC2 is not a Kubernetes service, it does not have access to Kubernetes Service from ClusterIP. Accordingly, it will not be able to pull images from port 8082 to nexus3.ops-nexus-ns.svc.cluster.local.

In the Helm chart, it is possible to create a separate Ingress and set all the parameters:

...
      # to be able to connect from Kubernetes WorkerNodes, we have to have a dedicated AWS LoadBalancer, not only Kubernetes Service with ClusterIP
      ingress:
        enabled: true
        annotations:
          alb.ingress.kubernetes.io/group.name: ops-1-30-internal-alb
          alb.ingress.kubernetes.io/target-type: ip
        ingressClassName: alb
        hosts:
        - nexus.ops.example.co
...
Enter fullscreen mode Exit fullscreen mode

I use the alb.ingress.kubernetes.io/group.name annotation to combine multiple Kubernetes Ingresses through a single AWS LoadBalancer, see Kubernetes: a single AWS Load Balancer for different Kubernetes Ingresses.

An important note here: in the Ingress parameters, you do not need to add ports and hosts that are specified in the Service.

That is, for the:

...
      # add additional TCP port for the Docker caching listener
      service:
        additionalPorts:
          - port: 8082
            name: docker-proxy
            containerPort: 8082
            hosts:
              - nexus-docker.ops.example.co
...
Enter fullscreen mode Exit fullscreen mode

A route to Ingress will be automatically created in the Helm chart:

$ kk -n ops-nexus-ns get ingress nexus3 -o yaml
...
spec:
  ingressClassName: alb
  rules:
  - host: nexus.ops.example.co
    http:
      paths:
      - backend:
          service:
            name: nexus3
            port:
              name: http
        path: /
        pathType: Prefix
  - host: nexus-docker.ops.example.co
    http:
      paths:
      - backend:
          service:
            name: nexus3
            port:
              name: docker-proxy
        path: /
        pathType: Prefix
Enter fullscreen mode Exit fullscreen mode

See ingress.yaml.

Next.

Configuring ContainerD mirror on Kubernetes WorkerNodes

We have already seen it locally, and in an AWS EKS cluster basically everything is the same.

The only thing is that locally we have a v2.0.3 version, and in the AWS EKS it’s the 1.7.25, so the configuration format will be slightly different.

On AWS EKS WorkerNode/EC2, check the /etc/containerd/config.toml file:

[plugins."io.containerd.grpc.v1.cri".registry]
config_path = "/etc/containerd/certs.d:/etc/docker/certs.d"
...
Enter fullscreen mode Exit fullscreen mode

So far, we’re adding a new mirror here manually — this is where there are some differences from what we did locally. See examples in Configure Registry Credentials Example — GCR with Service Account Key Authentication.

That is, for containerd version 1 — configuration version == 2, and for containerd version 2 — configuration version == 3…? Okay, man…

On an EC2, the config will look like this:

[plugins."io.containerd.grpc.v1.cri".registry]
  [plugins."io.containerd.grpc.v1.cri".registry.mirrors]
    [plugins."io.containerd.grpc.v1.cri".registry.mirrors."docker.io"]
      endpoint = ["http://nexus-docker.ops.example.co"]
Enter fullscreen mode Exit fullscreen mode

We don’t specify the port, because AWS ALB will handle this through the hostname, which will route the request to the correct Listener.

Restart the service:

[root@ip-10-0-32-218 ec2-user]# systemctl restart containerd
Enter fullscreen mode Exit fullscreen mode

Perform a pull of some image:

[root@ip-10-0-46-186 ec2-user]# crictl pull nginx
Enter fullscreen mode Exit fullscreen mode

Let’s check the Nexus:

Everything is there.

It remains to add the ContainerD settings when creating a server with Karpenter.

Karpenter and the config for ContainerD

My EC2NodeClasses for Karpenter are created in Terraform, see Terraform: building EKS, part 3 — Karpenter installation.

Of course, it is better to perform all these operations on a test environment, or create a separate NodeClass and NodePool.

In my case, the ~ec2-user/.ssh/authorized_keys file for SSH is configured there via a AWS EC2 UserData (see AWS: Karpenter and SSH for Kubernetes WorkerNodes), and here we also can add the creation of a file for ContainerD mirror.

By default, we have seen that containerd will check the following directories:

...
[plugins."io.containerd.grpc.v1.cri".registry] 
  config_path = "/etc/containerd/certs.d:/etc/docker/certs.d"
...
Enter fullscreen mode Exit fullscreen mode

So, we can add a new file there.

But do you remember that joke — “Now forget everything you learned at university”?

Well, forget everything we did with containerd configurations above, because it's already deprecated. Now it's cool and trendy to work with the Registry Host Namespace.

The idea is that a directory for the registry is created in /etc/containerd/certs.d with the hosts.toml file, and the registry settings are described in that file.

In our case, it will look like this:

[root@ip-10-0-45-117 ec2-user]# tree /etc/containerd/certs.d/
/etc/containerd/certs.d/
└── docker.io
    └── hosts.toml
Enter fullscreen mode Exit fullscreen mode

And the hosts.toml file's content:

server = "https://docker.io"

[host."http://nexus-docker.ops.example.co"]
  capabilities = ["pull", "resolve"]
Enter fullscreen mode Exit fullscreen mode

Ok. We describe this whole thing in the UserData of our testing EC2NodeClass.

So here is our “favorite” YAML — I’ll give you the entire configuration to avoid problems with indents, because I got a little headache with it:

resource "kubectl_manifest" "karpenter_node_class_test_latest" {
  yaml_body = <<-YAML
    apiVersion: karpenter.k8s.aws/v1
    kind: EC2NodeClass
    metadata:
      name: class-test-latest
    spec:
      kubelet:
        maxPods: 110
      blockDeviceMappings:
        - deviceName: /dev/xvda
          ebs:
            volumeSize: 40Gi
            volumeType: gp3
      amiSelectorTerms:
        - alias: al2@latest
      role: ${module.eks.eks_managed_node_groups["${local.env_name_short}-default"].iam_role_name}
      subnetSelectorTerms:
        - tags:
            karpenter.sh/discovery: "atlas-vpc-${var.aws_environment}-private"
      securityGroupSelectorTerms:
        - tags:
            karpenter.sh/discovery: ${var.env_name}
      tags:
        Name: ${local.env_name_short}-karpenter
        nodeclass: test
        environment: ${var.eks_environment}
        created-by: "karpenter"
        karpenter.sh/discovery: ${module.eks.cluster_name}
      userData: |
        #!/bin/bash
        set -e

        mkdir -p ~ec2-user/.ssh/
        touch ~ec2-user/.ssh/authorized_keys
        echo "${var.karpenter_nodeclass_ssh}" >> ~ec2-user/.ssh/authorized_keys
        chmod 600 ~ec2-user/.ssh/authorized_keys
        chown -R ec2-user:ec2-user ~ec2-user/.ssh/

        mkdir -p /etc/containerd/certs.d/docker.io

        cat <<EOF | tee /etc/containerd/certs.d/docker.io/hosts.toml

        server = "https://docker.io"

        [host."http://nexus-docker.ops.example.co"]
          capabilities = ["pull", "resolve"]
        EOF

        systemctl restart containerd
  YAML

  depends_on = [
    helm_release.karpenter
  ]
}
Enter fullscreen mode Exit fullscreen mode

To create an EC2, I have a dedicated testing Pod that has tolerations and nodeAffinity (see Kubernetes: Pods and WorkerNodes – control the placement of the Pods on the Nodes), through which Karpenter should create EC2 with the_"class-test-latest"_ EC2NodeClass:

apiVersion: v1
kind: Pod
metadata:
  name: test-pod
spec:
  containers:
    - name: nginx
      image: nginx
      command: ['sleep', '36000']
  restartPolicy: Never
  tolerations:
    - key: "TestOnly"
      effect: "NoSchedule"  
      operator: "Exists"
  affinity:
    nodeAffinity:
      requiredDuringSchedulingIgnoredDuringExecution:
        nodeSelectorTerms:
          - matchExpressions:
              - key: component
                operator: In
                values:
                  - test
Enter fullscreen mode Exit fullscreen mode

Deploy the changes, deploy the Pod, log in to EC2 and check the config:

[root@ip-10-0-45-117 ec2-user]# cat /etc/containerd/certs.d/docker.io/hosts.toml 

server = "https://docker.io"

[host."http://nexus-docker.ops.example.co"]
  capabilities = ["pull", "resolve"]
Enter fullscreen mode Exit fullscreen mode

Once again, run a pull of some PHP image:

[root@ip-10-0-43-82 ec2-user]# crictl pull php
Enter fullscreen mode Exit fullscreen mode

And check the Nexus:

Done.

Useful links

Originally published at RTFM: Linux, DevOps, and system administration.

Top comments (0)