Hosting a Private Docker Registry - DigitalOcean Kubernetes - Part 2

In my previous article, I set up a Kubernetes cluster with hosting, ingress, and TLS. This time my goal was to deploy a private Docker registry on the cluster. Then upload an image to the registry and deploy it on the cluster.

Again, there's documentation on how to do that via DigitalOcean. This was my starting point, and I made a few adjustments along the way.

DNS

My first task was setting up another A record to point to the cluster. I went with registry.mattkruskamp.com to stay with the tutorial as close as possible.

Creating spaces storage

The registry services run on the cluster, but the services need a place to store the images. That's where DigitalOcean spaces come in. Again, creating things in DigitalOcean is easy. Click the Create button in the top right.

no spaces

I entered the basic information for my space.

configure space

Once created, I uploaded a file to it. For some reason having a file is required for initialization.

upload file

Access keys must be generated to authenticate the cluster with the space. In the spaces section click the Manage Keys button on the right. In the Tokens/Keys section there's a Spaces access keys section.

access key

At this point, it's time to do some configuration. The first thing I did was remove the production instances I created from the first article.

kubectl delete -f hello-kubernetes-ingress.yaml

kubectl delete -f hello-kubernetes-first.yaml

Then create a workspace to add files.

mkdir k8s-registry

cd k8s-registry/

Set up a file to override the defaults of the Helm installation.

chart-values.yaml

ingress:
  enabled: true
  hosts:
    - <registry.yoursite.com>
  annotations:
    kubernetes.io/ingress.class: nginx
    certmanager.k8s.io/cluster-issuer: letsencrypt-prod
    nginx.ingress.kubernetes.io/proxy-body-size: '30720m'
  tls:
    - secretName: letsencrypt-prod
      hosts:
        - <registry.yoursite.com>

storage: s3

secrets:
  htpasswd: ''
  s3:
    accessKey: '<your-space-access-key>'
    secretKey: '<your-space-secret-key>'

s3:
  region: <your-space-region>
  regionEndpoint: <your-space-region.digitaloceanspaces.com>
  secure: true
  bucket: <your-space-name>

I was a little confused about what to configure. The the hosts and tls values get set to the A record setup. The secretName value must be unique to this ingress. The accessKey and secretKey are pulled from creating the keys in the previous step. The region refers to the region prefix for the data center used. In my case it was sfo2. The same region's used for regionEndpoint. Ex: sfo2.digitaloceanspaces.com. Finally, bucket is the name of the space. Now Helm comes back to install the registry.

helm repo update

helm install stable/docker-registry -f chart_values.yaml --name docker-registry

Resources get created for the docker-registry. At this point, it's time to test it out.

Testing the registry

The best way to test is by pulling an existing repo, re-tagging it, and pushing it to the new registry.

sudo docker pull mysql

sudo docker tag mysql <registry.yoursite.com>/mysql

sudo docker push <registry.yoursite.com>/mysql

Once this was successful, clean up the existing resources and pull it from the new remote registry.

sudo docker rmi <registry.yoursite.com>/mysql && sudo docker rmi mysql

sudo docker pull <registry.yoursite.com>/mysql

Now verify the image is there.

sudo docker images

Securing the registry

A private repository isn't valuable if there's no authentication. Securing it is done by adding usernames and passwords to the configuration via htpasswd. Htpasswd can get accessed via a docker image.

docker run --rm -v ${PWD}:/app -it httpd htpasswd -b -c /app/htpasswd_file <username> <password>

touch htpasswd_file

Then username and password combinations are added to the file.

htpasswd -B htpasswd_file <username>

Once all the usernames and passwords get added, I opened the file and copied the contents. Then the configuration for the registry is opened and htpasswd: "" section gets replaced with htpasswd: |- followed by the usernames and password hashes.

chart-values.yaml

---
secrets:
  htpasswd: |-
    <username>:<password-hash>

Next, the registry is updated to include the authentication configuration.

helm upgrade docker-registry stable/docker-registry -f chart-values.yaml

This is easily tested by trying to pull the image down.

sudo docker pull <registry.yoursite.com>/mysql

Retry after authenticating docker and trying again.

sudo docker login <registry.yoursite.com>

sudo docker pull <registry.yoursite.com>/mysql

The next step was to authenticate the Kubernetes cluster with the registry via sudo kubectl create secret generic regcred --from-file=.dockerconfigjson=/home/user/.docker/config.json --type=kubernetes.io/dockerconfigjson. This didn't work for me due to Mac not storing the auth credentials in the config.json. I searched the comments and found a command that works.

kubectl create secret docker-registry regcred --docker-server=<your-registry-server> --docker-username=<your-name> --docker-password=<your-pword>

Running a deployment

In the previous article, I used the hello-kubernetes docker image to test deployments. I did the same this time around by tagging a private version and deploying it. Then I created another configuration file and publishing it using the private registry.

sudo docker pull paulbouwer/hello-kubernetes:1.5

sudo docker tag paulbouwer/hello-kubernetes:1.5 <registry.yoursite.com>/paulbouwer/hello-kubernetes:1.5

sudo docker push <registry.yoursite.com>/paulbouwer/hello-kubernetes:1.5

hello-world.yaml

apiVersion: extensions/v1beta1
kind: Ingress
metadata:
  name: hello-kubernetes-ingress
  annotations:
    kubernetes.io/ingress.class: nginx
    cert-manager.io/cluster-issuer: letsencrypt-prod
    nginx.ingress.kubernetes.io/rewrite-target: /
spec:
  tls:
    - hosts:
        - <your.domain.com>
      secretName: hello-kubernetes-tls
  rules:
    - host: <your.domain.com>
      http:
        paths:
          - path: /
            backend:
              serviceName: hello-kubernetes
              servicePort: 80
---
apiVersion: v1
kind: Service
metadata:
  name: hello-kubernetes
spec:
  type: NodePort
  ports:
    - port: 80
      targetPort: 8080
  selector:
    app: hello-kubernetes
---
apiVersion: apps/v1
kind: Deployment
metadata:
  name: hello-kubernetes
spec:
  replicas: 3
  selector:
    matchLabels:
      app: hello-kubernetes
  template:
    metadata:
      labels:
        app: hello-kubernetes
    spec:
      containers:
        - name: hello-kubernetes
          image: <registry.yoursite.com>/paulbouwer/hello-kubernetes:1.5
          ports:
            - containerPort: 8080
      imagePullSecrets:
        - name: regcred

Update the cluster with the new configuration.

kubectl apply -f hello-world.yaml

Finally, go see the new site.

test site

Impressions

I'm happy Helm has configurations for private registries. The documentation is good, but I did run into a few snags that took some time to unwind. When I first set everything up I kept getting a "certificate is valid for ingress.local" error. This was sorted out by understanding every ingress must have a unique secretName. This documentation exists, but I missed it. The second issue was setting up regcred. This problem is a difference between Mac and Linux, but luckily the commenters on the article figured it out.

I don't like the htaccess solution for password management. I'm assuming there's a lot of ways to handle that so I'll be looking into better ways. Despite those small issues, I'm very happy with the cluster. It's been rock solid while I've been working with it, and the DigitalOcean monitoring tools are fantastic.

Next steps

In the next article I'll be converting the site to a Docker container and deploying it.