The Ops Community ⚙️: J. Marhee

Caddy Ingress Controller with Rancher-managed RKE2/K3s

J. Marhee — Tue, 25 Mar 2025 20:16:32 +0000

The default Ingress controller for RKE2 and K3s are Nginx and Traefik, respectively, but both can have this overridden; for example, K3s can be deployed using Nginx in place of Traefik, using the K3s Helm Controller, and likewise for Ingress controllers like the one provided for Caddy.

To do this when deploying clusters via Rancher Manager, there are two steps:

Disable the default Ingress in the Cluster specification.
Add an additional manifest to the spec.rkeConfig to add a Helm Chart for the Caddy Ingress.

For the first step, either uncheck the box for Nginx (or Traefik for K3s) Ingress in your Rancher UI:

Or, in the cluster yaml, ensure this line is present:

    machineGlobalConfig:
      cni: calico
      disable:
        - rke2-ingress-nginx

in your machineGlobalConfig.

For step 2, add the following:

---
apiVersion: v1
kind: Namespace
metadata:
  name: caddy-system
---
apiVersion: helm.cattle.io/v1
kind: HelmChart
metadata:
  name: caddy-ingress-controller
  namespace: kube-system
spec:
  repo: https://caddyserver.github.io/ingress
  chart: caddy-ingress-controller
  targetNamespace: caddy-system
#  valuesContent: |
#    minikube: true #hostNet for testing, otherwise use load balancer
#    loadBalancer:
#       enabled: false
#    automaticHTTPS: your@email.com

to, either, the UI field for additional manifests:

or to .spec.rkeConfig in the yaml editor:

  rkeConfig:
    additionalManifest: |-
      ---
      apiVersion: v1
      kind: Namespace
      metadata:
        name: caddy-system
      ---
      apiVersion: helm.cattle.io/v1
      kind: HelmChart
      metadata:
        name: caddy-ingress-controller
        namespace: kube-system
      spec:
        repo: https://caddyserver.github.io/ingress
        chart: caddy-ingress-controller
        targetNamespace: caddy-system

Once your cluster is done provisioning, you can verify that Caddy is running as your default Ingress:

Running Calico eBPF Dataplane kube-proxy replacement on RKE2

J. Marhee — Fri, 19 Jul 2024 22:17:00 +0000

Tigera outlines the following benefits to enabling the eBPF dataplane:

The eBPF dataplane mode has several advantages over standard Linux networking pipeline mode:
It scales to higher throughput.
It uses less CPU per GBit.
It has native support for Kubernetes services (without needing kube-proxy) that:
Reduces first packet latency for packets to services.
Preserves external client source IP addresses all the way to the pod.
Supports DSR (Direct Server Return) for more efficient service routing.
Uses less CPU than kube-proxy to keep the dataplane in sync.
To learn more and see performance metrics from our test environment, see the blog, Introducing the Calico eBPF dataplane.

As I've written about before, RKE2 can be deploy with the kube-proxy replacement for Cilium, and it can be enabled on RKE2 after installation as well with a couple of additional changes to your Rancher cluster config.

Note For existing clusters, you can proceed immediately to the Calico patching process and skip the following about RKE2 provisioning and logging into your control plane. You can apply the following ConfigMap (below) through Rancher or via kubectl, etc. as normal.

In Rancher, when you provision your cluster, ensure that disable-kube-proxy is set to true when your cni is set to calico:

Keep in mind that this will prevent the cluster from reading "Ready" in the UI until you complete this process as Calico's readiness probe will not signal to Rancher it is complete until we are done.

On the first Control Plane node, set your PATH and KUBECONFIG variables:

m-3c90900d-c758-459e-b010-e344d7668e48:~ # export PATH=$PATH:/var/lib/rancher/rke2/bin
m-3c90900d-c758-459e-b010-e344d7668e48:~ # export KUBECONFIG=/etc/rancher/rke2/rke2.yaml

and apply the following ConfigMap:

kind: ConfigMap
apiVersion: v1
metadata:
  name: kubernetes-services-endpoint
  namespace: tigera-operator
data:
  KUBERNETES_SERVICE_HOST: '${KUBERNETES_VIP}'
  KUBERNETES_SERVICE_PORT: '6443'

at which point, the Calico operator will restart the Pods:

watch kubectl get pods -n calico-system

and you can proceed to patch the Operator on this node to enable the eBPF dataplane:

  kubectl patch installation.operator.tigera.io default --type merge -p '{"spec":{"calicoNetwork":{"linuxDataplane":"BPF"}}}'

At which point in Rancher, you will see the calico readiness probe for this node clear, and begin to process for the remaining nodes as Calico had now become ready (and no longer expecting kube-proxy to come online).

More on the Calico eBPF dataplane can be found here.

Running Cilium on Rancher-deployed RKE2 without kube-proxy

J. Marhee — Fri, 17 May 2024 22:39:00 +0000

In Linux Kernel 5.8 and above, RKE2 can be run using Cilium without the use of kube-proxy and deploying in RKE2 only requires a simple change to a HelmChart partial. For Rancher-managed RKE2 clusters, the commensurate change is similar.

Ensure your CNI is set to Cilium:

Switch to the YAML edit view, and navigate down to the rke2-cilium chartValues key:

and save, and your cluster will begin creating.

Once the cluster is online, you can validate the status of the kube-proxy replacement using the following:

kubectl -n kube-system exec ds/cilium -- cilium-dbg status | grep KubeProxyReplacement

More about this feature, and further what can be done with Cilium can be found here.

Using Git for a Helm Chart Repo

J. Marhee — Thu, 09 May 2024 20:59:08 +0000

Git Repos can be used to store Helm Charts, and can be imported by the Helm CLI to install charts to your Kubernetes cluster. My repo of charts for this example can be found here.

Assume you create a repo with the following structure, a repo named charts:

 ~/repos/charts (main) $ ls
assets      charts

where charts subdirectory contains all of your Helm chart templates like you might normally normally expect, and assets where you are going to store the Helm tar.gz packages we will create that the Helm CLI will actually pull and install.

In my case, my repo contains the following charts:

~/repos/charts (main) $ ls charts
sysdepot-authoritative-dns  sysdepot-logging
sysdepot-bitbucket      sysdepot-postgres

and I want to publish a chart for sysdepot-logging, so I will package it:

helm package charts/sysdepot-logging -d assets/sysdepot-logging
Successfully packaged chart and saved it to: charts/assets/sysdepot-logging-0.1.0.tgz

but before committing this archive and use my Git repo with Helm, I have to create an index so Helm can actually know where in my Git repo to find the chart bundle:

helm repo index .

which creates an index.yaml file:

apiVersion: v1
entries:
  sysdepot-logging:
  - apiVersion: v2
    appVersion: 1.16.0
    created: "2024-05-09T10:08:39.866741-05:00"
    description: A Helm chart for Kubernetes
    digest: ac545bd91ad481795556a331b15bd4871e598d1e89d0c54ed338ab9afb2b8771
    name: sysdepot-logging
    type: application
    urls:
    - assets/sysdepot-logging-0.1.0.tgz
    version: 0.1.0
generated: "2024-05-09T10:08:39.866166-05:00"

in the root of my repo that points to the archive in my assets directory.

So, with that created, I commit my charts, and the archives, and the index file to my git repo as I would with any git repository (git add, commit, etc.)

For a Github repo, for example, you can add it to Helm using the following format:

helm repo add jmarhee https://raw.githubusercontent.com/{your_username}/{your_repo}/{target_branch}/

then after helm repo update, you can install charts from your new repo.

High Availability RKE2 with kube-vip

J. Marhee — Tue, 30 Apr 2024 00:15:11 +0000

RKE2 can be deployed in a multi-control plane, high availability configuration, and we can use kube-vip to provide virtual IP functionality to the cluster to further support endpoint availability in the event of the cluster losing a control plane node.

Let's start by defining a few variables ahead of time. What we'll need are the addresses of your three RKE2 nodes, for example, mine were:

NODE_ADDRESS_0=192.168.0.197
NODE_ADDRESS_1=192.168.0.198
NODE_ADDRESS_2=192.168.0.199

and then the address you intend to use for your virtual IP, and the name of the interface on the node(s) it'll be advertised on (this should be standard across the nodes, but most importantly on the first control plane node when we we install the DaemonSet):

INTERFACE=enp1s0 # confirm this in the output of `ip addr`
KUBE_VIP_ADDRESS=192.168.0.200

In my case, my nodes are assigned addresses via DHCP, so I just selected the next address to be provisioned (kube-vip relies on ARP to determine the leader in the cluster which at the time of installation will only be the first node in your cluster), and then other options like the RKE2 cluster token, the hostname for your VIP, and the kube-vip Version:

CLUSTER_TOKEN=$(tr -dc A-Za-z0-9 </dev/urandom | head -c 13; echo)
KUBE_VIP_VERSION=v0.8.0
HOSTNAME=rke2.somequant.club

We need to do some preparation on each of the hosts:

Creating a few directories RKE2 will need:

for i in {NODE_ADDRESS_0,NODE_ADDRESS_1,NODE_ADDRESS_2}; do ssh $USER@$i "sudo mkdir -p /etc/rancher/rke2; sudo mkdir -p /var/lib/rancher/rke2/server/manifests/"; done

if your HOSTNAME variable is not a FQDN or resolvable with DNS, you might want to do another loop to add an /etc/hosts entry as well:

for i in {NODE_ADDRESS_0,NODE_ADDRESS_1,NODE_ADDRESS_2}; do ssh $USER@$i "echo '$KUBE_VIP_ADDRESS $HOSTNAME' | sudo tee -a /etc/hosts"; done

Then, prepare the RKE2 config file for the first control plane node:

cat <<EOF | tee rke2-prime-config.yaml
token: ${CLUSTER_TOKEN}
tls-san:
- ${HOSTNAME}
- ${KUBE_VIP_ADDRESS}
- ${NODE_ADDRESS_0}
- ${NODE_ADDRESS_1}
- ${NODE_ADDRESS_2}
write-kubeconfig-mode: 600
etcd-expose-metrics: true
cni:
- cilium
disable:
- rke2-ingress-nginx
- rke2-snapshot-validation-webhook
- rke2-snapshot-controller
- rke2-snapshot-controller-crd
EOF

then create (and set aside for now) the config for the other two control plane nodes:

cat <<EOF | tee rke2-peer-config.yaml
server: https://${KUBE_VIP_ADDRESS}:9345
token: ${CLUSTER_TOKEN}
tls-san:
- ${HOSTNAME}
- ${KUBE_VIP_ADDRESS}
- ${NODE_ADDRESS_0}
- ${NODE_ADDRESS_1}
- ${NODE_ADDRESS_2}
write-kubeconfig-mode: 600
etcd-expose-metrics: true
cni:
- cilium
disable:
- rke2-ingress-nginx
- rke2-snapshot-validation-webhook
- rke2-snapshot-controller
- rke2-snapshot-controller-crd
EOF

Transfer rke2-prime-config.yaml to NODE_ADDRESS_0:

scp rke2-prime-config.yaml $USER@$NODE_ADDRESS_0:/etc/rancher/rke2/config.yaml

Now, login to NODE_ADDRESS_0, and we will install and enable RKE2:

curl -sfL https://get.rke2.io | INSTALL_RKE2_TYPE=server sh -

systemctl enable rke2-server.service
systemctl start rke2-server.service

Once this step completes, wait until the node is in a Ready state in this output:

export PATH=$PATH:/var/lib/rancher/rke2/bin
export KUBECONFIG=/etc/rancher/rke2/rke2.yaml
kubectl get nodes -w

You are now ready to install kube-vip. From the above, set your INTERFACE and KUBE_VIP_ADDRESS variables on the host as well as the KUBE_VIP_VERSION.

Now, we can proceed to create our kube-vip manifests:

kubectl apply -f https://kube-vip.io/manifests/rbac.yaml

alias kube-vip="ctr --address /run/k3s/containerd/containerd.sock --namespace k8s.io image pull ghcr.io/kube-vip/kube-vip:$KUBE_VIP_VERSION; ctr --address /run/k3s/containerd/containerd.sock --namespace k8s.io run --rm --net-host ghcr.io/kube-vip/kube-vip:$KUBE_VIP_VERSION vip /kube-vip"

kube-vip manifest daemonset \
    --interface $INTERFACE \
    --address $KUBE_VIP_ADDRESS \
    --inCluster \
    --taint \
    --controlplane \
    --services \
    --arp \
    --leaderElection

This latter command will output a YAML manifest, which will not be executed; you can, either, redirect the output to a file (i.e. | tee -a kube-vip.yaml and kubectl apply -f as you might normally) or, as RKE2 allows you to do, redirect it to /var/lib/rancher/rke2/server/manifests/kube-vip.yaml where it will be applied automatically.

Once applied, you can confirm the status of the kube-vip DaemonSet:

kubectl get ds -n kube-system kube-vip-ds

root@fed7534656e3:~# kubectl get ds -n kube-system kube-vip-ds
NAME          DESIRED   CURRENT   READY   UP-TO-DATE   AVAILABLE   NODE SELECTOR   AGE
kube-vip-ds   1         1         1       1            1           <none>          10s

At which point, you should be able to ping, both, your VIP and the hostname (if you added the hosts entry, or the DNS can be resolved, this was optional):

root@fed7534656e3:~# ping rke2.somequant.club
PING rke2-vip.freeipad.internal (192.168.0.200) 56(84) bytes of data.
64 bytes from rke2.somequant.club (192.168.0.200): icmp_seq=1 ttl=64 time=0.079 ms
64 bytes from rke2.somequant.club (192.168.0.200): icmp_seq=2 ttl=64 time=0.068 ms
64 bytes from rke2.somequant.club (192.168.0.200): icmp_seq=3 ttl=64 time=0.083 ms
64 bytes from rke2.somequant.club (192.168.0.200): icmp_seq=4 ttl=64 time=0.055 ms

Now, we can proceed to joining our other two control plane nodes.

Copy your rke2-peer-config.yaml to NODE_ADDRESS_1 and 2:

for i in {NODE_ADDRESS_1,NODE_ADDRESS_2}; do scp rke2-peer-config.yaml $USER@$i:/etc/rancher/rke2/config.yaml ; done

Then proceed to install and enable RKE2 as you did on the first node:

for i in {NODE_ADDRESS_1,NODE_ADDRESS_2}; do ssh $USER@$i "curl -sfL https://get.rke2.io | sudo INSTALL_RKE2_TYPE=server sh -"; done

for i in {NODE_ADDRESS_1,NODE_ADDRESS_2}; do ssh $USER@$i "sudo systemctl enable rke2-server.service; sudo systemctl start rke2-server.service"; done

At which point, you can log back into the first node, and monitor the progress of the newly joining nodes, as they will be using the new VIP (rather than the node address of the first node) to join the cluster (so if the first node goes down, either one (or new nodes) can still be added to the cluster, and worker nodes can remain in contact with the API):

export PATH=$PATH:/var/lib/rancher/rke2/bin
export KUBECONFIG=/etc/rancher/rke2/rke2.yaml
kubectl get nodes -w

kubectl get nodes -o wide
NAME           STATUS   ROLES                       AGE     VERSION           INTERNAL-IP     EXTERNAL-IP   OS-IMAGE             KERNEL-VERSION       CONTAINER-RUNTIME
6933af500526   Ready    control-plane,etcd,master   79s     v1.27.12+rke2r1   192.168.0.198   <none>        Ubuntu 22.04.4 LTS   5.15.0-102-generic   containerd://1.7.11-k3s2
8986366dd7e8   Ready    control-plane,etcd,master   34s     v1.27.12+rke2r1   192.168.0.199   <none>        Ubuntu 22.04.4 LTS   5.15.0-102-generic   containerd://1.7.11-k3s2
fed7534656e3   Ready    control-plane,etcd,master   8m51s   v1.27.12+rke2r1   192.168.0.197   <none>        Ubuntu 22.04.4 LTS   5.15.0-102-generic   containerd://1.7.11-k3s2

When you login to the node that is currently the leader in the most recent election, you will also see that the INTERFACE will have the KUBE_VIP_ADDRESS bound as well:

ubuntu@fed7534656e3:~$ ip addr show enp1s0 | grep inet
    inet 192.168.0.197/24 metric 100 brd 192.168.0.255 scope global dynamic enp1s0
    inet 192.168.0.200/32 scope global enp1s0

More on Kube-VIP and its features can be found here

Rancher Backups with NFS-backed PersistentVolume

J. Marhee — Tue, 26 Mar 2024 20:40:51 +0000

Prerequisites

An external NFS provisioner is required to use an NFS mount as a PersistentVolume backing. This example uses the nfs-subdir-external-provisioner which will create the NFS storage class, nfs-client.

It should be installed with the reclaim policy set to Retain, to ensure the backup is not lost when a volume is deleted.

helm install nfs-subdir-external-provisioner nfs-subdir-external-provisioner/nfs-subdir-external-provisioner     --set nfs.server=192.168.0.11     --set nfs.path=/var/nfs/general --set nfs.reclaimPolicy=Retain --set “nfs.mountOptions={}”

Configuring the Backup Operator

In your local cluster in Rancher, select the Rancher Backup application from the App catalog. Upon beginning the installation, you will select “Use an existing StorageClass”:

And select the newly installed nfs-client StorageClass and a size for the backup volume:

Upon completing installation, you can verify the status of the PV as Bound:

$ kubectl get pv

The Reclaim Policy, Access Mode, and Capacity will be visible in the output of this command as well.

Using Makefiles to build switching between Podman and Docker automatically

J. Marhee — Mon, 19 Jun 2023 01:49:00 +0000

I use a Makefile with most of my projects, and the ones that are packaged as containers, this usually consists of a step like:

docker build -t me/my-project:some-tag

but I recently began using a system that uses Podman by default, so I could alias podman to docker, but more properly, I just updated my Makefile with the following to make the build step conditional and more flexible between hosts:

ifeq ($(shell command -v podman 2> /dev/null),)
    CMD=docker
else
    CMD=podman
endif
...
build-image:
    $(CMD) build -t jmarhee/scrivener:$(TAG) . --no-cache
...
quick-launch:
    x11docker --backend=$(CMD) jmarhee/scrivener

where in the above example, it detects not only that I'm running podman, but also updates other configurations that rely on the container runtime as well (in this case, launching a GUI application from the container)

Updating DNS Resolution in Kubernetes

J. Marhee — Tue, 18 Apr 2023 23:22:33 +0000

There are situations where one would prefer a Kubernetes workload to have an updated set of DNS nameservers, and there are two sets of solutions, ones that involve overriding host DNS resolution (so for example, the values in /etc/resolv.conf are overridden by a custom resolv.conf provided using various methods) or those scoped to the workload as part of the Pod spec. This piece will cover a few of those methods.

If you are using kube-dns, this can be updated cluster-wide using a ConfigMap like the following to specify your DNS nameservers:

apiVersion: v1
kind: ConfigMap
metadata:
  name: kube-dns
  namespace: kube-system
data:
  upstreamNameservers: |
    ["8.8.8.8"]

In the alternative, a custom resolv.conf file can be provided to the Kubelet via its CLI flag, --resolv-conf string, so for example:

nameserver 8.8.8.8
nameserver 8.8.4.4
search ...

and the Kubelet will expose that alternative resolver configuration rather than the host's.

For multi-tenant configurations, a the Pod spec supports a dnsConfig block that allows you to build your desired resolution settings into your workload definitions, from the linked documentation:

apiVersion: v1
kind: Pod
metadata:
  namespace: default
  name: dns-example
spec:
  containers:
    - name: test
      image: nginx
  dnsPolicy: "None"
  dnsConfig:
    nameservers:
      - 192.0.2.1 # this is an example
    searches:
      - ns1.svc.cluster-domain.example
      - my.dns.search.suffix
    options:
      - name: ndots
        value: "2"
      - name: edns0

The reason this would be advantageous for multi-tenant clusters is that it would allow specific workloads to use these DNS settings, rather than setting them for the cluster level, where a single-tenant might find this more desirable.

Here are some additional DNS-related resources for, both, updating resolvers and modifying other DNS services in Kubernetes:

DNS for Seervices and Pods

Customizing DNS Service

How to override DNS results served by CoreDNS

Fixing GlusterFS not in 'Peer in Cluster' State Error

J. Marhee — Thu, 03 Nov 2022 19:24:52 +0000

When creating a Glusterfs deployment, after peering your other nodes:

gluster peer probe node1

you can see the status of the cluster using gluster peer status, where you might see something like this:

Hostname: 192.168.0.122
Uuid: 35c563b6-b54a-4fb6-98a9-243d34e0a82e
State: Accepted peer request (Connected)

but in this State, you cannot create a volume yet, and you might get an error like:

ubuntu@61d8372c31e8:~$ sudo gluster volume create replicated-data replica 3 192.168.0.121:/gluster/02 192.168.0.122:/gluster/02 192.168.0.123:/gluster/02
volume create: replicated-data: failed: Host 192.168.0.122 is not in 'Peer in Cluster' state

If it is stuck in this state, using the UUID from the peer status above, you can modify the /var/lib/glusterd/peers/{UUID} file, where you'll see a state=4 line.

Update this line to read state=3 on, both, this node, and the UUID for the first node as well on the peer node if needed, then restart glusterd:

systemctl restart glusterd

and recheck the peer status:

Hostname: 192.168.0.122
Uuid: 35c563b6-b54a-4fb6-98a9-243d34e0a82e
State: Peer in Cluster (Connected)

at which point you can proceed to create a volume.

Preparing a FreeBSD Cloud Image with Cloud-Init

J. Marhee — Sat, 22 Oct 2022 20:07:38 +0000

I use Kernel-based Virtual Machines or KVM as my primary virtualization method in my development environment, and use cloud-init to prepare my virtual machine images. There's a variety of ways to procure cloud images, but a lot of times these registries are focused on Linux images, or hypervisor technology like VMWare or VirtualBox, and while normally useful for me at work (where I primarily work on Linux systems), I was very excited to find a well maintained site for BSD cloud images.

After downloading the image I want (in this case freebsd-13.0-zfs.qcow2) to my image path in /var/lib/libvirt/images, I create a project directory for my cloud-init scripts in ~/freebsd-cidata:

instance-id: freebsd
local-hostname: freebsd

in a file called meta-data, and in user-data:

#cloud-config
users:
  - name: root
    lock_passwd: false
    hashed_passwd: {PASSWD_HASH}
    ssh_pwauth: true
  - name: freebsd
    ssh_authorized_keys:
      - ssh-rsa ...
    hashed_passwd: {PASSWD_HASH}
    groups: wheel
    ssh_pwauth: true

write_files:
  - path: /usr/local/etc/sudoers
    content: |
      %wheel ALL=(ALL) NOPASSWD: ALL
    append: true

This user-data updates the root password, creates a freebsd user, adds that user to the wheel group, adds its SSH public keys, and then updates the sudoers file to allow wheel group passwordless sudo access (common pattern in cloud images, but this can be done any way you'd like). I recommend taking a look at the cloud-init example for other options and automation you can bake into your user-data file.

For example, on my Ubuntu images, I like to do a little more that I am doing above for the selected FreeBSD image:

#cloud-config
users:
  - name: ubuntu
    ssh_authorized_keys:
      - ssh-rsa ...
    sudo: ['ALL=(ALL) NOPASSWD:ALL']
    groups: sudo
    shell: /bin/bash
runcmd:
  - hostnamectl set-hostname $(openssl rand -hex 6)
  - echo PubkeyAuthentication yes | sudo tee -a /etc/ssh/sshd_config
  - echo PubkeyAcceptedKeyTypes=+ssh-rsa | sudo tee -a /etc/ssh/sshd_config
  - systemctl restart ssh
write_files:
- content: |
    Ubuntu 22.04 LTS \n \l
    IPv4: \4{enp1s0}
  path: /etc/issue
permissions: '0644'

where in this case, on Ubuntu systems, I like to generate a random hostname, things like updating SSH configuration options, and updating the console to show the IP address of the instance.

I like to use a password for the users, if you prefer only to use SSH keys, you can add a new list item for each key under ssh_authorized_keys, but if you want to set a password as well, you can create the password hash using:

mkpasswd --method=SHA-512 --rounds=4096

and populating that value in your user-data file.

Once those files are saved, you can create the cloud-init iso:

sudo genisoimage -output /var/lib/libvirt/images/cidata-freebsd.iso -V cidata -r -J ./user-data ./meta-data

You can then, back in your /var/lib/libvirt/images directory, proceed to create the VM, first by creating a clone of your cloud image for use on the new VM:

sudo qemu-img create -b freebsd-13.0-zfs.qcow2 -f qcow2 -F qcow2 ${your VM name}.img 80G

and then optionally using a too like virt-install to automate some of the work of using libvirt with KVM, consume both the disk image and the cloud-init ISO file along with your other VM specifications:

sudo virt-install --name=${VM_NAME} --ram=4196 --vcpus=2 --import --disk path=${VM_NAME}.img,format=qcow2 --disk path=cidata-freebsd.iso,device=cdrom --os-variant=freebsd13.0 --network bridge=br0,model=virtio --graphics vnc,listen=0.0.0.0 --noautoconsole

You can then view the spin up by running virsh list, getting the VM ID, and attaching the console to watch for cloud-init errors, virsh console ${ID}. The console output will contain a field like this:

/usr/local/bin/cloud-init startingCloud-init v. 21.2 running 'init' at Sat, 22 Oct 2022 19:41:45 +0000. Up 13.766839027404785 seconds.
ci-info: ++++++++++++++++++++++++++++++Net device info++++++++++++++++++++++++++++++
ci-info: +--------+------+----------------+------------+-------+-------------------+
ci-info: | Device |  Up  |    Address     |    Mask    | Scope |     Hw-Address    |
ci-info: +--------+------+----------------+------------+-------+-------------------+
ci-info: |  lo0   | True |   127.0.0.1    | 0xff000000 |   .   |         .         |
ci-info: |  lo0   | True |    ::1/128     |     .      |   .   |         .         |
ci-info: |  lo0   | True | fe80::1%lo0/64 |     .      |  0x2  |         .         |
ci-info: | vtnet0 | True | 192.168.0.115  | 0xffffff00 |   .   | 52:54:00:f4:32:c2 |
ci-info: +--------+------+----------------+------------+-------+-------------------+
ci-info:

containing your instance networking information, so you can attempt to connect via SSH (if you don't know the instance IP), or if you set a password, you can proceed to login from this console as well.

How to delete a Kubernetes namespace stuck at Terminating Status

J. Marhee — Wed, 17 Aug 2022 21:27:46 +0000

After running a command like:

kubectl delete my-namespace

and the command hangs indefinitely after accepting the deletion request, you might find it sitting at Terminating:

$ kubectl get ns/my-namespace
NAME      STATUS         AGE
testing   Terminating    113m

This might be due to a Finalizer in the spec, specific conditions to be met during deletion. You can view the finalizers for your namespace:

...
    "spec": {
       "finalizers": [
           "kubernetes"
       ]
    }
...

This is what a standard namespace with no other conditions might otherwise look like, so if it's hanging, this is likely way.

You can modify the resource using kubectl edit ns/my-namespace, and replace the finalizer with an empty array:

...
    "spec": {
       "finalizers": [
       ]
    }
...

or use the Kube API to update the specification to allow it to finalize manually if this is unsuccessful by dumping the resource:

kubectl get ns/my-namespace -o json > my-namespace.json

and editing that my-namespace.json to remove the finalizer, and then making a PUT request to the API:

kubectl replace --raw "/api/v1/namespaces/my-namespace/finalize" -f ./my-namespace.json

or using an HTTP client to issue the request (usually will require authentication if not exposing via kubectl proxy) using that same json dump:

curl -k -H "Content-Type: application/json" -X PUT --data-binary @my-namespace.json "https://<Kube API Endpoint>/api/v1/namespaces/my-namespace/finalize"

You can then confirm, once you receive a successful response, that the namespace is deleted:

kubectl get ns/my-namespace

which should return no resource.

Exporting images with Containerd CLI (ctr)

J. Marhee — Sun, 07 Aug 2022 03:28:53 +0000

I recently found that a Docker image I use as a part of one of my Helm charts was no longer available on DockerHub, and I hadn't mirrored it in another location. It was running in a K3s cluster, meaning I couldn't docker tag original-maintainer/image:tag me/image:tag it and push to the Hub myself back on my local machine, which was running the Docker CLI.

First, logging into a node with the image on it, I ran the following:

ctr -n k8s.io images export bind9:9.11.tar docker.io/internetsystemsconsortium/bind9:9.11 --platform linux/amd64

which exported the image to a tarball. This is similar to docker image save.

Then, pulling it down to my Docker CLI machines:

scp jdmarhee@192.168.0.60:bind9:9.11.tar .

I then load the image from the tarball:

 docker image load --input bind9\:9.11.tar

which returns the original tag for the image (internetsystemsconsortium/bind9:9.11) so I can re-tag it and push:

docker image tag internetsystemsconsortium/bind9:9.11 jmarhee/bind9:9.11

docker push jmarhee/bind9:9.11

Now this image is available on Docker Hub for my (and your) new clusters to access.