The Ops Community ⚙️

Arseny Zinchenko
Arseny Zinchenko

Posted on • Originally published at rtfm.co.ua on

Kubernetes: PVC in a StatefulSet, and the “Forbidden updates to statefulset spec” error

#devops #kubernetes #todayilearned

Kubernetes: PVC in StatefulSet, and the “Forbidden updates to statefulset spec” error

We have a VictoriaLogs Helm chart with a PVC size of 30 GB, which is no longer enough for us, and we need to increase it.

But the problem is that .spec.volumeClaimTemplates[*].spec.resources.requests.storage in STS is immutable, that is, we can't just change the size through values.yaml file, because it will lead to the error "Forbidden: updates to statefulset spec for fields other than 'replicas', 'ordinals', 'template', 'updateStrategy', 'revisionHistoryLimit', 'persistentVolumeClaimRetentionPolicy' and 'minReadySeconds' are forbidden".

The chart values now look like this:

victoria-logs-single:
  server:
    persistentVolume:
      enabled: true
      storageClassName: gp2-retain
      size: 30Gi
    retentionPeriod: 7d
Enter fullscreen mode Exit fullscreen mode

And with the default type of StatefulSet in the chart, the volumeClaimTemplates is used to create PVCs:

...  
volumeClaimTemplates:
    - apiVersion: v1
      kind: PersistentVolumeClaim
      metadata:
        name: server-volume
        ...
      spec:
        ...
        resources:
          requests:
            storage: {{ $app.persistentVolume.size }}
...
Enter fullscreen mode Exit fullscreen mode

If instead of STS there was a Deployment type, then in the VictoriaLogs chart this would lead to the creation of a separate PVC  —  see the pvc.yaml.

You could simply create a separate PVC yourself and connect it through the existingClaim value, but you already have a PersistentVolume, and you don't want to create a new one and migrate data (although you can if you need to, see VictoriaMetrics: migrating VMSingle and VictoriaLogs data between Kubernetes clusters, but there will be a down time), so let's see how we can solve this differently - without deleting Pods and without stopping the service.

storageClassName and AllowVolumeExpansion

The storageClas used to create a Persistent Volume must support AllowVolumeExpansion - see Volume expansion:

$ kk describe storageclass gp2-retain
Name: gp2-retain
...
Provisioner: kubernetes.io/aws-ebs
Parameters: <none>
AllowVolumeExpansion: True
MountOptions: <none>
ReclaimPolicy: Retain
VolumeBindingMode: WaitForFirstConsumer
...
Enter fullscreen mode Exit fullscreen mode

Create this storageClass when creating an EKS cluster from a simple manifest:

...
resource "kubectl_manifest" "storageclass_gp2_retain" {

  yaml_body = <<YAML
    apiVersion: storage.k8s.io/v1
    kind: StorageClass
    metadata:
      name: gp2-retain
    provisioner: kubernetes.io/aws-ebs
    reclaimPolicy: Retain
    allowVolumeExpansion: true
    volumeBindingMode: WaitForFirstConsumer
  YAML
}
...
Enter fullscreen mode Exit fullscreen mode

Although there is a dedicated storage_class resource for Terraform, and would be better to use it instead for the kubectl_manifest.

And the kubernetes.io/aws-ebs driver is already deprecated (OMG, since Kubernetes 1.17!), it's time to update to ebs.csi.aws.com.

But we’ll fix this later, right now the goal is to simply increase the disk.

Reproducing the issue

For the test, let’s write our own STS with volumeClaimTemplates:

apiVersion: apps/v1
kind: StatefulSet
metadata:
  name: demo-sts
spec:
  serviceName: demo-sts-svc
  replicas: 1
  selector:
    matchLabels:
      app: demo
  template:
    metadata:
      labels:
        app: demo
    spec:
      containers:
        - name: app
          image: busybox
          command: ["sh", "-c", "sleep 3600"]
          volumeMounts:
            - name: data
              mountPath: /data
  volumeClaimTemplates:
    - metadata:
        name: data
      spec:
        accessModes: ["ReadWriteOnce"]
        storageClassName: gp2-retain
        resources:
          requests:
            storage: 1Gi
Enter fullscreen mode Exit fullscreen mode

In volumeClaimTemplates, set the storageClassName and the size to 1 gigabyte.

Deploy:

$ kk apply -f test-sts-pvc.yaml 
statefulset.apps/demo-sts created
Enter fullscreen mode Exit fullscreen mode

Check the PVC:

$ kk get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS VOLUMEATTRIBUTESCLASS AGE
data-demo-sts-0 Bound pvc-31a9a547-7547-4d34-bb2d-2c7015b9e0f3 1Gi RWO gp2-retain <unset> 15s
Enter fullscreen mode Exit fullscreen mode

Now, if we want to increase the size via volumeClaimTemplates from 1Gi to 2Gi:

...
  volumeClaimTemplates:
    ...
        resources:
          requests:
            storage: 2Gi
Enter fullscreen mode Exit fullscreen mode

Then we get an error:

$ kk apply -f test-sts-pvc.yaml 
The StatefulSet "demo-sts" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'ordinals', 'template', 'updateStrategy', 'revisionHistoryLimit', 'persistentVolumeClaimRetentionPolicy' and 'minReadySeconds' are forbidden
Enter fullscreen mode Exit fullscreen mode

The Fix

But we can get around this very easily:

  1. edit the PVC manually   — set a new size
  2. delete STS with the --cascade=orphan - see Delete owner objects and orphan dependents
  3. create STS again
  4. …
  5. profit!

Let’s try it.

Note : before changing disks, don’t forget about backups!

Edit the PVC manually —  change resources.requests.storage from 1Gi to 2Gi:

Check the Events of this PVC:

$ kk describe pvc data-demo-sts-0
...
  Normal ExternalExpanding 40s volume_expand CSI migration enabled for kubernetes.io/aws-ebs; waiting for external resizer to expand the pvc
  Normal Resizing 40s external-resizer ebs.csi.aws.com External resizer is resizing volume pvc-31a9a547-7547-4d34-bb2d-2c7015b9e0f3
  Normal FileSystemResizeRequired 35s external-resizer ebs.csi.aws.com Require file system resize of volume on node
Enter fullscreen mode Exit fullscreen mode

And after a few more seconds, it’s done:

...
  Normal FileSystemResizeSuccessful 19s kubelet
Enter fullscreen mode Exit fullscreen mode

Check CAPACITY:

$ kk get pvc
NAME STATUS VOLUME CAPACITY ACCESS MODES STORAGECLASS VOLUMEATTRIBUTESCLASS AGE
data-demo-sts-0 Bound pvc-31a9a547-7547-4d34-bb2d-2c7015b9e0f3 2Gi RWO gp2-retain <unset> 4m7s
Enter fullscreen mode Exit fullscreen mode

2Gi, everything is OK.

And now we also have 2 gigabytes in the Pod itself:

$ kk exec -ti demo-sts-0 -- df -h /data
Filesystem Size Used Available Use% Mounted on
/dev/nvme7n1 1.9G 24.0K 1.9G 0% /data
Enter fullscreen mode Exit fullscreen mode

But if we try to deploy the changes to volumeClaimTemplates.spec.resources.requests.storage again, we will still get an error:

$ kk apply -f test-sts-pvc.yaml 
The StatefulSet "demo-sts" is invalid: spec: Forbidden: updates to statefulset spec for fields other than 'replicas', 'ordinals', 'template', 'updateStrategy', 'revisionHistoryLimit', 'persistentVolumeClaimRetentionPolicy' and 'minReadySeconds' are forbidden
Enter fullscreen mode Exit fullscreen mode

So, delete the STS itself, but leave all its dependent objects:

$ kubectl delete statefulset demo-sts --cascade=orphan 
statefulset.apps "demo-sts" deleted
Enter fullscreen mode Exit fullscreen mode

Check if the Pod is alive:

$ kk get pod
NAME READY STATUS RESTARTS AGE
demo-sts-0 1/1 Running 0 3m13s
Enter fullscreen mode Exit fullscreen mode

And now we just create STS again, with a new value in the volumeClaimTemplates.spec.resources.requests.storage:

$ kk apply -f test-sts-pvc.yaml 
statefulset.apps/demo-sts created
Enter fullscreen mode Exit fullscreen mode

Done.

Originally published at RTFM: Linux, DevOps, and system administration.

Top comments (1)

Some comments may only be visible to logged-in visitors. Sign in to view all comments.