Mkdir permission denied on Talos Linux.

[Spaghetto392]

Hi, I'm trying to setup a k8s cluster on Talos Linux, everything so far has worked except for storage. Upon trying to create a PVC I receive the following:

Events:
  Type     Reason                Age                    From                                                               Message
  ----     ------                ----                   ----                                                               -------
  Normal   Provisioning          22h (x564 over 2d8h)   nfs.csi.k8s.io_talos-goz-704_7a7639b6-f595-400f-b5c9-df114cd120a6  External provisioner is provisioning volume for claim "test/pvc-test"
  Normal   ExternalProvisioning  22h (x8344 over 2d8h)  persistentvolume-controller                                        Waiting for a volume to be created either by the external provisioner 'nfs.csi.k8s.io' or manually by the system administrator. If volume creation is delayed, please verify that the provisioner is running and correctly registered.
  Normal   Provisioning          3m55s (x10 over 12m)   nfs.csi.k8s.io_talos-goz-704_169e4f00-a8c8-4f5f-b04f-0e51713846e0  External provisioner is provisioning volume for claim "test/pvc-test"
  Warning  ProvisioningFailed    3m55s (x10 over 12m)   nfs.csi.k8s.io_talos-goz-704_169e4f00-a8c8-4f5f-b04f-0e51713846e0  failed to provision volume with StorageClass "nfs-csi": rpc error: code = Internal desc = failed to make subdirectory: mkdir /tmp/pvc-c5435e50-2b7b-44ca-ae77-249b778b4530/pvc-c5435e50-2b7b-44ca-ae77-249b778b4530: permission denied
  Normal   ExternalProvisioning  110s (x42 over 12m)    persistentvolume-controller 

I've spoken to one of the developers of Talos, and so far they haven't been able to find anything that would potentially trigger this behavior. The /tmp directory on Talos is a small tmpfs volume and should be writable by services running in K8S.

The discussion can be found here in which I also provide my config files and general setup. If anyone could help, it would be greatly appreciated.

[andyzhangx]

related to this issue: #246?

[Spaghetto392]

Doesn't seem to be related. Other CSI drivers seem to be fine with the server's configuration.

[andyzhangx]

can you check the comments here? #246 (comment)
the nfs csi driver is trying to create a directory under root dir on the nfs server, pls make sure it has permission to do that. You could also specify subDir to avoid creating a subDir by the driver.

parameters:
  share: /
  subDir: /

[andyzhangx]

could you check whether readOnlyRootFilesystem: true is set in nfs container? that could be root cause, I have remove this setting in PR: #720