kubernetes: add support for persistent volume claim volumes

[d4l3k]

Description

Add support for PersistentVolumeClaim mounts to Kubernetes scheduler.

Motivation/Background

#420 adds bindmounts to K8S, we want to add in persistent volume claims for Kubernetes which will let us support most of the other remote mounts.

Detailed Proposal

Add a new mount type to specs:

class MountTypes(Enum):
    PERSISTENT_CLAIM = "persistent-claim"
    BIND = "bind"

class PersistentClaimMount(Mount):
    name: str
    dst_path: str
    read_only: bool = False

class Role:
    ...
    mounts: List[Union[BindMount,PersistentClaimMount]]

Add a new format to parse_mounts:

--mounts bind=persistent-claim,name=foo,dst=/foo[,readonly]

Alternatives

Users can already mount a volume on the host node and then bind mount it into kubernetes pod but this violates some isolation principles and can be an issue from a security perspective. It also is a worse experience for users since the mounts need to be mounted on ALL hosts.

Additional context/links

• V1Volume https://github.com/kubernetes-client/python/blob/master/kubernetes/docs/V1Volume.md
• V1PersistentVolume https://github.com/kubernetes-client/python/blob/master/kubernetes/docs/V1PersistentVolumeClaimVolumeSource.md
• FSx on EKS https://github.com/kubernetes-sigs/aws-fsx-csi-driver/blob/master/examples/kubernetes/static_provisioning/README.md