After moving on vSphere CSI, now the k8s PVs are not just vmdks, but container volumes, which in turn are FCD - first class disk. And the problem(or advantage) with FCD is that they are not removing after deleting the virtual machine and k8s cluster. So after regularly creating and deleting test clusters there are lots of stale container volumes aka fcd. In UI 6.7 you can't make any manipulations with container volumes. So, to clean the vSphere from unneeded container volumes you may use cli utility Govc.


# extract govc binary to /usr/local/bin
# note: the "tar" command must run with root permissions
curl -L -o - "$(uname -s)_$(uname -m).tar.gz" tar -C /usr/local/bin -xvzf - govc

Add environment variables:

export GOVC_URL=""
export GOVC_USERNAME="YourLogin"
export GOVC_PASSWORD='YourPassword'
export GOVC_INSECURE="true"
export GOVC_DATACENTER="OVH Kublr Dev"

How to look what volumes and disks do I have:

#For all vCenter
govc -l
#For some datastore
govc -l -ds '/OVH Kublr Dev/datastore/ESXi-0-SATA'
govc -ds '/OVH Kublr Dev/datastore/ESXi-0-SSD' -l
#Get in JSON for future manipulations
govc -json
govc -ds '/OVH Kublr Dev/datastore/ESXi-0-SSD' -l -json

When the k8s cluster is deleted, container volumes lose their ConsumerId, so we need to remove all volumes with null ConsumerId.

How to clean. Get all volumes without the consumer and delete it:

govc -ds '/OVH Kublr Dev/datastore/ESXi-0-SSD' -l -json|jq '.Objects[] | select(.Config.ConsumerId == null)| .Config.Id.Id'|xargs -I{} govc volume.rm {}
govc -ds '/OVH Kublr Dev/datastore/ESXi-0-SATA' -l -json|jq '.Objects[] | select(.Config.ConsumerId == null)| .Config.Id.Id'|xargs -I{} govc volume.rm {}

How to remove only the volumes that you need. For example just after removing the cluster:

$ govc -l
cd9b7bbc-b2b6-4b86-8628-ca4cd0bfd0a4    pvc-bfcdcfe9-31b0-4c59-947c-53fdeb20033e        64.0GB  KUBERNETES      some-k8s-cluster
cae6d507-baf8-4e40-a3af-3b09330433b2    464d266e-9295-11ed-8af7-d671f7772961            5.0GB   KUBERNETES      some-k8s-cluster-02
ca467d20-b452-45bc-bdcf-36e819e22b01    pvc-86bc68a1-573d-4ee1-9fb5-15f9f3276b65        8.0GB   KUBERNETES      some-k8s-cluster
67ffdc21-c5e6-4b19-b42c-290fb7705711    pvc-d30aa85f-bfa7-4df8-81c3-158da65e714a        1.0GB   KUBERNETES      some-k8s-cluster-02
628cb20b-7dec-4c29-b4fe-14c7f0df42b3    pvc-093fd47f-3908-4602-8e3e-06aea34e8120        29.8GB  KUBERNETES      some-deleted-cluster
# Let's say we want to remove volume that was owned by some-deleted-cluster
$ govc volume.rm 628cb20b-7dec-4c29-b4fe-14c7f0df42b3