After moving on vSphere CSI, now the k8s PVs are not just vmdks, but container volumes, which in turn are FCD - first class disk. And the problem(or advantage) with FCD is that they are not removing after deleting the virtual machine and k8s cluster. So after regularly creating and deleting test clusters there are lots of stale container volumes aka fcd. In UI 6.7 you can't make any manipulations with container volumes. So, to clean the vSphere from unneeded container volumes you may use cli utility Govc.
Installation:
Add environment variables:
export GOVC_URL="vcenter.your-domain.com"
export GOVC_USERNAME="YourLogin"
export GOVC_PASSWORD='YourPassword'
export GOVC_INSECURE="true"
export GOVC_DATACENTER="OVH Kublr Dev"
|
How to look what volumes and disks do I have:
#For all vCenter
govc volume.ls -l
#For some datastore
govc volume.ls -l -ds '/OVH Kublr Dev/datastore/ESXi-0-SATA'
govc disk.ls -ds '/OVH Kublr Dev/datastore/ESXi-0-SSD' -l
#Get in JSON for future manipulations
govc volume.ls -json
govc disk.ls -ds '/OVH Kublr Dev/datastore/ESXi-0-SSD' -l -json
|
When the k8s cluster is deleted, container volumes lose their ConsumerId, so we need to remove all volumes with null ConsumerId.
How to clean. Get all volumes without the consumer and delete it:
govc disk.ls -ds '/OVH Kublr Dev/datastore/ESXi-0-SSD' -l -json|jq '.Objects[] | select(.Config.ConsumerId == null)| .Config.Id.Id'|xargs -I{} govc volume.rm {}
govc disk.ls -ds '/OVH Kublr Dev/datastore/ESXi-0-SATA' -l -json|jq '.Objects[] | select(.Config.ConsumerId == null)| .Config.Id.Id'|xargs -I{} govc volume.rm {}
|
How to remove only the volumes that you need. For example just after removing the cluster:
$ govc volume.ls -l
cd9b7bbc-b2b6-4b86-8628-ca4cd0bfd0a4 pvc-bfcdcfe9-31b0-4c59-947c-53fdeb20033e 64.0GB KUBERNETES some-k8s-cluster
cae6d507-baf8-4e40-a3af-3b09330433b2 464d266e-9295-11ed-8af7-d671f7772961 5.0GB KUBERNETES some-k8s-cluster-02
ca467d20-b452-45bc-bdcf-36e819e22b01 pvc-86bc68a1-573d-4ee1-9fb5-15f9f3276b65 8.0GB KUBERNETES some-k8s-cluster
67ffdc21-c5e6-4b19-b42c-290fb7705711 pvc-d30aa85f-bfa7-4df8-81c3-158da65e714a 1.0GB KUBERNETES some-k8s-cluster-02
628cb20b-7dec-4c29-b4fe-14c7f0df42b3 pvc-093fd47f-3908-4602-8e3e-06aea34e8120 29.8GB KUBERNETES some-deleted-cluster
# Let's say we want to remove volume that was owned by some-deleted-cluster
$ govc volume.rm 628cb20b-7dec-4c29-b4fe-14c7f0df42b3
|
