After migrating to vSphere CSI driver from vSphere in-tree provider for vSphere volumes operations, the k8s PVs are not just vmdks, but container volumes, that are in turn FCD - first class disks.


The problem with FCDs is that they are not removed after deleting the virtual machine and the k8s cluster. As a result, after regularly creating and deleting test clusters a lot of stale container volumes (aka FCDs) are left.


In vSphere UI 6.7 it is impossible to manipulate container volumes, as a result it is necessary to use cli utility Govc to remove the stale volumes.


Govc installation

# extract govc binary to /usr/local/bin
# note: the "tar" command must run with root permissions
curl -L -o - \
  "https://github.com/vmware/govmomi/releases/latest/download/govc_$(uname -s)_$(uname -m).tar.gz" |
tar -C /usr/local/bin -xvzf - govc

Configure Govc

export GOVC_URL="vcenter.your-domain.com"
export GOVC_USERNAME="YourLogin"
export GOVC_PASSWORD='YourPassword'
export GOVC_INSECURE="true"
export GOVC_DATACENTER="Dev"

Investigate / list / remove volumes

# For the whole vCenter
govc volume.ls -l

# For specific datastore
govc volume.ls -l -ds '/Dev/datastore/ESXi-0-SATA'
govc disk.ls -ds '/Dev/datastore/ESXi-0-SSD' -l

# JSON output for future manipulations
govc volume.ls -json
govc disk.ls -ds '/Dev/datastore/ESXi-0-SSD' -l -json


When the k8s cluster is deleted, container volumes lose their ConsumerId, so to identify orphaned volumes, we need to look for volumes with null ConsumerId.


govc disk.ls -ds '/Dev/datastore/ESXi-0-SSD' -l -json |
  jq '.Objects[] | select(.Config.ConsumerId == null)| .Config.Id.Id'|xargs -I{} govc volume.rm {}

govc disk.ls -ds '/Dev/datastore/ESXi-0-SATA' -l -json | 
  jq '.Objects[] | select(.Config.ConsumerId == null)| .Config.Id.Id'|xargs -I{} govc volume.rm {}


How to remove only the volumes that you need. For example just after removing the cluster:


$ govc volume.ls -l
cd9b7bbc-b2b6-4b86-8628-ca4cd0bfd0a4    pvc-bfcdcfe9-31b0-4c59-947c-53fdeb20033e        64.0GB  KUBERNETES      some-k8s-cluster
cae6d507-baf8-4e40-a3af-3b09330433b2    464d266e-9295-11ed-8af7-d671f7772961            5.0GB   KUBERNETES      some-k8s-cluster-02
ca467d20-b452-45bc-bdcf-36e819e22b01    pvc-86bc68a1-573d-4ee1-9fb5-15f9f3276b65        8.0GB   KUBERNETES      some-k8s-cluster
67ffdc21-c5e6-4b19-b42c-290fb7705711    pvc-d30aa85f-bfa7-4df8-81c3-158da65e714a        1.0GB   KUBERNETES      some-k8s-cluster-02
628cb20b-7dec-4c29-b4fe-14c7f0df42b3    pvc-093fd47f-3908-4602-8e3e-06aea34e8120        29.8GB  KUBERNETES      some-deleted-cluster

# remove specific volume
$ govc volume.rm 628cb20b-7dec-4c29-b4fe-14c7f0df42b3