Recently, while working on a case we noticed that we were unable to make changes to any virtual machine on a particular NetApp NFS datastore. We were unable to add disks, increase existing VMDKs or create virtual machines on that datastore.
The error we received was:
![]()
When we login to the host directly via UI client then we were able to perform all the above mentioned changes. This pointed out that, there seemed to be an issue with the vCenter server and not the datastore.
So looking into the vpxd.log for vCenter, this is what we saw:
2018-03-16T09:11:48.485Z info vpxd[7FB8F56ED700] [Originator@6876 sub=Default opID=VmConfigFormMediator-applyOnMultiEntity-93953-ngc:70007296-f3] [VpxLRO] -- ERROR task-252339 -- vm-44240 -- vim.VirtualMachine.reconfigure: vim.fault.InvalidDatastoreState:
--> Result:
--> (vim.fault.InvalidDatastoreState) {
--> faultCause = (vmodl.MethodFault) null,
--> faultMessage = <unset>,
--> datastoreName = "dc1_vmware01"
--> msg = ""
--> }
--> Args:
-->
--> Arg spec:
--> (vim.vm.ConfigSpec) {
--> changeVersion = "2018-03-15T08:04:06.266218Z",
--> name = <unset>,
The error we received was:
When we login to the host directly via UI client then we were able to perform all the above mentioned changes. This pointed out that, there seemed to be an issue with the vCenter server and not the datastore.
So looking into the vpxd.log for vCenter, this is what we saw:
2018-03-16T09:11:48.485Z info vpxd[7FB8F56ED700] [Originator@6876 sub=Default opID=VmConfigFormMediator-applyOnMultiEntity-93953-ngc:70007296-f3] [VpxLRO] -- ERROR task-252339 -- vm-44240 -- vim.VirtualMachine.reconfigure: vim.fault.InvalidDatastoreState:
--> Result:
--> (vim.fault.InvalidDatastoreState) {
--> faultCause = (vmodl.MethodFault) null,
--> faultMessage = <unset>,
--> datastoreName = "dc1_vmware01"
--> msg = ""
--> }
--> Args:
-->
--> Arg spec:
--> (vim.vm.ConfigSpec) {
--> changeVersion = "2018-03-15T08:04:06.266218Z",
--> name = <unset>,
To fix this we had to change the thin_prov_space_flag from 1 to 0 on the vCenter server database. In my case, the vCenter was an appliance (The process remains more or less same for Windows based vCenter as well)
The fix:
1. Always have a snapshot of the vCenter server before making any changes within it.
2. Stop the vCenter server service using:
# service-control --stop vmware-vpxd
3. Connect to the vCenter database using:
# /opt/vmware/vpostgres/current/bin/psql -d VCDB -U vc
The password for the vCenter DB can be found in the below file:
/etc/vmware-vpx/vcdb.properties file
Run this below query to list out the datastores with this vCenter:
select * from vpx_datastore where name='<enter-your-datastore-name>';
The output would be something like:
11 | dc1_vmware01 | ds:///vmfs/volumes/7a567de9-3e3c0969/ | 5277655814144 | 1745005346816 | NFS | | <obj xmlns:xsd="http://www.w3.org/2001/XMLSchema" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xmlns="urn:vim25" versionId="6.5" xsi:type="DatastoreCapability"><directoryHierarchySupported>true</directoryHierarchySupported><rawDiskMappingsSupported>false</rawDiskMappingsSupported><perFileThinProvisioningSupported>true</perFileThi
nProvisioningSupported><storageIORMSupported>true</storageIORMSupported><nativeSnapshotSupported>true</nativeSnapshotSupported><topLevelDirectoryCreateSupported>true</topLevelDirectoryCreateSupported><seSparseSupported>true</seSparseSupp
orted><vmfsSparseSupported>true</vmfsSparseSupported><vsanSparseSupported>false</vsanSparseSupported></obj> | 2 | 0 | 30 | 0 | 1 | automatic
| 90 | 1
Too much information here, so we can filter it out using the below query
select id,thin_prov_space_flag from vpx_datastore;
Now you can see:
VCDB=> select id,thin_prov_space_flag from vpx_datastore;
id | thin_prov_space_flag
------+----------------------
5177 | 0
6449 | 0
5178 | 0
12 | 0
795 | 0
149 | 0
793 | 0
11 | 1
Now, we need to change the thin_prov_space_flag from 1 to 0 for id=11
So run this query:
update vpx_datastore set thin_prov_space_flag=0 where id=<enter-your-id>;
Quit the database view using \q
Start the web client service using:
# service-control --start vmware-vpxd
Re-login back to the vCenter and now you should be able to make the necessary changes.
Hope this helps!