The object 'vim.Datastore:datastore-XXXX' has already been deleted or has not been completely created
Tuesday mornings are our routine maintenance window, and today’s maintenance was upgrading our vSphere hosts to ESXi, 6.7.0, 16075168.
We have a fairly standard, run of the mill deployment, so upgrades to hosts are pretty much the standard “select all hosts, hit ‘Remidate’, grab a coffee” type procedure.
As expected, our first host went into Maintenance Mode, patched itself, and powered down. However when it came back up, I was greeted with:
1 | The object 'vim.Datastore:datastore-158' has already been deleted or has not been completely created. |
Interestingly, I’d not come across this paticular issue before. It was related to vSphere HA (we have all our hosts in HA clusters), and that was stopping the host coming out of maintenance mode and the upgrade operation proceeding.
vCenter recommended I try the simple approach of clicking the rather aptly named “Reconfigure vSphere HA” button, so I tried this and attempted to take the host out of maintenance mode again. Same issue. Damn.
I looked back through our change history, and realised that one of the LUNs this host had access to was deleted not too long ago, which would explain our missing datastore error message.
vSphere HA, by default, will select two datastores as your ‘Heartbeat Datastores’. The issue here was that the deleted LUN had previously been selected by one of our hosts as a Heartbeat Datastore.
To resolve this issue, I performed the following steps:
- Click on your Cluster within vCenter
- Click on the ‘Configure’ tab
- Click on the ‘vSphere Availability’ menu
- Click ‘Edit’
- Click ‘Heartbeat Datastores’
- Select Use datastores only from the specified list
- Select any datastore which all your hosts have access to, and click OK
- Proceed with your upgrade as before
- Once your upgrade is complete, it’s probably advisable you change this option back to ‘Automatically select datastores accessible from the hosts’
Once this had been done, the issue went away, and our upgrade proceeded without issue.