VMware KB1003409 and its consequences.

One of the biggest advantages of using ESXi over Linux or Microsoft based virtualization (xen, kvm, hyper-v) is independency from local drive. For a long time, I lived with knowledge that ESXi system partitions is only needed for system boot and then with some specific configuration ESXi doesn’t need the local drive. This is in particular extremely useful when you are booting ESXi by using SD card or some other devices not as reliable as two HDD in RAID1 😉

So, for many years my experience matches this theory until the moment where many of ESXi in cluster I was responsible for lost their access to the SD card (iLO simultaneous upgrade – other story).

In my case some of the hosts lost connection to SD card with error similar to this:

ScsiPath:: Failed to issue command 0x0 (cmdSN 0x0) on path vmhba32:C0:T0:L0: No connection

Skipping device mpx.vmhba32:C0:T0:L0 either due to VSI read error or abnormal state

Partition table read from device mpx.vmhba32:C0:T0:L0 failed: I/O error

I would say, nothing special BUT this time additionally the hosts began to be unstable and disconnected from vCenter. Reconnecting to vCenter was impossible.

I mean host was still running (no purple screen) but I wasn’t able to run some part of cli commands. Additionally, virtual systems which was perfectly running on this hosts couldn’t be migrated to others (no connection to vCenter).

This leads me to the following articles: https://kb.vmware.com/s/article/1003409 and https://kb.vmware.com/s/article/1003659.

Conclusion is that no one can be 100% sure that an underlying system disk failure will not harm your ESXi and/or underlying environment itself. In case you will have to reboot ESXi which is not connected to vCenter you will have to stop all workload for some time.

This situation didn’t happen few years ago (ESXi ver. 5.0, 5.5, 6.0) when I had a lot of troubles with SD cards. In case of failure I could easily migrate all workload to other hosts and then replace SD card. This is not the case anymore. Currently I rather recommend if possible to install two small disk for ESXi purpose in each solution.

No Comments

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Free(BSD)
Search for duplicated files

This will be short (but hopefully I will find more time to show entire process to search duplicated files together with some examples). In case you are searching for duplicated files I can recommend two software which actually rocks in openSource world

Azure
NFS issue, cannot be mounted or is not visible

The same kind of issue I have encountered numerous times while working across different environments and with various customers. The problem with NFS mounts connected from remote locations is so common. This issue extends beyond communication solely over WAN and also include connections between datacenters (DC) where we lack control …

Azure
Why Firefox is important and people should use this browser in 2024, my thoughts.

Can you remember the times when everyone was using Internet Explorer? Back in the ’90s and the early part of this century, Internet Explorer dominated the browser market. Software Incompatibility with Other Browsers Incompatibility issues with software and other browsers have been a persistent problem. Even in 2022, this remains …