VMware KB1003409 and its consequences.

One of the biggest advantages of using ESXi over Linux or Microsoft based virtualization (xen, kvm, hyper-v) is independency from local drive. For a long time, I lived with knowledge that ESXi system partitions is only needed for system boot and then with some specific configuration ESXi doesn’t need the local drive. This is in particular extremely useful when you are booting ESXi by using SD card or some other devices not as reliable as two HDD in RAID1 😉

So, for many years my experience matches this theory until the moment where many of ESXi in cluster I was responsible for lost their access to the SD card (iLO simultaneous upgrade – other story).

In my case some of the hosts lost connection to SD card with error similar to this:

ScsiPath:: Failed to issue command 0x0 (cmdSN 0x0) on path vmhba32:C0:T0:L0: No connection

Skipping device mpx.vmhba32:C0:T0:L0 either due to VSI read error or abnormal state

Partition table read from device mpx.vmhba32:C0:T0:L0 failed: I/O error

I would say, nothing special BUT this time additionally the hosts began to be unstable and disconnected from vCenter. Reconnecting to vCenter was impossible.

I mean host was still running (no purple screen) but I wasn’t able to run some part of cli commands. Additionally, virtual systems which was perfectly running on this hosts couldn’t be migrated to others (no connection to vCenter).

This leads me to the following articles: https://kb.vmware.com/s/article/1003409 and https://kb.vmware.com/s/article/1003659.

Conclusion is that no one can be 100% sure that an underlying system disk failure will not harm your ESXi and/or underlying environment itself. In case you will have to reboot ESXi which is not connected to vCenter you will have to stop all workload for some time.

This situation didn’t happen few years ago (ESXi ver. 5.0, 5.5, 6.0) when I had a lot of troubles with SD cards. In case of failure I could easily migrate all workload to other hosts and then replace SD card. This is not the case anymore. Currently I rather recommend if possible to install two small disk for ESXi purpose in each solution.

No Comments

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Linux
Migrate WordPress site to another hosting service.

IntroductionThis article details the migration of WordPress site (exactly this site you are now on) from one service provider to Amazon Lightsail. There might be various reason to do that (mine is outlined below) but in general I hope to share the message that especially with WordPress, migration can be …

VMware
VMware Workstation and Fusion can be installed and use for free (even for the enterprise)

For a while now, the VMware Workstation (and Fusion for MacOS) can be used without any additional fee for Personal use. That was a great Broadcom news and nice gesture from that software vendor. Recently Broadcom announced that the software will be available for all, even the commercial sector. This …

Linux
Salt, VMware implementation – part 1, introduction

As every IT administrator knows, the infrastructure (meaning storages, compute, VMware virtualisation stack) is just a fundaments to run various operating systems (OS) and finally (containerized) application. Therefore, installation of (let’s call it) infrastructure in the datacenter (SDDC), in that sense is just the beginning of the adventure. No wonder, …