Migration VM between environments and storage traps.

Problem definition:

VM migration between environments can have various subtle consequences. One of them is the behavior of the disk when migrating, for example, using HCX, from a traditional environment based on SAN storage to a vSAN environment. Traditional SAN environments usually (though not always) utilize thick-provisioned disks. Some arrays, such as advanced HPE arrays, have zero detection capabilities, enabling thin provisioning to occur only on the array, avoiding double thin provisioning. However, this approach may result in significant overhead on storage (exposing volumes on storage beyond the capacity of the disk array). The choice will, of course, depend on the administrator and their best practices or the capabilities of the specific hardware.

VM migration between environments can have various subtle consequences. One of them is the behavior of the disk when migrating, for example, using HCX, from a traditional environment based on SAN storage to a vSAN environment. Traditional SAN environments usually (though not always) utilize thick-provisioned disks. Some arrays, such as advanced HPE arrays, have zero detection capabilities, enabling thin provisioning to occur only on the array, avoiding double thin provisioning. However, this approach may result in significant overhead on storage (exposing volumes on storage beyond the capacity of the disk array). The choice will, of course, depend on the administrator and their best practices or the capabilities of the specific hardware.

In fact, what we anticipate from such calculations is that we have two entries in the table: occupied space and free space. This occupancy includes not only data but also metadata, such as file system information and logical volumes, which constitute a few percent. Typically, these can be safely omitted when adding a secure margin to the final allocations.

So, in general, our expectation can be represented (in very simplistic form) like below:

In reality, however, what we are dealing with also includes all files that have been created and deleted from the disk during its lifetime. This is due to the fact that older operating systems (such as Windows 2003, 2008) and older RedHat systems do not support the UNMAP function (iSCSI function), which could inform the ESXi host’s disk handler that a file has been deleted from the file system and should also be deleted at the VMDK file level. The current situation can be illustrated by the following diagram:

The “trash” value can quite huge. Based on my observation, it can take even 80-90% of free space.

As you can imagine, migrating such a disk structure means we will be migrating everything!!! This includes data that has been deleted, perhaps years ago, but still remains at the VMDK file level (and file system level).

Solution:

As usual, there is no simple solution, and as usual, there is not a single solution. There are two options: we can try to clean up before migration or after migration—depending on our organizational, technological, and time capabilities. The approach I took involved migrating the ‘junk’ to a cloud environment and then cleaning it up there.

Cleaning up

The algorithm for each disk looked as follows:

  • run a script that creates a large file filled with zeros, then delete it, and repeat the process to enhance efficiency.
  • copy VM (storage vMotion) between vSAN datastories
  • copy VM (storage vMotion) back where should be

Fortunately, creating a disk filled with zeros doesn’t take too long, especially in a fast cloud environment. Below is a PowerShell script that can be used for this purpose, script is based on one founded over the internet and run with success even including on older Windows systems.

<#
.SYNOPSIS
 Writes a large file full of zeroes to a volume in order to allow a storage
 appliance to reclaim unused space.

.DESCRIPTION
 Creates a file called ThinSAN.tmp on the specified volume that fills the
 volume up to leave only the percent free value (default is 5%) with zeroes.
 This allows a storage appliance that is thin provisioned to mark that drive
 space as unused and reclaim the space on the physical disks.
 
.PARAMETER Root
 The folder to create the zeroed out file in.  This can be a drive root (c:\)
 or a mounted folder (m:\mounteddisk).  This must be the root of the mounted
 volume, it cannot be an arbitrary folder within a volume.
 
.PARAMETER PercentFree
 A float representing the percentage of total volume space to leave free.  The
 default is .05 (5%)

.EXAMPLE
 PS> Write-ZeroesToFreeSpace -Root "c:\"
 
 This will create a file of all zeroes called c:\ThinSAN.tmp that will fill the
 c drive up to 95% of its capacity.
 
.EXAMPLE
 PS> Write-ZeroesToFreeSpace -Root "c:\MountPoints\Volume1" -PercentFree .1
 
 This will create a file of all zeroes called
 c:\MountPoints\Volume1\ThinSAN.tmp that will fill up the volume that is
 mounted to c:\MountPoints\Volume1 to 90% of its capacity.

.EXAMPLE
 PS> Get-WmiObject Win32_Volume -filter "drivetype=3" | Write-ZeroesToFreeSpace
 
 This will get a list of all local disks (type=3) and fill each one up to 95%
 of their capacity with zeroes.
 
.NOTES
 You must be running as a user that has permissions to write to the root of the
 volume you are running this script against. This requires elevated privileges
 using the default Windows permissions on the C drive.
#>
param(
  [Parameter(Mandatory=$true,ValueFromPipelineByPropertyName=$true)]
  [ValidateNotNullOrEmpty()]
  [Alias("Name")]
  $Root,
  [Parameter(Mandatory=$false)]
  [ValidateRange(0,1)]
  $PercentFree =.05
)
process{
  #Convert the $Root value to a valid WMI filter string
  $FixedRoot = ($Root.Trim("\") -replace "\\","\\") + "\\"
  $FileName = "ThinSAN.tmp"
  $FilePath = Join-Path $Root $FileName
  
  #Check and make sure the file doesn't already exist so we don't clobber someone's data
  if( (Test-Path $FilePath) ) {
    Write-Error -Message "The file $FilePath already exists, please delete the file and try again"
  } else {
    #Get a reference to the volume so we can calculate the desired file size later
    $Volume = gwmi win32_volume -filter "name='$FixedRoot'"
    if($Volume) {
      #I have not tested for the optimum IO size ($ArraySize), 64kb is what sdelete.exe uses
      $ArraySize = 64kb
      #Calculate the amount of space to leave on the disk
      $SpaceToLeave = $Volume.Capacity * $PercentFree
      #Calculate the file size needed to leave the desired amount of space
      $FileSize = $Volume.FreeSpace - $SpacetoLeave
      #Create an array of zeroes to write to disk
      $ZeroArray = new-object byte[]($ArraySize)
      
      #Open a file stream to our file 
      $Stream = [io.File]::OpenWrite($FilePath)
      #Start a try/finally block so we don't leak file handles if any exceptions occur
      try {
        #Keep track of how much data we've written to the file
        $CurFileSize = 0
        while($CurFileSize -lt $FileSize) {
          #Write the entire zero array buffer out to the file stream
          $Stream.Write($ZeroArray,0, $ZeroArray.Length)
          #Increment our file size by the amount of data written to disk
          $CurFileSize += $ZeroArray.Length
        }
      } finally {
        #always close our file stream, even if an exception occurred
        if($Stream) {
          $Stream.Close()
        }
        #always delete the file if we created it, even if an exception occurred
        if( (Test-Path $FilePath) ) {
          del $FilePath
        }
      }
    } else {
      Write-Error "Unable to locate a volume mounted at $Root"
    }
  }
}

After copying VM between vSAN datastore, VM size will be reduced almost immediately after migration.

There is another option to proceed with the newest VMware Public Cloud implementation (refering to AVS) where UNMAP function can be run from Azure AVS interface.

For testing purpose, having script to create large, non-empty (!) file could be also beneficial. After many tests (again on old version of Windows) below one I can recommend (because of speed and simplicity):

# Set the size of the file in bytes
$size = 10GB
$chunkSize = 1GB

# Create the file and write random data to it in chunks
$file = New-Item -ItemType File e:\file7.txt -Force
$stream = $file.OpenWrite()
$bytesWritten = 0
while ($bytesWritten -lt $size) {
    # Generate random data for the current chunk
    $chunk = New-Object byte[] $chunkSize
    $rand = New-Object System.Random
    $rand.NextBytes($chunk)

    # Write the chunk to the file
    $stream.Write($chunk, 0, $chunk.Length)
    $stream.Flush()

    # Update the number of bytes written
    $bytesWritten += $chunkSize
}

# Close the file stream
$stream.Close()

# Verify the file size
(Get-Item e:\file7.txt).length

Lessen learn:

Migraion operations, especially in large, complex environments, are not trivial. Beyond simple tasks, such as calculating the target space, one must ponder and identify potential risks and traps. I hope this article has clearly highlighted and perhaps reminded you of old challenges, aiding in a successful migration.”

No Comments

Leave a Reply

Your email address will not be published. Required fields are marked *

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Free(BSD)
Search for duplicated files

This will be short (but hopefully I will find more time to show entire process to search duplicated files together with some examples). In case you are searching for duplicated files I can recommend two software which actually rocks in openSource world

Azure
NFS issue, cannot be mounted or is not visible

The same kind of issue I have encountered numerous times while working across different environments and with various customers. The problem with NFS mounts connected from remote locations is so common. This issue extends beyond communication solely over WAN and also include connections between datacenters (DC) where we lack control …

Azure
Why Firefox is important and people should use this browser in 2024, my thoughts.

Can you remember the times when everyone was using Internet Explorer? Back in the ’90s and the early part of this century, Internet Explorer dominated the browser market. Software Incompatibility with Other Browsers Incompatibility issues with software and other browsers have been a persistent problem. Even in 2022, this remains …