How to Use cmmds-tool

Step 1: Access any vm in your vSAN datastore

Step 2: Look at the options of cmmds-tool

Step 3: Query the object id to find out its policy

Step 4: Get additional information by using other options

How to use objtool to manipulate a vSAN object

One of the coolest (and most dangerous) commands in vSAN is objtool. Since this datastore is not vmfs but vsanfs (and object based), the objtool command is the utility that performs different tasks such as creating, resizing and deleting objects.

What follows is a series of examples on how to use such command:

# find / -name objtool
/usr/lib/vmware/osfs/bin/objtool

# cd /usr/lib/vmware/osfs/bin

Want to learn how to use the command?

# ./objtool --help

usage: ./objtool <cmd> <options>

Commands:

create    Create an object.

  -s/--size n              size in bytes (or KB, MB, GB or TB)

  -n/--name <string>       Optional user-friendly name.

  -m/--haMetaData <string> Optional HA metadata.

  -p/--policy <string>     Optional VSAN policy.

  -g/--groupUuid <string>  Optional groupUuid of the object.

  -a/--allocType <value>   Allocation Type.

                           0 - Thick (default)

                           1 - Eager Zeroed Thick

                           2 - Zeroed Thick

                           3 - Thin

  --capabilities <value>   Capability flags.

                           0 - None (default)

                           1 - Strict GWE (VsanSparse)

  -t/--objClass <value>    Object Class.

                           0 - vdisk (default)

                           1 - snapshot

                           2 - vmnamespace

                           3 - vmswap

                           4 - vmem

                           5 - sidecar

                           6 - hbrpersist

                           7 - coredump

delete     Delete an object.

  -u/--uuid          UUID of the object

  -f/--forceDelete   Optional force delete mode

open       Create a node for an object in DevFS.

  -u/--uuid          UUID of the object

setAttr    Set attributes of an object

  -u/--uuid                UUID of the object

  -n/--name <string>       Optional user-friendly name.

  -m/--haMetaData <string> Optional HA metadata.

  -d/--descPath <string>   Optional descriptor path.

  --capabilities <value>   Capability flags.

                           0 - None (default)

                           1 - Strict GWE (VsanSparse)

getAttr    Get attributes of an object

  -u/--uuid                UUID of the object

  -c/--isComponent         Specified uuid is a component uuid

  -x/--diskUuid            Disk uuid of the component

  --bypassDom              Bypass DOM and read from LSOM

  --format                 Pretty print the output

                           (Currently, only JSON)

resize     Resize an object

  -u/--uuid          UUID of the object

  -s/--size n        size in bytes (or KB, MB, GB or TB)

  -l/--lockFilePath <path> Lock file path for the object

truncate   Truncate an object

  -u/--uuid          UUID of the object

  -s/--size n        size in bytes (or KB, MB, GB or TB)

setPolicy  Set policy of an object

  -u/--uuid             UUID of the object.

  -p/--policy <string>  VSAN policy.

setMirror  Set the read mirror for an object

  -u/--uuid          UUID of the object

  -r/--readMirror n  mirror to use (-1 to unset)

reset  Reset object.

  -u/--uuid   UUID of the object

Global Options

  -v/--verbose       Verbosity level.

  -q/--quiet         No need to print success messages

  -h/--help          Print this message.

# cat /vmfs/volumes/vsanDatastore/testvm/testvm.vmdk

# Disk DescriptorFile

version=4

encoding="UTF-8"

CID=d68d61b2

parentCID=ffffffff

isNativeSnapshot="no"

createType="vmfs"

# Extent description

RW 2097152 VMFS "vsan://2cc4ca59-f4d8-43c3-9a0b-005056013df7" <-- notice this object id

# The Disk Data Base

#DDB

ddb.adapterType = "lsilogic"

ddb.deletable = "true"

ddb.geometry.cylinders = "512"

ddb.geometry.heads = "128"

ddb.geometry.sectors = "32"

ddb.longContentID = "14bf5c5d81a00d863d0b300fd68d61b2"

ddb.thinProvisioned = "1"

ddb.uuid = "60 00 C2 9b 2d 72 3f 5c-1c 8a 12 5f ca f9 2d 38"

ddb.virtualHWVersion = "13"

Want to delete an object? Better have a backup...

# ./objtool delete -u 2cc4ca59-f4d8-43c3-9a0b-005056013df7 -f

Will this delete the object? You bet... Better have a good backup.

Notice that after the components for the virtual disk object are no longer present.

What about renaming an object? Objtool to the rescue...

# ls -lh

total 5120

drwxr-xr-t    1 root     root        2.1K Sep 26 21:18 26c4ca59-1028-c0c6-e31d-005056013df7

drwxr-xr-t    1 root     root        1.9K Sep 12 15:38 3fc7ad59-ef19-ba92-6088-005056013e28

drwxr-xr-t    1 root     root        1.4K Sep 26 17:56 6986ca59-70d0-0591-c8ff-005056013df7

drwxr-xr-t    1 root     root        3.0K Sep 26 21:18 bfc6ad59-b712-25d2-8939-005056013e24

drwxr-xr-t    1 root     root        1.9K Sep 15 17:40 cac7ad59-f2ac-2d3c-e054-005056013e28

lrwxr-xr-x    1 root     root          36 Sep 26 21:27 javier01 -> 6986ca59-70d0-0591-c8ff-005056013df7

lrwxr-xr-x    1 root     root          36 Sep 26 21:27 testvm -> 26c4ca59-1028-c0c6-e31d-005056013df7           <--notice this line and the name

lrwxr-xr-x    1 root     root          36 Sep 26 21:27 ub1404lts1 -> bfc6ad59-b712-25d2-8939-005056013e24

lrwxr-xr-x    1 root     root          36 Sep 26 21:27 ub1404lts2 -> 3fc7ad59-ef19-ba92-6088-005056013e28

lrwxr-xr-x    1 root     root          36 Sep 26 21:27 ub1404lts5 -> cac7ad59-f2ac-2d3c-e054-005056013e28

# /usr/lib/vmware/osfs/bin/objtool setAttr -u 26c4ca59-1028-c0c6-e31d-005056013df7 -n newtestvm

Object set attribute succeeded

# /etc/init.d/osfsd restart

watchdog-osfsd: Terminating watchdog process with PID 312029

Waiting for process to terminate...

osfsd stopped

osfsd started

# ls -lh

total 5120

drwxr-xr-t    1 root     root        2.1K Sep 26 21:18 26c4ca59-1028-c0c6-e31d-005056013df7

drwxr-xr-t    1 root     root        1.9K Sep 12 15:38 3fc7ad59-ef19-ba92-6088-005056013e28

drwxr-xr-t    1 root     root        1.4K Sep 26 17:56 6986ca59-70d0-0591-c8ff-005056013df7

drwxr-xr-t    1 root     root        3.0K Sep 26 21:18 bfc6ad59-b712-25d2-8939-005056013e24

drwxr-xr-t    1 root     root        1.9K Sep 15 17:40 cac7ad59-f2ac-2d3c-e054-005056013e28

lrwxr-xr-x    1 root     root          36 Sep 26 21:30 javier01 -> 6986ca59-70d0-0591-c8ff-005056013df7

lrwxr-xr-x    1 root     root          36 Sep 26 21:30 newtestvm -> 26c4ca59-1028-c0c6-e31d-005056013df7   <-- notice the new name

lrwxr-xr-x    1 root     root          36 Sep 26 21:30 ub1404lts1 -> bfc6ad59-b712-25d2-8939-005056013e24

lrwxr-xr-x    1 root     root          36 Sep 26 21:30 ub1404lts2 -> 3fc7ad59-ef19-ba92-6088-005056013e28

lrwxr-xr-x    1 root     root          36 Sep 26 21:30 ub1404lts5 -> cac7ad59-f2ac-2d3c-e054-005056013e28

Want information about an object?

# /usr/lib/vmware/osfs/bin/objtool getAttr -u 26c4ca59-1028-c0c6-e31d-005056013df7

Object Attributes --

UUID:26c4ca59-1028-c0c6-e31d-005056013df7

Object type:vsan

Object size:273804165120

User friendly name:newtestvm

HA metadata:(null)

Allocation type:Thick

Policy:((\"stripeWidth\" i1) (\"cacheReservation\" i0) (\"proportionalCapacity\" i0) (\"hostFailuresToTolerate\" i1) (\"forceProvisioning\" i0) (\"spbmProfileId\" \"aa6d5a82-1c88-45da-85d3-3d74b91a5bad\") (\"spbmProfileGenerationNumber\" l+0) (\"spbmProfileName\" \"vSAN Default Storage Policy\"))

Object class: vmnamespace

Object capabilities: NONE

Object path: /vmfs/volumes/vsan:521305e4430dd845-d59b1943736fdc88/

And how do I create an object?

# /usr/lib/vmware/osfs/bin/objtool create -s 1KB -a 1 -t 0-n javierobject

Creating object of size 1024 bytes
Setting object class to 0
UUID:3184dd59-c0ea-6ab8-b241-005056013df7how do I create my own object?



Note:  The previous command creates an object using the default policy.  This could be verified also with the rvc.

Note: Add the -t 0 option to the previous command to create a vdisk object. This can also be verified with the rvc.

How to create and remove directories in a vSAN datastore with osfs-mkdir and osfs-rmdir

Creating and removing directories in a vSAN datastore is not as simple as using mkdir and rmdir or rm -rf because the vSAN datastore is object based. In order to do that, special vSAN related commands exist to perform such tasks.

Notice what happens if you try to use the mkdir command:

# cd /vmfs/volumes/vsanDatastore
# mkdir testdir
mkdir: can't create directory 'testdir': Function not implemented

The commands that follow perform the same tasks as the icon with the green plus sign on the web client. These commands should be used if the vCenter server is not available.

How to do it:

Step 1: Access the folder that contains the utilities

# cd /usr/lib/vmware/osfs/bin

Step 2: List the contents of that directory

# ls

objtool     osfs-ls     osfs-mkdir  osfs-rmdir  osfsd

Step 3: Verify that a directory called testdir does not exist

# ls -lh /vmfs/volumes/vsanDatastore/testdir

ls: /vmfs/volumes/vsanDatastore/testdir: No such file or directory

Step 4: Create such directory

# ./osfs-mkdir /vmfs/volumes/vsanDatastore/testdir

92c0ca59-0c9a-84f0-d2f0-005056013e28

Step 5: Verify that it exists

# ls -lh /vmfs/volumes/vsanDatastore/testdir

lrwxr-xr-x    1 root     root          36 Sep 26 21:03 /vmfs/volumes/vsanDatastore/testdir -> 92c0ca59-0c9a-84f0-d2f0-005056013e28

Step 6: Delete the directory

# ./osfs-rmdir /vmfs/volumes/vsanDatastore/testdir

Deleting directory 92c0ca59-0c9a-84f0-d2f0-005056013e28 in container id 521305e4430dd845d59b1943736fdc88 backed by vsan (force=False)

Step 7: Verify that it is gone

# ls -lh /vmfs/volumes/vsanDatastore/testdir

ls: /vmfs/volumes/vsanDatastore/testdir: No such file or directory

VMworld 2017 Photos

Some photos for those who could not attend ...

It all starts with registration.

The calm before the storm.

Getting ready to greet participants...

Pat in action...

Nice Backpacks guys :)

Understanding PFTT and SFTT

vSAN 6.6 New Features

Here is what my cluster looks like:

This policy illustrates how to use primary and secondary failures to tolerate. The vms using this policy would be mirrored across sites and within each site as well.

Notice the vm is mirrored across sites and each replica in turn is also mirrored within each site. Notice the number and placement of the components and the witnesses.

This example demonstrates how to have a vm that is not mirrored across sites but is mirrored within one site.

\

Notice the 2 replica copies and the witness are in one site. The two components and the witness are all in the same site.

Pretty cool indeed !!!

VMworld 2017 is coming

See U there :)

Viva Las Vegas...

How to Enable Encryption on vSAN 6.6

Encrypting a vSAN 6.6 Datastore

Step 1: Select your vCenter Server, click on Configure, select Key Management Servers and click on the Green Plus Sign. Specify the name of the KMS cluster, the IP address and a port to use. Currently, there are two supported KMS Servers (Hytrust and EMC). The KMS servers need to be KMIP 1.1 compliant.

Step 2: You will have to establish a trust relationship with the KMS server. Since different KMS servers are supported, you will have to select the type of certificate to download. Different choices are available.

Step 3: Verify that the connection state is Normal and that the procedure succeeded.

Step 4: Select your vSAN cluster, click on Configure, select General and click on Edit. Enable Encryption. The KMS related information should be automatically populated. Click on OK.

Step 5: Once you enable Encryption, every disk will be reformatted. This process will take time. The amount of time will depend on how many drives need to be formatted and the size of the drives.

Once this is done, the entire datastore is encrypted. Encryption works with both the hybrid solution as well as the all-flash. If new servers are added to the cluster, the disk groups created on the new host will be formatted to support encryption.

Inducing Failures on a Virtual SAN cluster

The coolness of python scripts.

One of the many tools available for testing Virtual SAN is a buried python script called vsan.DiskFaultInjection.pyc. Located in the /usr/lib/vmware/vsan/bin directory, this utility can generate permanent or transient errors. Furthermore, it can emulate unplugging of disks.

Using the -h option (for help), an administrator can see the options available for this command. Only to be used pre-production, this script can generate failures to allow the user to understand what happens in such cases.

Below is an example of what happens when a capacity disk is affected by such permanent failure. In the case of a raid5 virtual machine, the virtual machine would continue to run. If enough servers and/or disks are available, the rebuilding of the date would take place immediately. The -p option is used for permanent errors and the -u option to unplug a disk.

Errors would be seen everywhere, notice the capture below.

The -c (clear option) is used to remove the permanent error. If using the -u option, simply rescan the storage via esxcfg-rescan -A.

Installing the vCenter Appliance on a One Node vSAN Cluster

Virtual SAN 6.6 introduces a graphical method to install a vCenter appliance on a freshly installed esxi host in order to eventually install and configure v vSAN cluster. The required software versions are: ESXi 5310538 , VC 5318154

As you start a fresh install, notice that the latest version of vSphere 6.5 introduces a new option that allows to "Install on a new Virtual SAN cluster containing the target host". Proceed with a normal installation.

Select the vCenter option with the embedded PSC.

Select the esxi host that will host the new vCenter appliance.

Name the appliance and provide the root password.

Here is where you see the big difference. Notice the option at the end. Select it.

Name your future datacenter and cluster.

Specify which drives will be used for the Virtual SAN datastore. Indicate which drives will be used for cache and capacity.

The rest is pretty much the same, provide the network related information and continue  as usual.

Once the installation is done, the administrator can verify that the vCenter is in working order.

Once the vCenter appliance is running, log in, create the vmkernel port for Virtual SAN on that node and proceed as usual. Add the remaining servers and their vsan ip addresses and you are done.

When 100% cpu utilization is not really 100%

100% does not always mean 100%

Some people mistakenly look at tools inside of a guest operating system (for example,  the task manager) and when faced with 100% cpu utilization, they automatically believe that such virtual machine needs more vcpus.  Not necessarily. You really need to look at what is taken place on the host and compare the results. Remember that the guest OS is not aware of what is actually happening on the host. 

Notice that this case this virtual machine running Windows displays 100% cpu utilization. 

However, notice that the esxi host does not have any of the logical cpus at 100% and that virtual machine is NOT using 100% of the actual lcpu (core). Notice the %MLMTD column and %RDY.

In this case, the reason is due to a cpu limit. Notice the capture below. This virtual machine has the limit set to 50% of the maximum number of cpu mhz. Yet, the guest OS is not aware of this.

How To Install Esxi From USB

Installing esxi to a usb device is simple, just insert a cd and during the installation point to a usb device. This shows how to prepare a bootable usb device that allows you to install esxi onto any type of device.

Step 1: Launch your browser and point to https://unetbootin.github.io/

Step 2: Download and launch unetbootin. Point to your .iso image and click OK.

Step 3: Click on Exit.

Step  4: Verify the contents of your USB drive.

Step 5: Boot your future esxi host from USB and start the installation.

PowerCLI using Linux

How to Install and Use PowerCLI from Linux. 

Disclaimer: 

Don't expect to find all the commands typically found in the Windows counterpart.

Good luck finding autodeploy, imagebuilder and vum commands.

1. From a Linux system (Ubuntu in my case), launch your browser and to to labs.vmware.com.

2. Find PowerCLI Core and download it. In the process, download the Instructions.pdf.

3. Download PowerShell for Linux using the curl command.

# curl -SLO https://github.com/PowerShell/PowerShell/releases/download/v6.0.0-alpha.12/powershell_6.0.0.-alpha.12-11ubuntu1.14.04.1_amd64.deb

4. Install the package by force using the dpkg command.

# sudo dpkg -i powershell*.deb
# sudo apt-get install -f

5. Verify the package is install with the dpkg -l command

6. Create the Modules directory for PowerCLI

# mkdir -p ~/.local/share/powershell/Modules

7. Copy the PowerCLI file and extract it.

# cp PowerCLI_Core* ~/local/share/powershell/Modules

8. Extract the zip files, including PowerCLI.ViCore and PowerCLI.Vds. Verify the contents.

# cd ~/local/share/powershell/Modules
# unzip PowerCLI.ViCore.zip
# unzip PowerCLI.Vds.zip

9. Launch powershell

10. Import the PowerCLI Modules

11. Verify the are loaded

12. Instruct powercli to ignore invalid certificates and use it.

Some useful links:

http://www.virtuallyghetto.com/2016/09/vmware-powercli-for-mac-os-x-linux-more-yes-please.html

https://www.vmguru.com/2016/10/powercli-os-x-linux-fling/