vROps/Log Insight Integration and Troubleshooting

Update: Special thanks to Yogita N. Patil and VMware Technical Support for their assistance with the issues below!

Last week, I was trying to integrate vRealize Log Insight with vRealize Operations (vROps) so that I could ‘launch in context’ from vROps. This adds a context-sensitive action to vROps that lets you pull up Log Insight’s Interactive Analysis feature against the alert or object you are currently viewing. This makes it easy to drill down into logs with a lot less clicking around:

Launch in context is a feature in vRealize Operations Manager that lets you launch an external application via URL in a specific context. The context is defined by the active UI element and object selection. Launch in context lets the vRealize Log Insight adapter add menu items to a number of different views within the Custom user interface and the vSphere user interface of vRealize Operations Manager.

The documentation to enable this features seems pretty simple. I ran into a few problems, though…

The requirements are pretty simple, but were the first thing to trip me up. You want to be on Log Insight 3.6 and vROps 6.3. While Log Insight had been upgraded a day or two earlier, vROps was at 6.1. When performing the upgrade of vROps, it did not register its extension properly! Going into the Managed Object Browser showed there was still a vCOps 6.1 registration instance (yes, the extension is still called vCOps!). In addition, the extension was registered by IP, not by DNS. The extension needs to be in place for the steps below, or you receive even more opaque error messages, so I encourage you to verify it now. You can investigate your own MOB at a link similar to https://vcsa.fqdn.example.com/mob, and specifically look at the vROps extension at https://vcsa.fqdn.example.com/mob/?moid=ExtensionManager&doPath=extensionList%5B“com.vmware.vcops”%5D.client

Continue reading

Tag-based Veeam Backups

As I teased in Using vCenter tags with PowerCLI, I want to explore how to use tags  in a practical application. To refresh our memory, we looked at creating an Ownership category with individual tags in it, and limited VMs to having just one of the tags. We created a little script that defines our schema, in case we need to re-create it. We are going to work on a new category today for our backups. Specifically, Veeam backups, based on SDDC6282-SPO, Using vSphere tags for advanced policy-driven data protection as presented at VMworld 2015.

Defining Policy and Tags

To create our backup jobs, we need to know a few things that will translate into our tag schema. Our backup policies are defined by a combination of ownership, recovery point objective (RPO), and the retention period. For example, our Development group is okay with a 24 hour RPO and backups that are retained for a week. Operations may require a 4 or 8 hour RPO and require 30 days of backups. Each of those combinations will require a separate backup job. We can combine these tuples of information into individual tags so that Veeam can make use of them. We also need one more tag for VMs that do not need any backup at all. We can put all of this in a tag category called VeeamPolicy. Here’s what that might look like, again in PowerShell:

New-TagCategory -Name VeeamPolicy -Description "Veeam Backup Policy" -Cardinality Single -EntityType VirtualMachine
New-Tag -Name "NoRPO" -Category VeeamPolicy -Description "This VM has no backups"
New-Tag -Name "Development24h7d" -Category VeeamPolicy -Description "Development VMs with 24 hour RPO, 7 days retention"
New-Tag -Name "Operations8h30d" -Category VeeamPolicy -Description "Operations VM with 8 hour RPO, 30 day retention"
New-Tag -Name "Sales48h30d" -Category VeeamPolicy -Description "Sales VM with 48 hour RPO, 30 day retention"

Continue reading

Using vCenter tags with PowerCLI

I was recently introduced to some practical usage of vCenter tags. I used tags to build a fairly easy and dynamic set of sources for Veeam backup jobs, but before I get into that, I want to explain tags a little bit for those who are not familiar with them.

Tags are something that are pretty easy to understand conceptually. You create a category which contains a few tags. An object can receive one or multiple tags from a given category. Tags are managed by vCenter, not by vSphere, so require a license that provides vCenter for management. That’s a pretty simple explanation and list of requirements.

A Tag Category has a collection of Tags available within it. If the Category is “single cardinality”, it means that an object can only be assigned one tag in the category. This might be good for a category associated with ownership or recovery point objective (RPO). A Category can also be “multiple cardinality”, where a single object can be assigned multiple tags in a category. This would be helpful to associate applications with a VM, or a list of support teams that need notified if there is a planned impact or outage.

I’m going to show you how to manage Tags and Tag Categories with PowerCLI (Specifically with the PowerShell ISE and a profile to load PowerCLI), but you can manually create and manage them through the vSphere Web Client, under the Tags item on the left hand menu. You can create and delete and rename tags and categories there all day long, and you can right click on an object almost anywhere else and select Edit Tags to edit the assigned tags on it. When you view most objects, you’ll see an area in the Summary tab labeled Tags that will display and let you manage assignments.

Continue reading

Announcement: Github repo for common vCenter roles

Last week, I was installing some of the vRealize Suite components and was creating accounts for each component using the Principle of Least Privilege. I was sometimes able to find vendor documentation on the required permissions, sometimes I found a few blog posts where people guessed at the required permissions, but in almost no cases was I able to find automated role creation. Perhaps my google-fu is poor! Regardless, I thought it would be nice to have documented and automated role creation in a single place.

To that end, I created a repo on GitHub called vCenter-roles. I use PowerCLI to create a role with the correct permissions, and only the correct permissions. Each cmdlet will allow you to specify the role name or it will use a default. For instance, to create a role for Log Insight, just run the attached ps1 script followed by the command:

New-LogInsightRole

It’s that easy!

I will be adding some other vRealize Suite roles as I work my way through the installation, but there are tons of other common applications out there that require their own role, and not just VMware’s own applications! I encourage you to open an issue or submit a Pull Request (PR) for any applications you use. The more roles we can collect in one place, the more helpful it is for the greater community. Thanks!

Deploying Windows Images with KMS keys

I’m not all that familiar with Windows licensing models, so I stumbled into a bit of surprise with KMS keys recently. If you are using a central KMS server that you do not maintain, and someone gives you a KMS key, you can ignore it! That’s for the KMS Host, which is where the licensing happens. Your nodes will be KMS Clients and they will use a Generic Volume License Key for activation. The Client communicates with the Host, which tells the client if it is activated and provides all the necessary information for that to happen (I don’t know how the Host does that, that’s the beauty of letting someone else run that service!). In this case, you are often given media to use for the Windows install that includes the GVLK, so you don’t need to do anything but communicate with the KMS Host. It’s a pretty nice setup, all considering.

However, IF you do something silly like put the KMS Host key on your Clients, you won’t get far. The Host key can only be activated 10 times on 6 hosts, so very soon you’ll run into trouble, if not immediately. You have to switch back over to the GVLK and activate using that. Microsoft maintains a list of GVLKs for each edition of Windows. The lookup of the KMS Host is done by DNS, but you can manually configure the KMS Client as well. Once the GVLK is in place, activate the key. Here are the three commands you will need, using Windows 2012R2 Datacenter as the GVLK:

cscript c:\windows\system32\slmgr.vbs /ipk W3GGN-FT8W3-Y4M27-J84CP-Q3VJ9
cscript c:\windows\system32\slmgr.vbs /skms kms.example.com:1688
cscript c:\windows\system32\slmgr.vbs /ato

These commands need to be run from an administrator-privilege command prompt or PowerShell session.

If you are using templates, run the first command on the template. Ensure the deployment process is not adding license information. In vCenter, this means removing all options from the License Information portion of the Customization Specifications in ALL customization specs. Add the /skms and /ato commands to the existing commands in the Run Once section:

KMS Fig 1

KMS Fig 2

When you deploy a VM, it should now automatically activate itself! If you run into issues, ensure that the Client can communicate with the Host and no firewalls are blocking the communication. I’ve found that a global any/<KMS Server>/<kms port> rule in your firewalls is handy to ensure that random networks aren’t blocked from activation.

If you’re interested in learning more about Windows Licensing, Microsoft has a great amount of documentation. I suggest starting with Learn About Product Activation and then moving through the relevant sections.

vCenter 6 permissions model changes

If you have not yet upgraded from vCenter 5.5 to 6, you may want to give it a shot. vCenter 6 is much faster and introduces a lot of nice UI improvements. One of my favorites is the relatively minor change that when you delete a VM, it displays the VM name in the dialog instead of “this object”. In any case, it’s pretty swanky and I think you’ll like it.

However, there is one extremely noticeable, but (in my opinion) poorly documented change, in the permissions model for non-Administrator users and groups. In 5.x, you could assign user rights to a folder or VM and the user would be granted the proper level of access to the folder members or the individual VM. If this was your folder layout…

vCenter 6 Permissions 1

…and you wanted to grant Virtual Machine Power User access to the VM autodeploy-test, you could grant that permission to a user or group on the folder or VM here and the user could manipulate their VM just fine:

vCenter 6 Permissions 2

After upgrading to 6.0, the same user would not have permission to access the folder or VM. They might simply see Empty Inventory when looking at the VM and Templates view. To remedy this, you need to grant non-propagating permissions at the intermediate Datacenter tier (you do NOT need to grant permissions to the vCenter object). These permissions can be as limited as read-only in most cases, but as they are not propagating, you can use the same permission level (e.g. Virtual Machine Power User). The new permission is highlighted in red:

vCenter 6 Permissions 3

If you have multiple intermediate folders, you’ll need to assign the non-propagating permissions at every level. You’ll need to do the same on the Hosts and Clusters, Datastores, and Network views as well. In an additional wrinkle, Network view permissions for Distributed vSwitches cannot be assigned on the DVS, they need to be assigned on a folder containing the DVS and optionally on an individual Distributed PortGroup.

I have heard rumors that this new “feature” may be fixed soon, but until then, these changes are required for non-Administrator users to maintain their permission levels.

Update vSphere host with esxcli

I’ve seen a number of articles describing how to update a vSphere host from the CLI. The problem is that all the articles I found pointed to a particular profile (5.1U3, 6.0GA, etc), which quickly gets out of date. After some searching, I figured out how to list the available profiles from a depot so I could choose the one I want, then upgrade the host. I hope this helps you now and in the future!

First, make sure your host is in maintenance mode and all VMs are migrated off, otherwise you will have an interruption of service. If you only have a single host, this may be unavoidable, and you should let your users know about the upcoming interruption. You also need to have the ESXi shell enabled, and ssh if you don’t have console/out-of-band access.

Second, list the available profiles in the specified depot (in this case, VMware’s Online Depository). The results seem to be unordered, so use grep and sort to help out. For instance, I want the latest 6.0.0 version:

[root@esxi02:~] esxcli software sources profile list -d https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/vmw-depot-index.xml | grep 6.0.0 | sort
ESXi-6.0.0-20150404001-no-tools   VMware, Inc.  PartnerSupported
ESXi-6.0.0-20150404001-standard   VMware, Inc.  PartnerSupported
ESXi-6.0.0-20150504001-no-tools   VMware, Inc.  PartnerSupported
ESXi-6.0.0-20150504001-standard   VMware, Inc.  PartnerSupported
ESXi-6.0.0-20150701001s-no-tools  VMware, Inc.  PartnerSupported
ESXi-6.0.0-20150701001s-standard  VMware, Inc.  PartnerSupported
ESXi-6.0.0-20150704001-no-tools   VMware, Inc.  PartnerSupported
ESXi-6.0.0-20150704001-standard   VMware, Inc.  PartnerSupported
ESXi-6.0.0-20150901001s-no-tools  VMware, Inc.  PartnerSupported
ESXi-6.0.0-20150901001s-standard  VMware, Inc.  PartnerSupported
ESXi-6.0.0-20150902001-no-tools   VMware, Inc.  PartnerSupported
ESXi-6.0.0-20150902001-standard   VMware, Inc.  PartnerSupported
ESXi-6.0.0-20151004001-no-tools   VMware, Inc.  PartnerSupported
ESXi-6.0.0-20151004001-standard   VMware, Inc.  PartnerSupported
ESXi-6.0.0-2494585-no-tools       VMware, Inc.  PartnerSupported
ESXi-6.0.0-2494585-standard       VMware, Inc.  PartnerSupported

Pick the version you want, then use esxcli to perform the upgrade. You won’t get any feedback during the upgrade, which can take 5+ minutes, then you get hit with the wall of text. Don’t get anxious until at least 15 minutes go by. The important feedback is at the top of the output and is emphasized in bold:

[root@esxi02:~] esxcli software profile update -d https://hostupdate.vmware.com/software/VUM/PRODUCTION/main/vmw-depot-index.xml -p ESXi-6.0.0-20151004001-standard
Update Result
   Message: The update completed successfully, but the system needs to be rebooted for the changes to be effective.
   Reboot Required: true
   VIBs Installed: VMware_bootbank_esx-base_6.0.0-1.20.3073146, VMware_bootbank_lsi-mr3_6.605.08.00-7vmw.600.1.17.3029758, VMware_
bootbank_lsi-msgpt3_06.255.12.00-8vmw.600.1.17.3029758, VMware_bootbank_lsu-lsi-lsi-mr3-plugin_1.0.0-2vmw.600.0.11.2809209, VMware
_bootbank_lsu-lsi-megaraid-sas-plugin_1.0.0-2vmw.600.0.11.2809209, VMware_bootbank_lsu-lsi-mpt2sas-plugin_1.0.0-4vmw.600.1.17.3029
758, VMware_bootbank_misc-drivers_6.0.0-1.17.3029758, VMware_bootbank_nvme_1.0e.0.35-1vmw.600.1.17.3029758, VMware_bootbank_sata-a
hci_3.0-22vmw.600.1.17.3029758, VMware_bootbank_scsi-bnx2i_2.78.76.v60.8-1vmw.600.0.11.2809209, VMware_bootbank_vsanhealth_6.0.0-3
000000.2.0.1.17.2972216, VMware_bootbank_xhci-xhci_1.0-2vmw.600.1.17.3029758, VMware_locker_tools-light_6.0.0-1.17.3029758
   VIBs Removed: VMware_bootbank_esx-base_6.0.0-0.6.2715440, VMware_bootbank_lsi-mr3_6.605.08.00-6vmw.600.0.0.2494585, VMware_boot
bank_lsi-msgpt3_06.255.12.00-7vmw.600.0.0.2494585, VMware_bootbank_lsu-lsi-lsi-mr3-plugin_1.0.0-1vmw.600.0.0.2494585, VMware_bootb
ank_lsu-lsi-megaraid-sas-plugin_1.0.0-1vmw.600.0.0.2494585, VMware_bootbank_lsu-lsi-mpt2sas-plugin_1.0.0-1vmw.600.0.0.2494585, VMw
are_bootbank_misc-drivers_6.0.0-0.0.2494585, VMware_bootbank_nvme_1.0e.0.35-1vmw.600.0.0.2494585, VMware_bootbank_sata-ahci_3.0-21
vmw.600.0.0.2494585, VMware_bootbank_scsi-bnx2i_2.78.76.v60.8-1vmw.600.0.0.2494585, VMware_bootbank_xhci-xhci_1.0-2vmw.600.0.0.249
4585, VMware_locker_tools-light_6.0.0-0.0.2494585
   VIBs Skipped: VMWARE_bootbank_mtip32xx-native_3.8.5-1vmw.600.0.0.2494585, VMware_bootbank_ata-pata-amd_0.3.10-3vmw.600.0.0.2494
585, VMware_bootbank_ata-pata-atiixp_0.4.6-4vmw.600.0.0.2494585, VMware_bootbank_ata-pata-cmd64x_0.2.5-3vmw.600.0.0.2494585, VMwar
e_bootbank_ata-pata-hpt3x2n_0.3.4-3vmw.600.0.0.2494585, VMware_bootbank_ata-pata-pdc2027x_1.0-3vmw.600.0.0.2494585, VMware_bootban
k_ata-pata-serverworks_0.4.3-3vmw.600.0.0.2494585, VMware_bootbank_ata-pata-sil680_0.4.8-3vmw.600.0.0.2494585, VMware_bootbank_ata
-pata-via_0.3.3-2vmw.600.0.0.2494585, VMware_bootbank_block-cciss_3.6.14-10vmw.600.0.0.2494585, VMware_bootbank_cpu-microcode_6.0.
0-0.0.2494585, VMware_bootbank_ehci-ehci-hcd_1.0-3vmw.600.0.0.2494585, VMware_bootbank_elxnet_10.2.309.6v-1vmw.600.0.0.2494585, VM
ware_bootbank_emulex-esx-elxnetcli_10.2.309.6v-0.0.2494585, VMware_bootbank_esx-dvfilter-generic-fastpath_6.0.0-0.0.2494585, VMwar
e_bootbank_esx-tboot_6.0.0-0.0.2494585, VMware_bootbank_esx-xserver_6.0.0-0.0.2494585, VMware_bootbank_ima-qla4xxx_2.02.18-1vmw.60
0.0.0.2494585, VMware_bootbank_ipmi-ipmi-devintf_39.1-4vmw.600.0.0.2494585, VMware_bootbank_ipmi-ipmi-msghandler_39.1-4vmw.600.0.0
.2494585, VMware_bootbank_ipmi-ipmi-si-drv_39.1-4vmw.600.0.0.2494585, VMware_bootbank_lpfc_10.2.309.8-2vmw.600.0.0.2494585, VMware
_bootbank_lsu-hp-hpsa-plugin_1.0.0-1vmw.600.0.0.2494585, VMware_bootbank_lsu-lsi-lsi-msgpt3-plugin_1.0.0-1vmw.600.0.0.2494585, VMw
are_bootbank_lsu-lsi-mptsas-plugin_1.0.0-1vmw.600.0.0.2494585, VMware_bootbank_misc-cnic-register_1.78.75.v60.7-1vmw.600.0.0.24945
85, VMware_bootbank_net-bnx2_2.2.4f.v60.10-1vmw.600.0.0.2494585, VMware_bootbank_net-bnx2x_1.78.80.v60.12-1vmw.600.0.0.2494585, VM
ware_bootbank_net-cnic_1.78.76.v60.13-2vmw.600.0.0.2494585, VMware_bootbank_net-e1000_8.0.3.1-5vmw.600.0.0.2494585, VMware_bootban
k_net-e1000e_2.5.4-6vmw.600.0.0.2494585, VMware_bootbank_net-enic_2.1.2.38-2vmw.600.0.0.2494585, VMware_bootbank_net-forcedeth_0.6
1-2vmw.600.0.0.2494585, VMware_bootbank_net-igb_5.0.5.1.1-5vmw.600.0.0.2494585, VMware_bootbank_net-ixgbe_3.7.13.7.14iov-20vmw.600
.0.0.2494585, VMware_bootbank_net-mlx4-core_1.9.7.0-1vmw.600.0.0.2494585, VMware_bootbank_net-mlx4-en_1.9.7.0-1vmw.600.0.0.2494585
, VMware_bootbank_net-nx-nic_5.0.621-5vmw.600.0.0.2494585, VMware_bootbank_net-tg3_3.131d.v60.4-1vmw.600.0.0.2494585, VMware_bootb
ank_net-vmxnet3_1.1.3.0-3vmw.600.0.0.2494585, VMware_bootbank_nmlx4-core_3.0.0.0-1vmw.600.0.0.2494585, VMware_bootbank_nmlx4-en_3.
0.0.0-1vmw.600.0.0.2494585, VMware_bootbank_nmlx4-rdma_3.0.0.0-1vmw.600.0.0.2494585, VMware_bootbank_ohci-usb-ohci_1.0-3vmw.600.0.
0.2494585, VMware_bootbank_qlnativefc_2.0.12.0-5vmw.600.0.0.2494585, VMware_bootbank_rste_2.0.2.0088-4vmw.600.0.0.2494585, VMware_
bootbank_sata-ata-piix_2.12-10vmw.600.0.0.2494585, VMware_bootbank_sata-sata-nv_3.5-4vmw.600.0.0.2494585, VMware_bootbank_sata-sat
a-promise_2.12-3vmw.600.0.0.2494585, VMware_bootbank_sata-sata-sil24_1.1-1vmw.600.0.0.2494585, VMware_bootbank_sata-sata-sil_2.3-4
vmw.600.0.0.2494585, VMware_bootbank_sata-sata-svw_2.3-3vmw.600.0.0.2494585, VMware_bootbank_scsi-aacraid_1.1.5.1-9vmw.600.0.0.249
4585, VMware_bootbank_scsi-adp94xx_1.0.8.12-6vmw.600.0.0.2494585, VMware_bootbank_scsi-aic79xx_3.1-5vmw.600.0.0.2494585, VMware_bo
otbank_scsi-bnx2fc_1.78.78.v60.8-1vmw.600.0.0.2494585, VMware_bootbank_scsi-fnic_1.5.0.45-3vmw.600.0.0.2494585, VMware_bootbank_sc
si-hpsa_6.0.0.44-4vmw.600.0.0.2494585, VMware_bootbank_scsi-ips_7.12.05-4vmw.600.0.0.2494585, VMware_bootbank_scsi-megaraid-mbox_2
.20.5.1-6vmw.600.0.0.2494585, VMware_bootbank_scsi-megaraid-sas_6.603.55.00-2vmw.600.0.0.2494585, VMware_bootbank_scsi-megaraid2_2
.00.4-9vmw.600.0.0.2494585, VMware_bootbank_scsi-mpt2sas_19.00.00.00-1vmw.600.0.0.2494585, VMware_bootbank_scsi-mptsas_4.23.01.00-
9vmw.600.0.0.2494585, VMware_bootbank_scsi-mptspi_4.23.01.00-9vmw.600.0.0.2494585, VMware_bootbank_scsi-qla4xxx_5.01.03.2-7vmw.600
.0.0.2494585, VMware_bootbank_uhci-usb-uhci_1.0-3vmw.600.0.0.2494585

Afterward, you’ll need to reboot the host and you can then take it out of maintenance mode. If you have a large number of hosts, you probably want to investigate VUM, Auto Deploy, or another technology to automate upgrades. On the other hand, this works great for my home lab where I have two hosts and don’t want to maintain a VM just for VUM.

Lessons Learned from Migrating a DVS

Recently, I “fortunate” enough to experience some issues with a vCenter Server or VCS, the version of vCenter that you install on Windows. It had been upgraded 5.1->5.5->6.0->6.0U1 and apparently something broke along the way. I much prefer the VCSA, so this wasn’t the end of the world, but the issues we encountered forced our hand as to timing. Therefore, some of the migration steps were a bit ad hoc. This was also a small installation (<25 hosts) and a one-time event, so the question Is It Worth The Time? to automate the migration was judged to be a no. I still think that was the right evaluation, but we did find one thing that added some time to the effort: migrating a Distributed vSwitch (DVS).

The process we followed at first was:

  1. Export the DVS from the VCS
  2. Import the DVS to the VCSA with the Preserve box checked
  3. Disconnect/Remove From Inventory a host from the VCS
  4. Add A Host to the VCSA
  5. Attach the host to the DVS
  6. Repeat steps 3-5 for all hosts

Continue reading

NFS Export Settings for vSphere

Over the past few days, I’ve had to set up an NFS export for use with vSphere. I found myself frustrated with the vSphere docs because they only seem to cover the vSphere side of the house – when discussing the export that vSphere will use, the docs simply said, “Ask your vendor.” If you’re enabling NFS on your SAN and they have a one-click setup for you or a document with settings to enable, great, but if you’re setting up on a bare metal server or VM running Linux like me, you don’t have a vendor to ask.

Since I didn’t have a guide, I took a look at what happens when you enable NFS on a Synology. I don’t know if this is optimal, but this works for many people with the defaults. You can replicate this in Control Panel -> Shared Folders -> Highlight and Edit -> NFS Permissions. Add a new rule, add a hostname or IP entry, and hit OK. Here’s what the defaults look like:

NFS Exports fig 1

Let’s take a look at what happened in the busybox shell. SSH to your Synology as root with the admin password. Take a look at the permissions on the mount path (e.g. /volume1/rnelson0) and the contents of /etc/exports.

NFS Exports fig 2

(there is no carriage return at the end of the file, the ‘ds214>’ is part of the prompt not the exports contents)

A working mode for the directory is ‘0777’ and there’s a long string of nfs options. They are described in detail in the exports(5) man page. Here’s a high-level summary of each:

  • rw: Enable writes to the export (default is read-only).
  • async: Allow the NFS server process to accept additional writes before the current writes are written to disk. This is very much a preference and has potential for lost data.
  • no_wdelay: Do not delay writes if the server suspects (how? I don’t know) that another write is coming. This is a default with async so actually has no specific benefit here unless you remove async. This can have performance impacts, check whether wdelay is more appropriate.
  • no_root_squash: Do not map requests from uid/gid 0 (typically root) to the anonymous uid/gid.
  • insecure_locks: Do not require authentication of locking requests.
  • sec=sys: There are a number of modes, sys means no cryptographic security is used.
  • anonuid/anongid: The uid/gid for the anonymous user. On my Synology these are 1025/100 and match the guest account. Most Linux distros use 99/99 for the nobody account. vSphere will be writing as root so this likely has no actual effect.

I changed the netmask and anon(u|g)id values, as it’s most likely that a linux box with a nobody user would be the only non-vSphere client. Those should be the only values you need to change; async and no_wdelay are up to your preference.

If you use Puppet, you can automate the setup of your NFS server and exports. I use the echocat/nfs module from the forge (don’t forget the dependencies!). With the assumption that you already have a /nfs mount of sufficient size in place, the following manifest will create a share with the correct permissions and export flags for use with vSphere:

node server {
  include ::nfs::server

  file{ '/nfs/vsphere':
    ensure => directory,
    mode   => '0777',
  }
  ::nfs::server::export{ '/nfs/vsphere':
    ensure  => 'mounted',
    nfstag  => 'vsphere',
    clients => '10.0.0.0/24(rw,async,no_wdelay,no_root_squash,insecure_locks,sec=sys,anonuid=99,anongid=99)',
  }
}

To add your new datastore to vSphere, launch the vSphere Web Client. Go to the Datastores page, select the Datacenter you want to add the export to, and click on the Related Objects tab. Select Datastores and the first icon is to add a new datastore. Select NFS and the NFS version (new to vSphere 6), give your datastore a name, and provide the name of the export (/nfs/vsphere) and the hostname/IP of the server (nfs.example.com). Continue through the wizard, checking each host that will need the new datastore.

NFS Exports fig 3

Click Finish and you have a datastore that uses NFS! You may need to tweak the export flags for your environment, but this should be a good starting point. If you are aware of a better set of export flags, please let me know in the comments. Thanks!

vCenter Web Client and OVF deployment issues with Teredo

Tonight I was deploying an OVF on a newly upgraded vCenter version 6.0U1 through the vSphere Web Client. This is the Windows server (VCS), not the appliance (VCSA). I ran into this wonderful error while deploying an OVF on the storage selection step:

VCS Storage Issue Fig 1

There is KB 2053229 that says this was resolved in vCenter 5.1U2. However, the underlying issue is that the client attempts to communicate with the vCenter service in a new session to validate the storage and so the problem may remain in certain environments. I found that a teredo tunneling address was being used during storage selection rather than the IPv4 address that I connected to in my web browser. These addresses are provided for “free” with Windows Server 2008R2. The (magic?) design of teredo is that it should not be used when an IPv4 A record exists and should only be used for IPv6 URLs. I’m not sure why VCS would use this, just that I observed it being used. I discovered this with the help of two artifacts: 1) another node on a remote network was able to make it past storage selection; and 2) I saw a deny entry in syslog with my problematic management node to the VCS’s IPv4 addresses with IPv6 as the service:

VCS Storage Issue Fig 2

The KB article points to DNS, so I checked and found a AAAA record for vCenter. Delete that record, wait for it to replicate throughout Active Directory and expire, and reload the vSphere Web Client. You should no longer see a log failure for IPv6 and your OVF deploy should at least make it past the storage selection stage. I hope this helps!