Making the Puppet vRealize Automation plugin work with vRealize Orchestrator

I’m pretty excited about this post! I’ve been building up Puppet for vSphere Admins for a few years now but the final integration aspects between Puppet and vSphere/vCenter were always a little clunky and difficult to maintain without specific dedication to those integration components. Thanks to Puppet and VMware, that’s changed now.

Puppet announced version 2.0 of their Puppet Plugin for vRealize Automation this week. There’s a nice video attached, but there’s one problem – it’s centered on vRealize Automation (vRA) and I am working with vRealize Orchestrator (vRO)! vRO is included with all licenses of vCenter, whereas vRA is a separate product that costs extra, and even though vRA requires a vRO engine to perform a lot of its work, it abstracts a lot of the configuration and implementation details away that vRO-only users need to care about. This means that much of the vRA documentation and guides you find, for the Puppet plugin or otherwise, are always missing some of the important details needed to implement the same functionality – and sometimes won’t work at all if it relies on vRA functionality not available to us.

Don’t worry, though, the Puppet plugin DOES work with vRO! We’ll look at a few workflows to install, run, and remove puppet from nodes and then discuss how we can use them within larger customized workflows. You must already have an installed vRealize Orchestrator 7.x instance configured to talk to your vCenter instance. I’m using vRO 7.0.0 with vCenter 6.0. If you’re using a newer version, some of the dialogs I show may look a little different. If you’re still on vRO 6.x, the configuration will look a LOT different (you may have to do some research to find the equivalent functionality) but the workflow examples should be accurate.

Puppet provides a User Guide for use with a reference implementation. I’ll be mostly repeating Part 2 when installing and configuring, but reality tends to diverge from reference so we’ll explore some non-reference details as well.

Continue reading

Getting started with vCheck

If you use vSphere and particularly vCenter, you’re probably at least familiar in passing with PowerCLI, a package of snap-ins and modules for PowerShell. This is my preferred language for interacting with the vSphere/vCenter APIs, since it has (IMO) the best documentation of the available languages and API SDKs. If not, I recommend downloading it and playing with it, it can really help you automate many of your repetitive tasks with less Flash and less right clicking.

One of the most popular tools built with PowerCLI is vCheck. It’s a framework for running a number of checks against your vSphere infrastructure and determining what operational issues are present – something every Ops team needs. It won’t replace a monitoring system such as vROps or even Nagios, but it augments such systems very well. For example, it can report on VMs that have ISOs attached, or where snapshots have been present for more than 7 days – issues that probably aren’t worth paging anyone out for, but need to be dealt with eventually. Many of us have built some homegrown solutions for this, maybe even with PowerCLI, but it is difficult to beat a tool designed to meet the needs of a large percentage of vSphere users, is actively developed by VMware employees, and is a framework that you can extend with instance-specific needs. You can always run your tools and vCheck together, too.

Let’s take a look at vCheck and how to get started with it today. We’ll download it, configure it, schedule it as a daily task, review how to enable and disable checks, and store your configuration in version control. This provides a solid base that you can tweak until it fits your specific needs just right.

Continue reading

vROps/Log Insight Integration and Troubleshooting

Update: Special thanks to Yogita N. Patil and VMware Technical Support for their assistance with the issues below!

Last week, I was trying to integrate vRealize Log Insight with vRealize Operations (vROps) so that I could ‘launch in context’ from vROps. This adds a context-sensitive action to vROps that lets you pull up Log Insight’s Interactive Analysis feature against the alert or object you are currently viewing. This makes it easy to drill down into logs with a lot less clicking around:

Launch in context is a feature in vRealize Operations Manager that lets you launch an external application via URL in a specific context. The context is defined by the active UI element and object selection. Launch in context lets the vRealize Log Insight adapter add menu items to a number of different views within the Custom user interface and the vSphere user interface of vRealize Operations Manager.

The documentation to enable this features seems pretty simple. I ran into a few problems, though…

The requirements are pretty simple, but were the first thing to trip me up. You want to be on Log Insight 3.6 and vROps 6.3. While Log Insight had been upgraded a day or two earlier, vROps was at 6.1. When performing the upgrade of vROps, it did not register its extension properly! Going into the Managed Object Browser showed there was still a vCOps 6.1 registration instance (yes, the extension is still called vCOps!). In addition, the extension was registered by IP, not by DNS. The extension needs to be in place for the steps below, or you receive even more opaque error messages, so I encourage you to verify it now. You can investigate your own MOB at a link similar to, and specifically look at the vROps extension at“com.vmware.vcops”%5D.client

Continue reading

Tag-based Veeam Backups

As I teased in Using vCenter tags with PowerCLI, I want to explore how to use tags  in a practical application. To refresh our memory, we looked at creating an Ownership category with individual tags in it, and limited VMs to having just one of the tags. We created a little script that defines our schema, in case we need to re-create it. We are going to work on a new category today for our backups. Specifically, Veeam backups, based on SDDC6282-SPO, Using vSphere tags for advanced policy-driven data protection as presented at VMworld 2015.

Defining Policy and Tags

To create our backup jobs, we need to know a few things that will translate into our tag schema. Our backup policies are defined by a combination of ownership, recovery point objective (RPO), and the retention period. For example, our Development group is okay with a 24 hour RPO and backups that are retained for a week. Operations may require a 4 or 8 hour RPO and require 30 days of backups. Each of those combinations will require a separate backup job. We can combine these tuples of information into individual tags so that Veeam can make use of them. We also need one more tag for VMs that do not need any backup at all. We can put all of this in a tag category called VeeamPolicy. Here’s what that might look like, again in PowerShell:

New-TagCategory -Name VeeamPolicy -Description "Veeam Backup Policy" -Cardinality Single -EntityType VirtualMachine
New-Tag -Name "NoRPO" -Category VeeamPolicy -Description "This VM has no backups"
New-Tag -Name "Development24h7d" -Category VeeamPolicy -Description "Development VMs with 24 hour RPO, 7 days retention"
New-Tag -Name "Operations8h30d" -Category VeeamPolicy -Description "Operations VM with 8 hour RPO, 30 day retention"
New-Tag -Name "Sales48h30d" -Category VeeamPolicy -Description "Sales VM with 48 hour RPO, 30 day retention"

This creates the category for the VeeamPolicy tags, a single tag for VMs that explicitly do NOT require backups, and three group/RPO/retention tuples. Go ahead and place this in revision control, so that over time, you can track the tags you are using. Tags are part of vCenter, so if you migrate to another or recreate your vCenter, you’ll need this! It also makes it easy to see what you’ve defined vs what’s actually in vCenter.

Next, we want to tag some VMs. There’s a number of ways this can be done, depending on how your VMs are organized. If we assume a simple VM and Templates folder structure where VMs are grouped by the owning organization (Development, Operations, Sales, Marketing, etc), we can tag all the VMs with NoRPO, then tag VMs in the relevant folders with the relevant tag. That would look like this:

# Default policy is NoPRO
Get-VM | % {
  New-TagAssignment -Entity $_ -Tag "NoRPO"

# Find Development VMs by Location
@(Get-VM -Location "Development") +
@(Get-Template -Location "Development") | % {
  Remove-TagAssignment -TagAssignment (Get-TagAssignment -Entity $_ -Category VeeamPolicy) -Confirm:$false
  New-TagAssignment -Entity $_ -Tag Development24h7d

It’s important to look for VMs and templates. The @() structure creates arrays, which can be added together. We then have to remove the previous assignment before placing the new one, since our category is Single cardinality (Unfortunately you cannot overwrite the current tag assignment, boo!).

Note: The Location parameter takes a string, and it finds ALL folders that match the string. If you have a VM and Templates folder named Operations and a Datastore folder called Operations, All the VMs in BOTH will be discovered. I recommend adding ” DS” to the end of datastore folders, ” VDS” or ” VSS” to the end of Network folders, etc., to ensure there’s no conflict here.

You could also look for VMs by datastore folders:

# Find Development VMs by Datastore Location
$DevelopmentDatastores = Get-Datastore -Location "Development Datastores"
@(Get-Template -Datastore $DevelopmentDatastores) +
@(Get-VM -Datastore $DevelopmentDatastores) | % {
  Remove-TagAssignment -TagAssignment (Get-TagAssignment -Entity $_ -Category VeeamPolicy) -Confirm:$false
  New-TagAssignment -Entity $_ -Tag Development24h7d

You may also have a more complicated layout for groups, such as Operations, where you do not want all the VMs underneath a top level folder to receive the same tag. We can instead combine some individual folders underneath and list some VMs in them that we want to be left as NoRPO by adding some selection criteria and skip the individual VMs:

# Find Operations VMs
$OperationsExemptVMs = 'veeamproxy1', 'veeamproxy2'
@(Get-VM -Location "Domain Controllers", "Management", "vSphere", "Templates for patching", "Veeam") +
@(Get-Template | ? { (Get-TagAssignment -Entity $_) -eq $null } ) | % {
  # Skip these VMs which will not be backed up
  if ( $OperationsExemptVMs -contains $_.Name.ToString() ) {

  Remove-TagAssignment -TagAssignment (Get-TagAssignment -Entity $_ -Category VeeamPolicy) -Confirm:$false
  New-TagAssignment -Entity $_ -Tag Operations24h30d

There’s a ton of other ways to select your VMs, this is just a sampling. If your organization could best be described as “Chaotic”, then you might have to assign Tags individually. However your tagging happens, when you’re done, you probably want a report so you can ensure everything was tagged properly. We can do that, too!

# Report on the current tag status
@(Get-VM) + @(Get-Template) | Get-TagAssignment -Category VeeamPolicy | Select Entity, Tag | Export-CSV -Path $env:USERPROFILE\Downloads\Tags.csv -NoTypeInformation

Open the resulting CSV file and we’ll have all our VMs and Templates and the Tag they were assigned. Print it out, red pen it, and tweak our categorizations. You can re-run the Find GROUP VMs sections, or if you want to start over, just delete the tag category and run the whole file again:

Remove-TagCategory VeeamPolicy

Tweak it till you get it right, and commit it to version control again. One last set of statements you may want to use is to find non-tagged VMs, either as part of your script (if you do not assign everything NoRPO) or on an intermittent basis to ensure every VM and Template is tagged:

@(Get-VM) + @(Get-Template) | ? { (Get-TagAssignment -Category "VeeamPolicy" -Entity $_) -eq $null }

Creating Veeam Backup Jobs

Now that we have our policy defined and tags applied, let’s create some jobs. We will be creating three jobs, one each for Development, Operations, and Sales. We will assume that automatic proxy selection will work and that there is only one repository, as those details aren’t important to the use of tags. This article was written using Veeam Backup & Replication v9, so if you’re reading this in the future, things may look different!

One last thing to note, and you may have figured this out already, is that when using Tags for our selection process, it’s an OR operation, not an AND operation. This means that if you select two tags, Veeam will find all VMs that match either tag, rather than VMs that much all the tags. That’s why we have three pieces of information in our tag! There’s a request to support logical AND – please comment in that thread if you think you’d find this interesting as well! If this is delivered in the future, it could really change how this is done!

Create a new backup job for Development:

Veeam Tag Based Backups fig 1

On the next page, click Add, then click on the right-most icon (Tags) and expand and select the Development24h7d tag and hit Add then Next.

Veeam Tag Based Backups fig 2

On the storage page, change the Retention Policy 7 days. This is also where we can hit Advanced and modify Quiesce settings, Notifications, and most other backup settings. Click Next when done.

Veeam Tag Based Backups fig 3

Make any selections we need to on the Guest Processing screen (not shown) and click Next. Now set up our schedule (not shown). Since this is a 24h RPO, this should run at least once a day (more is okay, if we’d like). Click Create to finish the job setup. We should end up with something like this:

Veeam Tag Based Backups fig 4

Now, every day at 10:00:00PM, the Development job runs. Veeam will contact the vCenter server and enumerate every VM with the Development24h7d tag, add them to the job, and then start backing them up. If the Development team deletes a VM, it’s automatically no longer part of the job. If they add a VM, and they tag it properly, it’s automatically part of the job. All we need to do is teach or coworkers how to tag their new VMs, hopefully with the assistance of automation, and use our PowerShell command from above to run a report every days to find any untagged VMs. It’s okay to tag a VM with NoRPO, because it’s explicit acknowledgement that a VM does not require backups, but the lack of a tag does not mean that backups are not needed.

We can also use combine the use of tags with exclusions. For example, if we have some “close” VMs and some “remote” VMs on a slower link, they probably do not use the same repository at HQ. In the “close” job, we exclude the remote site’s clusters or datastores, and on the “remote” job, we exclude HQ’s clusters or datastores. If we have two or three sites, this is probably pretty easy. If we have multiple sites, we may want to revisit our tag scheme and add a fourth item to the tags – Location – and re-align our jobs. Of course, if the “AND” selection criteria is added in the future, this will change, as individual tags won’t need to carry EVERY piece of information.


Today, we created a tag schema oriented around our backup policies. We also built some reports showing how VMs are tagged and which VMs are not tagged. This script was commited to version control for accurate tracking over time and re-creation of the schema if needed. We then created some simple but dynamic Veeam backup jobs based on the tags, and showed our coworkers how to leverage the tags. This is a great start for simplifying your Veeam backup jobs and to see the practical power of vCenter Tags. There’s a lot more you can do with Tags, and with your Veeam jobs. I’d love to hear how you’re using Tags in the real world in the comments or on twitter. Thanks!

Using vCenter tags with PowerCLI

I was recently introduced to some practical usage of vCenter tags. I used tags to build a fairly easy and dynamic set of sources for Veeam backup jobs, but before I get into that, I want to explain tags a little bit for those who are not familiar with them.

Tags are something that are pretty easy to understand conceptually. You create a category which contains a few tags. An object can receive one or multiple tags from a given category. Tags are managed by vCenter, not by vSphere, so require a license that provides vCenter for management. That’s a pretty simple explanation and list of requirements.

A Tag Category has a collection of Tags available within it. If the Category is “single cardinality”, it means that an object can only be assigned one tag in the category. This might be good for a category associated with ownership or recovery point objective (RPO). A Category can also be “multiple cardinality”, where a single object can be assigned multiple tags in a category. This would be helpful to associate applications with a VM, or a list of support teams that need notified if there is a planned impact or outage.

I’m going to show you how to manage Tags and Tag Categories with PowerCLI (Specifically with the PowerShell ISE and a profile to load PowerCLI), but you can manually create and manage them through the vSphere Web Client, under the Tags item on the left hand menu. You can create and delete and rename tags and categories there all day long, and you can right click on an object almost anywhere else and select Edit Tags to edit the assigned tags on it. When you view most objects, you’ll see an area in the Summary tab labeled Tags that will display and let you manage assignments.

vCenter Tags fig 1vCenter Tags fig 2

Let’s look at how to automate this with PowerCLI. Don’t forget to Connect-VIServer <vCenter name> before continuing. Use the cmdlets *-TagCategory, *-Tag, and *-TagAssignment to manage Categories, Tags, and the assignment of tags to objects, respectively. We start by creating a category using New-TagCategory. This accepts the parameters Name*, Cardinality, Description, EntityType, and Server. A new tag is created with New-Tag, as you’d expect, with parameters Category*, Name*, Description, and Server. Asterisks denote required parameters. Both accept the optional Confirm and WhatIf arguments, the latter of which is extremely helpful while you plan out your tag schemes.

As an example, let’s create an Ownership tag category. This will be a single cardinality for most, as most objects will be owned by a single group, but if some objects share ownership, be sure to create it as a multiple-cardinality category. We can restrict the type of entities it applies to with a comma separated list of any of the valid types: Cluster, Datacenter, Datastore, DatastoreCluster, DistributedPortGroup, DistributedSwitch, Folder, ResourcePool, VApp, VirtualPortGroup, VirtualMachine, VM, VMHost (you can see this list any time by reading the New-TagCategory help pages). Many can probably restrict this to VMs and maybe Datastores, as everything else is likely owned by the same single team. WhatIf is helpful to make sure we have the syntax we want before we apply it. We can put these values into the Command add-on and click Insert to see what the PowerCLI looks like:

vCenter Tags fig 3

C:\> New-TagCategory -Name Ownership -Cardinality Single -Description "Group ownership for VMs" -EntityType VirtualMachine -WhatIf

Select New-Tag in the command add-on and add a tag for a group called Development in the new category, then click Insert:

vCenter Tags fig 4

C:\> New-Tag -Category Ownership -Name Development -Description "We put the Dev in DevOps" -WhatIf

You can now create a script with a few more groups, copy and pasting the above line as a start, and plan out your tag scheme:

vCenter Tags fig 5

Run the commands (F5 in ISE) and make sure there are no errors, but unfortunately there will be:

C:\> New-TagCategory -Name Ownership -Cardinality Single -Description "Group ownership for VMs" -EntityType VirtualMachine -WhatIf
New-Tag -Category Ownership -Name Development -Description "We put the Dev in DevOps" -WhatIf
New-Tag -Category Ownership -Name Operations -Description "We put the Ops in DevOps" -WhatIf
New-Tag -Category Ownership -Name Sales -Description "We sell what doesn't exist" -WhatIf
New-Tag -Category Ownership -Name Financial -Description "We make sure your paycheck shows up" -WhatIf

What if: Create tag category 'Ownership' on server 'vcsa'
New-Tag : 6/27/2016 4:16:02 PM    New-Tag        Could not find TagCategory with name 'Ownership'.    
At line:2 char:1
+ New-Tag -Category Ownership -Name Development -Description "We put the Dev in De ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Unfortunately, each command in WhatIf mode is run without knowledge of the previous or following statements, so the first New-Tag statement is unaware that a new tag category would have been created. Remove the WhatIf from the first line and run it again:

C:\> New-TagCategory -Name Ownership -Cardinality Single -Description "Group ownership for VMs" -EntityType VirtualMachine
New-Tag -Category Ownership -Name Development -Description "We put the Dev in DevOps" -WhatIf
New-Tag -Category Ownership -Name Operations -Description "We put the Ops in DevOps" -WhatIf
New-Tag -Category Ownership -Name Sales -Description "We sell what doesn't exist" -WhatIf
New-Tag -Category Ownership -Name Financial -Description "We make sure your paycheck shows up" -WhatIf

Name                                     Cardinality Description                                                        
----                                     ----------- -----------                                                        
Ownership                                Single      Group ownership for VMs                                            
What if: Create tag 'Development' in category 'Ownership'
What if: Create tag 'Operations' in category 'Ownership'
What if: Create tag 'Sales' in category 'Ownership'
What if: Create tag 'Financial' in category 'Ownership'

You can delete or edit the Tags and Categories using Remove-TagCategory, Remove-Tag, SetTagCategory, and Set-Tag. These cmdlets use mostly the same parameters; use the Command Add-On or Get-Help for more details. You can inspect the Tags and Categories that exist with Get-Tag and Get-TagCategory.

C:\> Set-TagCategory -Category Ownership -Description "Organizational ownership for VMs"

Name                                     Cardinality Description
----                                     ----------- -----------
Ownership                                Single      Organizational ownership for VMs

C:\> Get-TagCategory

Name                                     Cardinality Description
----                                     ----------- -----------
Ownership                                Single      Organizational ownership for VMs

I recommend saving your Tag scripts in version control, so you can recreate or adjust them as needed. Next time, we’ll look at how to assign tags to objects in order to solve a specific problem: backups!

Announcement: Github repo for common vCenter roles

Last week, I was installing some of the vRealize Suite components and was creating accounts for each component using the Principle of Least Privilege. I was sometimes able to find vendor documentation on the required permissions, sometimes I found a few blog posts where people guessed at the required permissions, but in almost no cases was I able to find automated role creation. Perhaps my google-fu is poor! Regardless, I thought it would be nice to have documented and automated role creation in a single place.

To that end, I created a repo on GitHub called vCenter-roles. I use PowerCLI to create a role with the correct permissions, and only the correct permissions. Each cmdlet will allow you to specify the role name or it will use a default. For instance, to create a role for Log Insight, just run the attached ps1 script followed by the command:


It’s that easy!

I will be adding some other vRealize Suite roles as I work my way through the installation, but there are tons of other common applications out there that require their own role, and not just VMware’s own applications! I encourage you to open an issue or submit a Pull Request (PR) for any applications you use. The more roles we can collect in one place, the more helpful it is for the greater community. Thanks!

Deploying Windows Images with KMS keys

I’m not all that familiar with Windows licensing models, so I stumbled into a bit of surprise with KMS keys recently. If you are using a central KMS server that you do not maintain, and someone gives you a KMS key, you can ignore it! That’s for the KMS Host, which is where the licensing happens. Your nodes will be KMS Clients and they will use a Generic Volume License Key for activation. The Client communicates with the Host, which tells the client if it is activated and provides all the necessary information for that to happen (I don’t know how the Host does that, that’s the beauty of letting someone else run that service!). In this case, you are often given media to use for the Windows install that includes the GVLK, so you don’t need to do anything but communicate with the KMS Host. It’s a pretty nice setup, all considering.

However, IF you do something silly like put the KMS Host key on your Clients, you won’t get far. The Host key can only be activated 10 times on 6 hosts, so very soon you’ll run into trouble, if not immediately. You have to switch back over to the GVLK and activate using that. Microsoft maintains a list of GVLKs for each edition of Windows. The lookup of the KMS Host is done by DNS, but you can manually configure the KMS Client as well. Once the GVLK is in place, activate the key. Here are the three commands you will need, using Windows 2012R2 Datacenter as the GVLK:

cscript c:\windows\system32\slmgr.vbs /ipk W3GGN-FT8W3-Y4M27-J84CP-Q3VJ9
cscript c:\windows\system32\slmgr.vbs /skms
cscript c:\windows\system32\slmgr.vbs /ato

These commands need to be run from an administrator-privilege command prompt or PowerShell session.

If you are using templates, run the first command on the template. Ensure the deployment process is not adding license information. In vCenter, this means removing all options from the License Information portion of the Customization Specifications in ALL customization specs. Add the /skms and /ato commands to the existing commands in the Run Once section:

KMS Fig 1

KMS Fig 2

When you deploy a VM, it should now automatically activate itself! If you run into issues, ensure that the Client can communicate with the Host and no firewalls are blocking the communication. I’ve found that a global any/<KMS Server>/<kms port> rule in your firewalls is handy to ensure that random networks aren’t blocked from activation.

If you’re interested in learning more about Windows Licensing, Microsoft has a great amount of documentation. I suggest starting with Learn About Product Activation and then moving through the relevant sections.