Conference Gadget OpSec

I’m getting ready for PuppetConf shortly and that got me thinking about how to survive conferences with your gadgets operations security (opsec) intact. Here are a few things I’ve learned over the last few years, in no particular order:

  • Charge your devices every night. Check them in the morning to see they actually charged; if not, make sure they’re plugged in while you’re taking a shower and getting breakfast so they can survive the long day. Nothing like sitting down in the keynote and realizing your phone is at 20% and it hasn’t even started. Don’t forget to charge any battery packs you brought.
  • Reduce brightness settings on anything with a screen. Your lapaptop, tablet, phone, watch, etc. It should be very low, somewhere between “no-one else can read this” and “I can’t read this.” This serves two purposes:
    • Prevent others from reading your screen. The person behind you probably doesn’t need to read your email, and definitely not your KeePass/LastPass/etc. Nor do they need to be blinded by it during a presentation where the lights are dimmed.
    • Save battery life. You won’t miss as much of the conference and you save yourself from another risky event…
  • Bring your own charging cables/adapters and battery packs. Do not borrow them or use USB charging stations. (If you really must borrow a charge, make sure you trust that person with your digital life.) Most devices use a USB cable of some sort, and in case you haven’t heard, USB security is pretty horrible and opens you up to being rooted and data exfiltration (see BadUSB, Mactans, USB keystroke loggers and plenty of others). It’s just not worth it.
  • Determine if you want to bring your gadgets at all. This is especially true at security-oriented conferences. Hacks abound at these things, including hacking the cell service. If you must bring a device, it might be best to acquire something for use only at that conference and destroy it afterward. That seems harsh, but flashing the device may not remove some infections. Are you willing to risk it?
  • Use a VPN or at least prefer cell service over wifi. Make sure that any data you transmit is protected from malicious and inadvertent snooping. Most of us are not at security conferences where the cell service is hacked, so if you don’t have a VPN it’s probably pretty secure in comparison to wifi, but not always (know the atmostphere). Adding the VPN on top is the best, though. If your company doesn’t provide one, find a trustworthy service or set one up at home.
  • Ensure you have good password hygiene. At a minimum, make sure they’re of reasonable quality and aren’t shared between services. Jessy Irwin talks about this on a Digital Underground PodCast.
  • Don’t log into anything you don’t have to. For persistent-access services, like email or file sync, log in at home so you have a working token and do not need to enter the password again. For anything you need to authenticate to every time, it’s probably not a good idea. Every use of credentials potentially exposes them to onlookers. Pay your mortgage before you leave or after you get back, not from the hotel wifi.
  • Have a Two Factor Authentication (TFA) backup plan. TFA is much more secure than Two Step Authentication (TSA), but often has some limitations for certain use cases that you need to understand. TSA codes can usually be sent to a new device, whereas adding a new device to your TFA device list may require the existing TFA device. If the original is lost or hacked, you may have no way to recover your account, or it may take significant effort above your “worth my time” threshold. Understand what services would be affected and make sure you have another way to recover access. This might include disabling TFA for the duration – if so, ask yourself again if you really want to bring that gadget. This is best thought through before converting a service to TFA, but now is the time to double check.
  • Keep your devices with you, or in something more secure than the hotel safe. Those safes are often easily broken, as shown here and here. Especially at those security conferences. Definitely don’t leave your laptop unlocked and unattended at the bloggers table. Same thing with your charging and battery equipment.
  • If you don’t need a particular gadget, leave it at home. This is so important, I’m mentioning it twice. Earlier I talked about devices being hacked, but you also cannot lose something if it’s in the dresser at home. Maybe you need your phone, but the FitBit can stay.
  • Bring non-gadget backups. This is especially true for payments. If your phone is hacked, lost, or falls in the toilet, make sure you have at least one physical credit card with you.
  • Maintain a list of devices, services, payment methods you travel with. When something bad does happen, it’s really helpful to have a list of what’s affected. Keep a list at home in case you lose it all, as well as taking a (modified?) copy with you. The list should help you determine what you need to recover, but not have information that someone else could use to steal your identity. In other words, “Bank account check card, $phone” is fine, “Bank Of Bad Opsec, $phone, $card_number, $expiration, $ccv” is way too much. If something happens, start making phone calls. If the list was lost as well, that’s why you have a list at home. Make the calls now, do not wait till you get home and find $20k in charges to dispute or that your enter cloud drive was emptied.
  • Be paranoid. It may not come naturally to all of us, but it is key to good OpSec. If you think something might expose you unnecessarily, don’t do it. It is better to be safe than sorry.

I also have one non-opsec tip for conferences: always call your vendor reps and ask what they have going on at the conference. You can usually arrange some one on one time with their engineering team or attend their event where you can meet others using the same products and compare notes.

If you have your own tips, drop them in the comments or send them to me on twitter!

Getting started with vCheck

If you use vSphere and particularly vCenter, you’re probably at least familiar in passing with PowerCLI, a package of snap-ins and modules for PowerShell. This is my preferred language for interacting with the vSphere/vCenter APIs, since it has (IMO) the best documentation of the available languages and API SDKs. If not, I recommend downloading it and playing with it, it can really help you automate many of your repetitive tasks with less Flash and less right clicking.

One of the most popular tools built with PowerCLI is vCheck. It’s a framework for running a number of checks against your vSphere infrastructure and determining what operational issues are present – something every Ops team needs. It won’t replace a monitoring system such as vROps or even Nagios, but it augments such systems very well. For example, it can report on VMs that have ISOs attached, or where snapshots have been present for more than 7 days – issues that probably aren’t worth paging anyone out for, but need to be dealt with eventually. Many of us have built some homegrown solutions for this, maybe even with PowerCLI, but it is difficult to beat a tool designed to meet the needs of a large percentage of vSphere users, is actively developed by VMware employees, and is a framework that you can extend with instance-specific needs. You can always run your tools and vCheck together, too.

Let’s take a look at vCheck and how to get started with it today. We’ll download it, configure it, schedule it as a daily task, review how to enable and disable checks, and store your configuration in version control. This provides a solid base that you can tweak until it fits your specific needs just right.

Continue reading

vROps/Log Insight Integration and Troubleshooting

Update: Special thanks to Yogita N. Patil and VMware Technical Support for their assistance with the issues below!

Last week, I was trying to integrate vRealize Log Insight with vRealize Operations (vROps) so that I could ‘launch in context’ from vROps. This adds a context-sensitive action to vROps that lets you pull up Log Insight’s Interactive Analysis feature against the alert or object you are currently viewing. This makes it easy to drill down into logs with a lot less clicking around:

Launch in context is a feature in vRealize Operations Manager that lets you launch an external application via URL in a specific context. The context is defined by the active UI element and object selection. Launch in context lets the vRealize Log Insight adapter add menu items to a number of different views within the Custom user interface and the vSphere user interface of vRealize Operations Manager.

The documentation to enable this features seems pretty simple. I ran into a few problems, though…

The requirements are pretty simple, but were the first thing to trip me up. You want to be on Log Insight 3.6 and vROps 6.3. While Log Insight had been upgraded a day or two earlier, vROps was at 6.1. When performing the upgrade of vROps, it did not register its extension properly! Going into the Managed Object Browser showed there was still a vCOps 6.1 registration instance (yes, the extension is still called vCOps!). In addition, the extension was registered by IP, not by DNS. The extension needs to be in place for the steps below, or you receive even more opaque error messages, so I encourage you to verify it now. You can investigate your own MOB at a link similar to, and specifically look at the vROps extension at“com.vmware.vcops”%5D.client

Continue reading

Deploy your #Puppet Enterprise license key with Puppet

Since I manage my Puppet infrastructure with Puppet itself, I am for full automation. For Puppet Enterprise, that includes deploying the license key file from the puppet fileserver (profile/files/master/license.key served as puppet:///modules/profile/master/license.key). When upgrading to the latest Puppet Enterprise version, 2016.2.0, I encountered a change that was tricky to resolve – the puppet_enterprise::license class accepted a license_key parameter, which was marked as deprecated:

Warning: puppet_enterprise::license::license_key is deprecated and will be removed in the next
    PE version. Please use puppet_enterprise::license_key_path. If using the Node Manager, the class
    is located in the PE Infrastructure node group.

Easy, I’ll just use the parameter license_key_path instead! Except, it wants a location for a file on the master, and I’m trying to deploy a file to the master!

Continue reading

Upgrading to Puppet 4 at #PuppetConf 2016

As I did last year, I submitted a proposal for PuppetConf 2016 and it was accepted! As I did last year, I am requesting your help with it.

The talk,  Enjoying the Journey from Puppet 3.x to 4.x, will help attendees lay out a plan to get to Puppet 4. I will be sharing my experiences from POSS and PE upgrades,  including tools to assist with the migration and some pitfalls to avoid. There are many ways to perform these upgrades and my experiences are limited, so I’d like to hear about yours. If you are interested in sharing your experiences and grant me permission to share them in my talk, you can contact me on twitter/DM or by submitting a PR against my PuppetConf github repo. Let me know if you would like to be credited or keep it anonymous. Thanks!

Change your default linux shell without root access

A friend recently mentioned his hatred of ksh and inability to change his default shell because he is not root and root will not make the change. I concur, ksh is turrible, that needs fixed. Here’s a little trick for anyone else who has this issue. Let’s say you want to use bash and the shell forced upon you is ksh. The Korn shell uses .profile to set up your environment when you log in, so append these two lines to your profile:

# run bash damnit
[ ! "`which bash | egrep "^no"`" ] && [ -z $BASH ] && exec bash --login

We look for the output of which bash and ensure it does not say no bash in …, and if that file is non-zero, we exec bash as a login shell. You can read man exec for details on specifically how it works in a given shell, but it essentially replaces the running shell with the new command. If the first two checks are positive, then bash is executed and replaced the ksh process. Once the files are added to your profile, log in again in a new session. Voila, you now have a bash login shell!

rnelson0@dumbserver ~ $ ps -ef | grep bash
  rnelson0  5927  5298   0 19:56:42 pts/1       0:00 grep bash
  rnelson0  5298  5292   0 19:47:48 pts/1       0:00 bash --login

If you’re forced to use a different shell, you may need to locate a different profile file. If you want to use a different shell, just sub out bash for the name of your preferred shell. Enjoy!

Tag-based Veeam Backups

As I teased in Using vCenter tags with PowerCLI, I want to explore how to use tags  in a practical application. To refresh our memory, we looked at creating an Ownership category with individual tags in it, and limited VMs to having just one of the tags. We created a little script that defines our schema, in case we need to re-create it. We are going to work on a new category today for our backups. Specifically, Veeam backups, based on SDDC6282-SPO, Using vSphere tags for advanced policy-driven data protection as presented at VMworld 2015.

Defining Policy and Tags

To create our backup jobs, we need to know a few things that will translate into our tag schema. Our backup policies are defined by a combination of ownership, recovery point objective (RPO), and the retention period. For example, our Development group is okay with a 24 hour RPO and backups that are retained for a week. Operations may require a 4 or 8 hour RPO and require 30 days of backups. Each of those combinations will require a separate backup job. We can combine these tuples of information into individual tags so that Veeam can make use of them. We also need one more tag for VMs that do not need any backup at all. We can put all of this in a tag category called VeeamPolicy. Here’s what that might look like, again in PowerShell:

New-TagCategory -Name VeeamPolicy -Description "Veeam Backup Policy" -Cardinality Single -EntityType VirtualMachine
New-Tag -Name "NoRPO" -Category VeeamPolicy -Description "This VM has no backups"
New-Tag -Name "Development24h7d" -Category VeeamPolicy -Description "Development VMs with 24 hour RPO, 7 days retention"
New-Tag -Name "Operations8h30d" -Category VeeamPolicy -Description "Operations VM with 8 hour RPO, 30 day retention"
New-Tag -Name "Sales48h30d" -Category VeeamPolicy -Description "Sales VM with 48 hour RPO, 30 day retention"

This creates the category for the VeeamPolicy tags, a single tag for VMs that explicitly do NOT require backups, and three group/RPO/retention tuples. Go ahead and place this in revision control, so that over time, you can track the tags you are using. Tags are part of vCenter, so if you migrate to another or recreate your vCenter, you’ll need this! It also makes it easy to see what you’ve defined vs what’s actually in vCenter.

Next, we want to tag some VMs. There’s a number of ways this can be done, depending on how your VMs are organized. If we assume a simple VM and Templates folder structure where VMs are grouped by the owning organization (Development, Operations, Sales, Marketing, etc), we can tag all the VMs with NoRPO, then tag VMs in the relevant folders with the relevant tag. That would look like this:

# Default policy is NoPRO
Get-VM | % {
  New-TagAssignment -Entity $_ -Tag "NoRPO"

# Find Development VMs by Location
@(Get-VM -Location "Development") +
@(Get-Template -Location "Development") | % {
  Remove-TagAssignment -TagAssignment (Get-TagAssignment -Entity $_ -Category VeeamPolicy) -Confirm:$false
  New-TagAssignment -Entity $_ -Tag Development24h7d

It’s important to look for VMs and templates. The @() structure creates arrays, which can be added together. We then have to remove the previous assignment before placing the new one, since our category is Single cardinality (Unfortunately you cannot overwrite the current tag assignment, boo!).

Note: The Location parameter takes a string, and it finds ALL folders that match the string. If you have a VM and Templates folder named Operations and a Datastore folder called Operations, All the VMs in BOTH will be discovered. I recommend adding ” DS” to the end of datastore folders, ” VDS” or ” VSS” to the end of Network folders, etc., to ensure there’s no conflict here.

You could also look for VMs by datastore folders:

# Find Development VMs by Datastore Location
$DevelopmentDatastores = Get-Datastore -Location "Development Datastores"
@(Get-Template -Datastore $DevelopmentDatastores) +
@(Get-VM -Datastore $DevelopmentDatastores) | % {
  Remove-TagAssignment -TagAssignment (Get-TagAssignment -Entity $_ -Category VeeamPolicy) -Confirm:$false
  New-TagAssignment -Entity $_ -Tag Development24h7d

You may also have a more complicated layout for groups, such as Operations, where you do not want all the VMs underneath a top level folder to receive the same tag. We can instead combine some individual folders underneath and list some VMs in them that we want to be left as NoRPO by adding some selection criteria and skip the individual VMs:

# Find Operations VMs
$OperationsExemptVMs = 'veeamproxy1', 'veeamproxy2'
@(Get-VM -Location "Domain Controllers", "Management", "vSphere", "Templates for patching", "Veeam") +
@(Get-Template | ? { (Get-TagAssignment -Entity $_) -eq $null } ) | % {
  # Skip these VMs which will not be backed up
  if ( $OperationsExemptVMs -contains $_.Name.ToString() ) {

  Remove-TagAssignment -TagAssignment (Get-TagAssignment -Entity $_ -Category VeeamPolicy) -Confirm:$false
  New-TagAssignment -Entity $_ -Tag Operations24h30d

There’s a ton of other ways to select your VMs, this is just a sampling. If your organization could best be described as “Chaotic”, then you might have to assign Tags individually. However your tagging happens, when you’re done, you probably want a report so you can ensure everything was tagged properly. We can do that, too!

# Report on the current tag status
@(Get-VM) + @(Get-Template) | Get-TagAssignment -Category VeeamPolicy | Select Entity, Tag | Export-CSV -Path $env:USERPROFILE\Downloads\Tags.csv -NoTypeInformation

Open the resulting CSV file and we’ll have all our VMs and Templates and the Tag they were assigned. Print it out, red pen it, and tweak our categorizations. You can re-run the Find GROUP VMs sections, or if you want to start over, just delete the tag category and run the whole file again:

Remove-TagCategory VeeamPolicy

Tweak it till you get it right, and commit it to version control again. One last set of statements you may want to use is to find non-tagged VMs, either as part of your script (if you do not assign everything NoRPO) or on an intermittent basis to ensure every VM and Template is tagged:

@(Get-VM) + @(Get-Template) | ? { (Get-TagAssignment -Category "VeeamPolicy" -Entity $_) -eq $null }

Creating Veeam Backup Jobs

Now that we have our policy defined and tags applied, let’s create some jobs. We will be creating three jobs, one each for Development, Operations, and Sales. We will assume that automatic proxy selection will work and that there is only one repository, as those details aren’t important to the use of tags. This article was written using Veeam Backup & Replication v9, so if you’re reading this in the future, things may look different!

One last thing to note, and you may have figured this out already, is that when using Tags for our selection process, it’s an OR operation, not an AND operation. This means that if you select two tags, Veeam will find all VMs that match either tag, rather than VMs that much all the tags. That’s why we have three pieces of information in our tag! There’s a request to support logical AND – please comment in that thread if you think you’d find this interesting as well! If this is delivered in the future, it could really change how this is done!

Create a new backup job for Development:

Veeam Tag Based Backups fig 1

On the next page, click Add, then click on the right-most icon (Tags) and expand and select the Development24h7d tag and hit Add then Next.

Veeam Tag Based Backups fig 2

On the storage page, change the Retention Policy 7 days. This is also where we can hit Advanced and modify Quiesce settings, Notifications, and most other backup settings. Click Next when done.

Veeam Tag Based Backups fig 3

Make any selections we need to on the Guest Processing screen (not shown) and click Next. Now set up our schedule (not shown). Since this is a 24h RPO, this should run at least once a day (more is okay, if we’d like). Click Create to finish the job setup. We should end up with something like this:

Veeam Tag Based Backups fig 4

Now, every day at 10:00:00PM, the Development job runs. Veeam will contact the vCenter server and enumerate every VM with the Development24h7d tag, add them to the job, and then start backing them up. If the Development team deletes a VM, it’s automatically no longer part of the job. If they add a VM, and they tag it properly, it’s automatically part of the job. All we need to do is teach or coworkers how to tag their new VMs, hopefully with the assistance of automation, and use our PowerShell command from above to run a report every days to find any untagged VMs. It’s okay to tag a VM with NoRPO, because it’s explicit acknowledgement that a VM does not require backups, but the lack of a tag does not mean that backups are not needed.

We can also use combine the use of tags with exclusions. For example, if we have some “close” VMs and some “remote” VMs on a slower link, they probably do not use the same repository at HQ. In the “close” job, we exclude the remote site’s clusters or datastores, and on the “remote” job, we exclude HQ’s clusters or datastores. If we have two or three sites, this is probably pretty easy. If we have multiple sites, we may want to revisit our tag scheme and add a fourth item to the tags – Location – and re-align our jobs. Of course, if the “AND” selection criteria is added in the future, this will change, as individual tags won’t need to carry EVERY piece of information.


Today, we created a tag schema oriented around our backup policies. We also built some reports showing how VMs are tagged and which VMs are not tagged. This script was commited to version control for accurate tracking over time and re-creation of the schema if needed. We then created some simple but dynamic Veeam backup jobs based on the tags, and showed our coworkers how to leverage the tags. This is a great start for simplifying your Veeam backup jobs and to see the practical power of vCenter Tags. There’s a lot more you can do with Tags, and with your Veeam jobs. I’d love to hear how you’re using Tags in the real world in the comments or on twitter. Thanks!