A Full Stack What?

You’ve probably heard a lot of talk about the term “Full Stack Engineer”. You may even hear that everyone’s looking for one, so you probably want to be one to help your career. A Full Stack Engineer (hereafter FSE) is someone who doesn’t just know their one area deeply, but knows a bit about the rest of the stack. That depth of knowledge varies from very shallow to deep expertise, with the idea that the FSE knows how the different levels of the stack work together so they can make decisions that benefit the entire stack, rather than a local optimization that may harm the rest of the stack. You don’t want someone making an application decision that blows up the storage stack, or vice versa, so this kind of wide knowledge rather than deep knowledge is definitely helpful.

There’s a huge challenge to becoming an FSE, one of which is the sheer amount of layers in the stack to learn about these days. There’s so much to learn that it’s not actually feasible that any one person can learn all those layers deeply enough to really know the full stack. There’s absolutely nothing wrong with taking the journey toward Full Stack Engineer, but I think there’s another worthwhile goal out there:

A quipped about a “Full Stack Human,” a little bit of a tongue-in-cheek response to the overuse of the FSE term, but there’s some seriousness behind it. What it really means is that you should try and be a well-rounded person. In a sentence: Be more than a job.

A job is (hopefully) only 40 hours out of each 144 hour week and 2000 hours out of 8766 hours a year, less than 25% of what you do in a year. Sleeping should take up about 30% more – and it really should, we have to work very hard to not have a perpetual sleep deficit. Many of us will spend some of that remaining time trying to advance our work and careers, which is perfectly fine. This still leaves a lot of time, time in which we can find some hobbies and activities to enjoy so we’re more than just a working machine.

For exercise, I really like playing flag football. There’s a very diverse assortment of players out there and it’s far more entertaining than a treadmill or machine. When I feel creative, I enjoy woodworking. It requires deliberation, planning, and care in ways that my day job doesn’t – well, since I like having all of my fingers, anyway. I really like my sci-fi and fantasy novels, but I also make sure I fit some classics like War & Peace in between them. My wife and I don’t do anything truly adventurous, but we have been fortunate to visit a number of countries and enjoy their different cultures.

These activities gives you depth and adds dimensions to your character. (I realize I’m starting to sound like your parents did when you were filling out college applications, but bear with me a bit longer!) You meet other people and cultures and gain new viewpoints in which to perceive life. For example, in a decade of flag football, I’ve learned so many different ways to inspire teammates – and which ones don’t work! – and how to calm people down so they don’t lose the game.

I’d never get those experiences just by focusing on working my way up the stack at work, and those experiences help me out just as much at work. We talk a lot about encouraging diversity in tech, and in my opinion, it has to start in your personal life. A well-rounded person, a Full Stack Human, has those diverse experiences and can bring that diversity back into tech.

Your hobbies also give you a healthy escape from work. You aren’t just the project you released last week, and you shouldn’t kill yourself over work (figuratively or literally!). Identification and burnout can be a significant problem for everyone. If you don’t think so, you either aren’t there yet, or you’re there and you don’t know it! When you get too wrapped up in work – the deadlines are pressing down on you, politics got heated, you missed a family event because you were working late and didn’t even realize it – you need a safety valve to relieve that pressure and your personal time should help with that. PSA: If you’re struggling with burnout, please reach out to someone. We’re here to help!

Be a Full Stack Human. I guarantee it will be rewarding on its own, and it’s a huge step up on becoming a Full Stack Engineer!

2015 Recap: How did I do?

Just like I did at this time last year, it’s time to take a look at my goals for the previous year and see how I did.

Learn Ruby

I’d like to think I grok ruby at a more advanced level, now. I’ve written my first gem (and documented the ordeal) and contributed a number of patches to Ruby-ish projects here and there, mostly based around Puppet. I’ve also started writing “throw away” code in Ruby when possible, furthering my transition away from a bash-everywhere mentality. Grade: Pass

Blog more about Security

I started incorporating more security elements into my writing, but I haven’t really done a lot of security-focused writing. I only added one item to the Security category in 2015. I’m sitting on a bunch of drafts about security but am too timid to finish and publish them. Grade: Fail

Home Network

  1. I got my new home network up in running in the late spring, thanks to my partner-in-crime Mike SoRelle. I wrote an article about it as well.
  2. I made some progress here but not in the anticipated direction. I have all of my home network running Linux in Puppet and am working toward the same on the few Windows boxes. There was a lot of turmoil on the VMware side of things (5.5 updates, 6.0, 6.0 updates, changes to VCSA) and it slowed the work there. No IPAM, but I’m not feeling the burn very much because DNS at least is in Puppet.

Grade: Pass. But barely.

Expand PuppetInABox

I’ve learned a lot about software development in the past year. I’ve not only expanded and revamped PuppetInABox (support for Puppet 4 coming soon!), but I’m maintaining a few puppet modules, a puppet-related gem, and am actively participating in VoxPupuli (previously Puppet Community) and contributing features and fixes to Puppet itself. I think I’m making progress here, but still have a ways to go. Grade: Pass.

Propose a PuppetConf Talk

This was originally a goal to propose a VMworld talk, but I changed that as I didn’t have good subject matter for it before the CFP ended. I did submit a CFP to PuppetConf and was accepted! I presented in October and you can catch the video and slides online. I enjoyed the hell out of the conference and I dare say my talk did well, too! Grade: Pass.


I have made zero progress here. It was a busy year! I have until April to get this or renew the VCP and I’m not sure which it will be. Grade: Fail

Read War & Peace

This wasn’t on the list, but it was a personal goal. I’ve read a lot of Barnes and Nobles Classics and I love the Russian literature (Crime and Punishment in particular!), but at ~1100 pages of translated mid-1800 Russian, War & Peace was intimidating. I started this in the latter part of the year and I underestimated the time required to chew on it. I’m around 450 pages in after a few months. It’s been slow but very rewarding. You’ll be happy to know that in 1810, the Russians had meetings about having a meeting. What’s old is new again! Grade: C

I’d say I had a successful year. I didn’t hit all my goals, but like New Year’s Resolutions, I knew some goals would change and others wouldn’t be as important, it was more a guide for the year. I will be posting some new goals for 2016 shortly, though, so I can stay grounded this year as well!

Minimum Viable Configuration (MVC)

In my PuppetConf talk, I discussed a concept I call “Minimum Viable Configuration”, or MVC. This concept is similar to that of the Minimum Viable Product (MVP), in which you develop and deploy just the core features required to determine if there’s a market fit for your anticipated customer base. The MVC, however, is targeted at your developers, and is the minimum amount of customization required for the developers to be productive with the languages and tools your organization uses. This can include everything from having preferred IDEs available, language plugins, build tools, etc.

A Minimum Viable Configuration may not appear necessary to many, especially those who have been customizing their own environment for years or decades. The MVC is really targeted at your team, or as the organization as a whole. You may have a great customized IDE setup for writing Puppet or Powershell code, but others on your team may just be starting. The MVC allows the organization to share that accumulated wealth, making full use of the tens or hundreds of years of experience on the team. A novice developer can sit down and be productive with any language or tool covered by the MVC by standing on the shoulders of their teammates.

The MVC truly is the minimum customization required to get started – for instance, a .vimrc file that sets the tabstop to 2 characters and provides enhanced color coding and syntax checking for various languages – but that still allows users to add their own customizations. If you enforce the minimum, but don’t limit further customization, new hires can not only check their email on day one, but can actually delve through the codebase and start making changes on day one. You can also tie it into any vagrant images you might maintain.

Your MVC will change over time, of course. Use your configuration management tool, like Puppet, to manage the MVC. When the baseline is updated, all the laptops and shared nodes can be updated quickly to the new standard. You can see an example of a Minimum Viable Configuration for Linux in PuppetInABox’s role::build and the related profiles (build, rcfiles::vim, rcfiles::bash). You can easily develop similar roles and profiles for other languages or operating systems.

I feel the MVC can be a very powerful tool for teams who work with an evolving variety of tools and languages, who hire novices and grow expertise internally, and especially organizations that are exposing Operations teams to development strategies (i.e. DevOps). What do you think about the MVC? Are you using something similar now, or is there another way to address the issue?

Ravello and SimSpace: Security in the cloud

Ravello and SimSpace’s On-Demand Cyber Ranges

Last year, many of us were introduced to Ravello Systems and their nested virtualization product. Their hypervisor, HVX, and their network and storage overlay technologies allow you to run any VM from your enterprise on a cloud – specifically Amazon AWS and Google Compute Engine. You can sign up for a free trial and migrate your VMs into the cloud instantly.

Many in the #vExpert community have used Ravello to augment or replace their home lab. We’ve also seen some pretty interesting uses of Ravello over the last year – AutoLab in the cloud, Ravello/vCloud Air DR setups and numerous blueprints (pre-defined multi-node system designs) such as Puppet and Openstack on AWS.

Yesterday, I had the pleasure of speaking with SimSpace Corporation, a security company focused on cyber assessments, training, and testing. SimSpace has a history of working with and testing next generation cyber-security tools and helping their clients to rapidly build network models, called Cyber Ranges, using these tools at scale. Today, SimSpace and Ravello announced a partnership to expand this functionality and allow users to create their own cyber ranges in the cloud in a product called SimSpace VCN (press release). A VCN is a virtual clone network that is self-contained and isolated from the internet. VCN instances can be spun up and down on demand. This is a pretty awesome use of Ravello that goes a bit beyond what I’ve seen before.

Virtual Clone Networks and Use Cases

Each VCN starts as a blueprint and multiple instances can be deployed using Ravello’s hypervisor in the target cloud. You can deploy multiple DMZs, layer on additional networking like VLANs and port mirroring, and add just about anything else you want to replicate from your production environment. The network will contain not only the server OS VMs but a plethora of network and security devices from vendors such as Cisco, Checkpoint, Fortinet, and Palo Alto Networks. Existing policy settings (firewall, threat, etc.) can then be deployed on the appropriate VCN components. Each instance is completely isolated, allowing the user to treat each VCN as if it were production, but without the negative side effects if something goes wrong.  SimSpace’s traditional clientele would then run cyber defense simulations in the VCN to identify faults, train new users, and test the behavior of modifications such as replacing a firewall of one type with another or modifying policies. SimSpace’s product has an attack framework with the ability to inject common network attacks and even simulate “zero day” attacks.

I see a number of other use cases that SimSpace’s VCN product is useful for. The ability to replace a blueprint node or set of nodes can be used to test how different vendor’s products behave and whether they are suitable for the environment. Even in a virtualized data center, lab testing is often not representative of production behavior, but making the change in production is highly risky and expensive. Testing in a VCN can help provide similar scale to production that a lab cannot and at greatly reduced cost and risk.

Another potential use case is disaster recovery’s awkward sibling, business continuity (BC). Disaster recovery typically involves an online site where some portion of the system is always hot, at least to receive data replication from the primary environment. Business continuity, on the other hand, tends to involve cold and sometimes non-existent datacenters that are built from scratch to provide a minimum level of service during crisis times. Most BC exercises involve numerous runbooks and often end with some level of failure as runbooks tend to get out of date quickly. A VCN, however, can be generated rapidly from production documentation and deployed in less than an hour (more details below) and without the expensive of standby hardware or a business continuity contract.

Finally, auditing for compliance is always tricky. For example, the latest version of PCI-DSS standards require penetration testing, which introduces risks that some tests could cause outages or destroy data. Giving the auditor access to the VCN replica of production allows you and the auditor to map out the likely impact of penetration testing in a controlled manner with zero risk, enumerating the most likely outage scenarios and avoiding surprises. When the real penetration testing occurs in production, the risk can be reduced to an acceptable level for the business.

Product Offerings

SimSpace’s product will be offered in two flavors. A number of pre-defined blueprints exist that can be taken advantage of by users whose production environments closely match or who do not need a higher level of fidelity. These users can be up and running with their first VCN in about an hour, including signup time.

Customers who desire a higher level of fidelity or whose environments do not match the pre-defined blueprints can engage SimSpace about a customized VCN blueprint. SimSpace has a number of tools they are developing, the most promising of which works with Visio-like network diagrams that can be exported as a blueprint. The tool aims to be as simple as adding some metadata (IP, hostname, OS, etc.) to an existing diagram which should result in rapid turnarounds. If the VCN’s blueprint is updated, only the changes need to be deployed to the instance so deployment times remain low.

How It Works

SimSpace has shared some under-the-covers details with me. Each VM has at least two vNICS, one connected to a management network. All the management traffic is segregated from the production network to ensure management has no affect on the security testing results. Puppet is used to manage much of the node configuration, including networking and any user-provided software deployments. Just upload your software to the provided repository and assign the correct version to each node, puppet does the rest. (I mention this for no particular reason of course!) Spinning up a VCN instance with ~40 nodes takes less than 10 minutes for Ravello to deploy and 10 minutes for SimSpace to populate and configure, or about 20 minutes for an average configuration. The minimum network size is about 20 nodes and the current maximum is around 80 nodes. Their developers are pushing that to 150 nodes in tests now and will continue to increase that number.

In addition to replicating your production environment, SimSpace has a “internet bubble” component that can be added to any blueprint that adds a fake internet. A few VMs with thousands of IPs are able to replicate some level of core routing, root DNS, and fake versions of Facebook, Google, and other popular websites, to help simulate the isolated VCN communicating with the greater internet. I imagine this is helpful if you want to test some watering hole exploits or DNS amplification attacks.

There is currently no provided cost for the service. The target model is a monthly subscription service with additional fees for cloud usage and commercial licenses used in the VCN. Commercial licenses for products in each VCN instance will be handled by SimSpace, so there’s no need for users to worry about vendor management with SimSpace VCN. An early access program will be starting in the next week or two and general availability is expected in the 4th quarter of 2015. If you’re interested in the early access program, you can contact SimSpace directly.

All in all, I am very excited about SimSpace VCN. The amount of functionality it enables and the risk it reduces should have value to many individuals and businesses, and the reduction in cost of test environments is nearly limitless. Technologically, it’s also a really novel and powerful use of Ravello’s nested virtualization technology. I cannot wait to see SimSpace VCN in action and see its promise realized.

Why I Blog

I’ve wanted to write about why I blog for a while, and I was recently encouraged to stop procrastinating by Mattias Geniar:

Much is said, and frequently, about why you should blog. As I find most such articles to be impersonal, I thought I might share the reasons and rewards that have driven me to blog and keep me going at it. So, why do I blog?

  • To express myself. Sometimes this means artistically – being creative and showing it off – but other times it simply means organizing my thoughts and presenting them to other human beings. This forces me to clarify my thoughts, construct an actual hypothesis, and begin to test it. The end result is a refined idea that can be actually be consumed by myself and others. This is especially helpful if I will be presenting the idea to my boss or coworkers, even when that is done in a different format or medium.
  • To improve at writing. Communication is vital in any relationship, personal or business, and the written word can be tricky to wield effectively. I write emails every day, but I had not written a long-form article since college (15+ years ago, at the time!) and not on deeply technical subjects. I like to think this has been paying off for me, even with non-written communication as I’ve become more methodical and self-aware of how I communicate in all forms.
  • For community. I consume a lot from a number of different communities – security, virtualization, automation, etc. – and I feel that a good citizen contributes back when possible. Maybe I only help one other person, but I hope that I enable or inspire that person to do something awesome – like get home an hour earlier to spend more time with their family that evening.
  • As a portfolio of work. We all need to keep a portfolio, resume, C.V., etc. A blog is part of that – even if I don’t view it as a portfolio, others may, so it’s in my best interest to treat it as such. I keep this in mind before hitting publish – is this something that I want other people to see? Is it of high enough quality? Does it say something worthwhile? Does it send a positive message? Will someone else want to read this, and would they be satisfied if they did? Set your bar high and make sure you’re hitting it every time you publish something.
  • For recognition. This isn’t a very altruistic reason, but it has contributed to my efforts. A desire to write well enough to have a popular blog used by people everyday isn’t a bad thing to aim for, is it? Page views also give feedback on who your audience actually is, not who you think they are, and helps you see how they react to various article types and formats. Stats drive my morale and motivation. I like seeing that my page views went up 10% for a week, it makes me more eager to blog again. If page views go down for a few weeks, I want to know why and do better. Use it as a healthy feedback loop for your writing.

The last two reasons may seem a bit selfish, but I think that blogging as an independent is in many ways inherently self-serving. Improving my writing probably benefits me even more than building a portfolio or gaining recognition. Regardless, we all have egos and by acknowledging how they drive us, we can harness our drive rather than be controlled by it.

However, the most rewarding reason I blog, by far, is:

  • For my future self. I’ve referenced my own blog numerous times and even it had it come up as a Google result when I forgot that I had already solved a problem. Writing, reading, and applying my own article is a great feedback loop. Do something, write about it, do it again based on the article, rewrite the article, repeat until accurate. All the assumed knowledge is discovered and added to the article, bit by bit, so that anyone can follow the process. This is a practice you can apply to general documentation, as well. I also follow my own blog articles to replicate the results from my lab work, in my work environment (e.g. everything puppet related). This is critical to me, as I can prove to myself that I really have gained an understanding of the subject matter.

If you’re looking at blogging anytime soon, think about what it is you intend to get out of it. It can be extremely rewarding, but only if you go into it with some awareness. Have fun!

Visible Ops Phase Four: Enable Continual Improvement

The final phase of Visible Ops is Enable Continual Improvement. To really succeed with our efforts, we need to make sure that the resources we have are allocated optimally toward our business goals. With most of our fires put out and significant efforts into avoiding future fires, we have the time available to do this right. To determine where our resources should be allocated, we need to look at metrics.

Continue reading