Create a Yum Repo

In last week’s article, we learned how to build a package with FPM, specifically an RPM. Today, we’ll look at creating a Yum repository to host your packages. The repo can be built with puppet, which can also distribute settings so all your managed nodes can use the repo. By adding the package to the repo, it becomes available to install, again via puppet. This is the first step on the road to the automated software packing and delivery that is vital for continuous integration.

A repo has a few components.

  • Webserver – Content is served up over http.
  • createrepo – A piece of software that manages the repo’s catalog.
  • RPMs – What it’s serving up.

We don’t need to know how the pieces work, though. We’ll rely on palli/createrepo to manage the repo itself. We just make sure a webserver is available, the directories are there, and that there’s some content available.

Configure a host

I’m going to start moving faster now, because we’ve done this part a few times already. Please let me know on twitter if I’m going to fast and I can update the page and make sure future articles keep the same pace.

The first thing is to choose a node. Since our series so far relies on the hostname for a role, spin up a new VM called ‘yumrepo01′ or something similar. You don’t need to configure anything afterward, except possibly update Puppet (as I’m writing this, v3.7.0 was just released). Of course, you should run the agent and accept the cert so that the node can communicate with puppet before continuing.

Next up is the manifest. Start with the profile.

[rnelson0@puppet profile]$ cat manifests/yumrepo.pp
class profile::yumrepo {
  include '::profile::apache'

  apache::vhost {'yum.nelson.va':
    docroot    => '/var/www/html/puppetrepo',
  }
}

Our existing ::profile::apache ensures the web server is installed and running and creates firewall rules to allow traffic in. The apache::vhost definition creates the virtual host for our web server. We will place our repo underneath /var/www/html/puppetrepo but you can choose anywhere you want. The next piece is the role.

[rnelson0@puppet role]$ cat manifests/yumrepo.pp
class role::yumrepo {
  include profile::base  # All roles should have the base profile
  include profile::yumrepo

  $repodirs = hiera('repodirs')
  file { $repodirs :
    ensure => 'directory',
  }
  create_resources('::createrepo', hiera_hash('yumrepos'), {require => File[$repodirs]} )
}

After adding the base and yumrepo commands, we need to create the directories for the repo itself. The vhost doesn’t actually create the specified directory, and there’s also a cache directory. We also create some ::createrepo defines. All of this content comes from hiera:

[rnelson0@puppet hiera-tutorial]$ cat puppet_role/yumrepo.yaml
---
classes:
  - role::yumrepo
repodirs:
  - '/var/www/html/puppetrepo'
  - '/var/cache/puppetrepo'
yumrepos:
  'el-6.5':
    repository_dir       : '/var/www/html/puppetrepo/el-6.5'
    repo_cache_dir       : '/var/cache/puppetrepo/el-6.5'
    suppress_cron_stdout : true

Our repo has the name el-6.5 (based on our CentOS 6.5 base image) and exists one level underneath our docroot and cache directory (there’s a good reason why). There’s a cron job that runs every minute, rebuilding the index. Unless you like getting new mail every minute, I suggest suppressing output. Here’s the cron job it will create:

# Puppet Name: update-createrepo-el-6.5
*/1 * * * * /usr/bin/createrepo --cachedir /var/cache/puppetrepo/el-6.5 --changelog-limit 5 --update /var/www/html/puppetrepo/el-6.5 1>/dev/null

There are other options you can feed ::createrepo, but this will suffice to start. With the role and profile manifest and hiera yaml in place on the master, you can run the agent on yumrepo01 and everything should install properly. You can test access by visiting http://yum.nelson.va/ afterward. You should see the el-6.5/repodata directory structure, and the bottom dir will have a few files. Great success!

Populating the repo

This step is easy – copy files to the repo. Where packages go is up to you. You can toss them in the top level, you can create an RPMS or Packages directory; wherever you through it, createrepo will find it. We’ll put our files at the top level (/var/www/html/puppetrepo/el-6.5/ in this case). The crontab job runs once a minute, so after a brief wait it will be ready. You can again browse to the repo at http://yum.nelson.va/el-6.5/, using the correct hostname for your environment.

If you do use subdirectories, ::createrepo does not create or manage this subdirectory, so you would have to. Be aware that by doing so, you’ll be managing a directory that is managed by the define and you’ll have to ensure you don’t manage the same directory twice, once via ::createrepo and twice via a File resource. You could set manage_repo_dirs to false in your ::createrepo definition if you’d rather manage all the directories yourself.

Using the repo

Finally, we need a way to make use of the repo. A yumrepo resource type is native to puppet, so let’s look at the definition for our new repo:

  yumrepo {'el-6.5':
    descr    => 'rnelson0 El 6.5 - x86_64',
    baseurl  => 'http://yum.nelson.va/el-6.5/',
    enabled  => 'true',
    gpgcheck => 'false',
  }

We are defining the repo el-6.5, giving it a description and the URL to use. Adding a repo will just create the repo file but it defaults to disabled, so we set enabled to true. The default for gpgcheck, which ensures signatures on RPMs are valid, is true. Since we haven’t signed our package, we’ll set that to false. I’ll leave configuration of gpg, or lack thereof, up to you. Let’s go back to server01 and use puppet apply to attach the repo. Afterward, search for helloworld.

[rnelson0@server01 ~]$ yum repolist
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * base: bay.uchicago.edu
 * extras: mirror.steadfast.net
 * updates: mirror.anl.gov
repo id                                   repo name                                                     status
base                                      CentOS-6 - Base                                               6,367
extras                                    CentOS-6 - Extras                                                15
puppetlabs-deps                           Puppet Labs Dependencies El 6 - x86_64                           68
puppetlabs-products                       Puppet Labs Products El 6 - x86_64                              422
updates                                   CentOS-6 - Updates                                            1,467
repolist: 8,339
[rnelson0@server01 ~]$ sudo puppet apply
  yumrepo {'el-6.5':
    descr    => 'rnelson0 El 6.5 - x86_64',
    baseurl  => 'http://yum.nelson.va/el-6.5/',
    enabled  => 'true',
    gpgcheck => 'false',
  }
Notice: Compiled catalog for server01.nelson.va in environment production in 0.11 seconds
Notice: /Stage[main]/Main/Yumrepo[el-6.5]/ensure: created
Notice: Finished catalog run in 0.07 seconds
[rnelson0@server01 ~]$ yum repolist
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * base: bay.uchicago.edu
 * extras: mirror.steadfast.net
 * updates: mirror.anl.gov
el-6.5                                                                                 | 2.9 kB     00:00
el-6.5/primary_db                                                                      | 2.3 kB     00:00
repo id                                   repo name                                                     status
base                                      CentOS-6 - Base                                               6,367
el-6.5                                    rnelson0 El 6.5 - x86_64                                          1
extras                                    CentOS-6 - Extras                                                15
puppetlabs-deps                           Puppet Labs Dependencies El 6 - x86_64                           68
puppetlabs-products                       Puppet Labs Products El 6 - x86_64                              422
updates                                   CentOS-6 - Updates                                            1,467
repolist: 8,342
[rnelson0@server01 ~]$ yum search helloworld
Loaded plugins: fastestmirror
Loading mirror speeds from cached hostfile
 * base: bay.uchicago.edu
 * extras: mirror.steadfast.net
 * updates: mirror.anl.gov
========================================== N/S Matched: helloworld ===========================================
helloworld.x86_64 : no description given

  Name and summary matches only, use "search all" for everything.

Go ahead and try and install it with yum (abbreviated output):

[rnelson0@server01 ~]$ sudo yum install helloworld
...
Resolving Dependencies
--> Running transaction check
---> Package helloworld.x86_64 0:1.0-1 will be installed
--> Finished Dependency Resolution
...
Is this ok [y/N]: y
Downloading Packages:
http://yum.nelson.va/el-6.5/helloworld-1.0-1.x86_64.rpm: [Errno 14] PYCURL ERROR 22 - "The requested URL returned error: 403 Forbidden"

If you check the permissions on the file, you might wonder why a world readable file isn’t available. Our CentOS template enforces selinux, so let’s take a look at the context:

[root@yumrepo01 el-6.5]# ls -laZ /var/www/html/puppetrepo/el-6.5/helloworld-1.0-1.x86_64.rpm
-rw-rw-r--. rnelson0 rnelson0 unconfined_u:object_r:user_home_t:s0 /var/www/html/puppetrepo/el-6.5/helloworld-1.0-1.x86_64.rpm
[root@yumrepo01 el-6.5]# ls -laZ /var/www/html/puppetrepo/el-6.5/repodata/repomd.xml
-rw-r--r--. root root unconfined_u:object_r:httpd_sys_content_t:s0 /var/www/html/puppetrepo/el-6.5/repodata/repomd.xml

Here’s our issue: the new rpm is not in the httpd_sys_content_t context but is in the user_home_t context. This occurred because I scp’ed the file from server01 as rnelson0 to yumrepo01 as rnelson0, then on yumrepo01 moved the file to /var/www/html/puppetrepo as root. Let’s move the file back, then try copying it as root:

[root@yumrepo01 el-6.5]# mv /var/www/html/puppetrepo/el-6.5/helloworld-1.0-1.x86_64.rpm ~rnelson0/
[root@yumrepo01 el-6.5]# cp ~rnelson0/helloworld-1.0-1.x86_64.rpm /var/www/html/puppetrepo/el-6.5/
[root@yumrepo01 el-6.5]# ls -laZ /var/www/html/puppetrepo/el-6.5/helloworld-1.0-1.x86_64.rpm
-rw-r--r--. root root unconfined_u:object_r:httpd_sys_content_t:s0 /var/www/html/puppetrepo/el-6.5/helloworld-1.0-1.x86_64.rpm

That’s better! You can also use the restorecon command to restore the default context on files, i.e. restorecon /var/www/html/puppetrepo/el-6.5/helloworld-1.0-1.x86_64.rpm. Now try and install it again:

[rnelson0@server01 ~]$ sudo yum install helloworld
...
Is this ok [y/N]: y
Downloading Packages:
helloworld-1.0-1.x86_64.rpm                                                            | 1.7 kB     00:00
Running rpm_check_debug
Running Transaction Test
Transaction Test Succeeded
Running Transaction
  Installing : helloworld-1.0-1.x86_64                                                                    1/1
  Verifying  : helloworld-1.0-1.x86_64                                                                    1/1

Installed:
  helloworld.x86_64 0:1.0-1

Complete!
[rnelson0@server01 ~]$ rpm -qa | grep helloworld
helloworld-1.0-1.x86_64
[rnelson0@server01 ~]$ rpm -ql helloworld
/var/www/html/index.php

Summary

You now have a package, a repository that hosts it, and a client that can install the packages hosted by the repo. Go ahead and spin up server02-server10 and you should be able to easily define the repo and install the rpm. As we haven’t created any dependencies yet, the lack of httpd will prevent our app from working. If you install apache manually or via puppet, your hello world app will work. We’ll work on some more details of FPM, including dependencies, and integration of packages and repos with our puppet manifests next time.

References

Creating packages with FPM

In my exploration of Puppet, I’ve found a lot of oblique references to managing software deployments with it, but very little solid guides on how to do so. I need to tackle this for work, so I figured I should start at the top – creating a software deployment. To be clear, I’m speaking of internally developed software, or modifications to public software, not something you can find in your distribution’s packages and install with the puppet package resource type.

Creating Some Software

Going back even further, we need to create some software. I’d wager that most already have something laying around – perhaps a few scripts in a directory along with a Makefile that lets you run “make install” to put them in the final destination, or a tarball and config file that are “installed” by a script that untars the software and copies your customized config in place. If you don’t have something like that, let’s make something. How about a simple PHP application? It’s just a Hello World, nothing special, so you don’t need to know PHP for this.

Spin up a new VM, or requisition one of your existing dev VMs. I’m going to use server01 from the Puppet series. Make sure apache and php are installed, and if this node isn’t managed via our web server role, iptables may block connections so we will stop it:

[rnelson0@server01 ~]$ sudo puppet apply -e "package {['httpd', 'php']: ensure => present}"
Notice: Compiled catalog for server01.nelson.va in environment production in 0.33 seconds
Notice: /Stage[main]/Main/Package[php]/ensure: created
Notice: Finished catalog run in 20.70 seconds
[rnelson0@server01 ~]$ sudo service httpd restart
Stopping httpd:                                            [FAILED]
Starting httpd:                                            [  OK  ]
[rnelson0@server01 ~]$ sudo service iptables stop
iptables: Setting chains to policy ACCEPT: filter          [  OK  ]
iptables: Flushing firewall rules:                         [  OK  ]
iptables: Unloading modules:                               [  OK  ]

Continue reading

Puppet Scale Up with Apache/Passenger

Welcome back! I hope everyone had a good summer and recharged their batteries. Bonus points if you found time to play with puppet, too! Now that we’ve had a healthy break, let’s get back to it!

When we left the series in July, we had a Puppet master, a few nodes, were implementing the roles and profiles pattern, and used r10k to manage it all. However, we didn’t address scalability. Today, we’ll take a look at addressing this by using Apache and Passenger.

Scaling Up

There are two ways to scale – out and up. If we were to scale out, we’d be concerned with running multiple masters and synchronizing all data between them. That’s something we might look at eventually, but first we want to scale up, which is the process of providing more resources to our master. Since we are vSphere admins, we can easily increase the resources provided to the VM. For instance, our VM has 1 vCPU and 2GB of RAM. It would be easy, and helpful, to increase that, perhaps to 2×4 or 4×8 vCPUxRAM.

Unfortunately, system resources are not the only limitation in our system. Out of the box, Puppet uses WEBrick and scales to about 10 nodes. More than one nodes trying to talk at the same time will generate conflicts and cause some or all nodes to fail to receive a catalog. No matter the resources available, these limitations persist. The answer is to use a dedicated web server with a Rack-based application stack. While any server will work, if you don’t have a preference, then PuppetLabs suggests you use Apache with the Passenger mod. There is a lot of information on Puppet’s site about the limitations and the remedy.

Continue reading

Summer Wrap-Up – #VirtualDesignMaster, #HyperV, and more

With summer coming to a close, there’s an unavoidable temptation to look book on what was. This year has been pretty busy, so I thought I’d take some time to look back on my accomplishments.

In February, I started this blog. I set out with a goal of one article per week on the blog. In the last seven months, I’ve met that average for the week even if I don’t write another article (52 articles in 29 weeks), but I also learned that it’s not feasible for me to actually write one article every week. I wrote three series, on PowerCLI, Auto Deploy, and Puppet for the vSphere Admin (ongoing) and managed over 14,000 page views in ~7 months. I have been lucky enough to convince Jason Crichton (@hawkbox) to contribute articles on Hyper-V while I took a summer break, and there’s really no point in trying to advertise new content during VMworld next week. I count the blog as a success and I learned some lessons along the way.

Speaking of which, I am unable to attend VMworld this year. Last year was my first VMworld and I sorely miss the opportunity to go back and to meet all my tweeps. I’ll be watching online, however!

In the area of personal growth, I acquired my VCP5-DCV certification in April and became a vExpert in July. Both were exciting, the certification because it validated my technical knowledge and the vExpert because it validated my community activities. The fact that both came with some free licenses didn’t hurt, either. I plan to put those to good use in the coming months to continue my education! I’m looking forward to being part of these two communities and learning from the other members.

In July, Season 2 of Virtual Design Master (VDM) began. I was one of 11 contestants. I survived round 1 and round 2, but I had to fold on round 3 of 4 due to time commitments. Congratulations to Timothy Patterson (@PcProfessionals) for winning Season 2! I learned a LOT during this competition – taking all the skills I use on a day to day basis as a vSphere admin, what I studied for my VCP, and a number of vSphere design books and applying it to design and engineering, which is not part of my day job in operations. I haven’t done any actual design work in years, and then it was always on a much smaller scale. VDM was challenging, exciting, and scarey. It was also very rewarding. In addition to the technical growth, there was a great sense of community and co-opetition between the contestants and the viewers of the competition. Check out #VirtualDesignMaster on Twitter for some great discussions. There’s some good content about IPv6 and Openstack in particular.

As I mentioned above, I was able to convince Jason to become a contributing editor on my blog. He is in the middle of his own Hyper-V series, which now has its own page. As a vSphere admin myself, I find the content very helpful – the world isn’t all VMware – and I hope that others find it useful as well. We would both love to hear any comments you have on the series and what direction it should go in.

In addition to all of the above, my wife changed jobs in August and we moved from Richmond, VA to Indianapolis, IN. We’re really excited about the new job and home, but we’re both very exhausted, too. If you’re in the Indianapolis area and I didn’t meet you at the IndyVMUG, drop me a line.

Have a great summer, and enjoy VMworld if you are attending!

Installing SCVMM for Hyper-V

Welcome Back! For this article I am going to try and cover SCVMM (System Center Virtual Machine Manager) well enough to allow you to install it for testing purposes. This isn’t intended to cover all of the details as I think there is too much material here for one post and I haven’t had time with my new job to properly explore deeply yet. If I have missed something important or you would like me to go into more detail on a part of this, please send me a message and I will be happy to look into it.

SCVMM is the functional Hyper-V equivalent to vCenter in a vSphere environment. I have rather extensive experience dealing with vCenter, so I wanted to try and familiarize myself with SCVMM in the same way. With System Center 2012 and 2012 R2 it comes as part of the entire package, which is rather handy. I am going to go through step by step and show my experience with it. I will make note of any caveats or landmines I come across during the setup.

Probably the number one thing to consider before getting started is your SQL environment, where are you going to host your SQL DB? I am in the process of familiarizing myself with SQL 2012 right now so I have chosen to deploy this on another server. You are also able to use a local SQL Express install should you only want to run this in testing. I  would recommend a dedicated server if you plan to run this in production.
Continue reading

Hyper-V memory oversubscription

Welcome back to my third article, I hope these provide some useful information. This posting is based around my observations of the differences in how vSphere and Hyper-V manage memory on the host. Without further rambling, let’s get started.

Hyper-V handles memory in a noticeably different way than vSphere does. This has taken me some getting used to but the largest take away is that it does not overcommit memory. Microsoft uses the term Dynamic Memory for their version, and based on my observations that is a good term for it.

Memory over allocation in vSphere is handled through the VMTools and the balloon driver in instances of the memory allocation actually being utilized. This historically, to my understanding, results in paging to disk when recovered memory isn’t adequate for the needs to be met. This is where Dynamic Memory kind of breaks my brain. Unlike vSphere, which more or less just assumes you will overcommit at some point, you need to explicitly enable this functionality. It’s not complicated, just not something you might think to do when coming from a VMware environment.
Continue reading

Hyper-V Replication

Hello again, welcome back to the second of my indeterminate number of articles covering my various observations regarding differences around vSphere and Hyper-V. This post is covering host replication without a central controller. So hopefully you benefit from and enjoy this one.

As near as I can tell there is no direct analog in vSphere to compare this to.  As such I’m going to give as much of an explanation as I realistically can and show where I think this would be extremely beneficial.

Hyper-V replication does not require SCVMM or any kind of centrally managed vCenter equivalent, all it requires is two or more Hyper-V hosts that meet the necessary requirements for performing the replication. (CPU, Memory, Storage, etc….) Once that is met it is quite straight forward to do as a test.
Continue reading