Being an IT Operations Manager, you must have come across an infrastructure which consists of hundreds of thousands of virtual machines which are being used by globally located development teams. Multiple environments like development, testing, staging and pre-production, share these machines apart from the production environment. One day, a business requirement comes that predicts major impacts on the overall application. In order to implement it, multiple development teams tighten their shoes and request for ready-to-work environments in order to carry out development activities. Even testing teams want to stay prepared with the test-environments dedicated for their activities.
Being a “Manager of IT Operations,” you are under tremendous pressure of delivering these machines with pre-loaded resources like databases, browsers, users, files etc., in a short span of time. I know thinking of doing these tasks manually will make you curse yourself for being “IT Operations Manager”. You can write various shell scripts for these tasks though! However, you cannot give assurance that it will work 100% of the time. If any of these scripts fails halfway during the execution, you will not have any idea where to start for the next run. In today’s DevOps world, Integration and Delivery are going Continuous every day. Spending time on building infrastructure manually for every requirement is not feasible, and something more than script automation has become the need of an hour. Apart from distributing these environments to different teams, you also become a responsible person to keep these environments on the same page or “in sync”. So, your job is not limited and you are earning a special place at your workplace. At the same time, you also realize that you are in need of something which will take care of these configuration tasks for you. This need of yours also brings infrastructure configuration management (Infra CM) tools into picture. Some of the market leaders in this area are Chef, Puppet, Salt Stack and Ansible.
In this blog, we will focus on Puppet. Please do not get surprised if you hear it from me, that using Puppet for all the tasks mentioned above can be done by creating only one file on one server (called master) and on a single click will be reflected in all the hundreds of thousands of nodes present in your infrastructure. That is the kind of job these Infra CM tools do in a much simplified manner. Yes you read that right! (smile)
Puppet is available in two different versions, Open Source Puppet and Puppet Enterprise. In this blog I am going to introduce you all to Open Source Puppet, a very widely used Infra CM tool. I will talk about Puppet Architecture, Installing Puppet on Master and Agent nodes, Writing our own Puppet Module, Writing Puppet Manifest file, Using Puppet to create a file on all the nodes, Using Puppet to install Chrome, Apache and MySQL on all the nodes.
Puppet Architecture:
Puppet runs in an agent and master architecture, where puppet master server controls important configuration information and managed agent nodes request only their own configuration catalogs. These managed nodes run the “puppet agent” application in the background. At the same time, there will be one or more servers which will have “puppet master” applications running in the background. In a periodic manner (by default 30 minutes), Puppet Agent (PA) will send its own facts to the Puppet Master (PM) and request a catalog that asks the PM, “do I look like the way I am supposed to look?”. In response to that question, PM will compile and return the respective node’s catalog. PM will use it’s access to several sources types of information for this activity. As per the description of each resource received in the catalog, if the answer is YES, nothing happens. However if the answer is NO, that means DRIFT has been occured. In a NO case, PM tells respective PA how it should look, then that node gets to work and returns itself to the state it should look like. In this way Puppet lets you specify the desired state within your configuration.

image2015-9-15-14-42-20-1.png

Puppet Installation:
Here I will try to show you how to set up above mentioned architecture in your infrastructure. This will help you to get started with Puppet. Or for a lack of better words, “Your Puppet Show begins!” In order to achieve this, try to meet below pre-requisites.
Pre-Requisites: In your IT infrastructure try to get
i. One Linux machine with at least 4GB RAM. This machine will serve as the Puppet Master in our architecture. I used Ubuntu for this blog purpose.
ii. One or Two machines with at least 1GB RAM and any operating system CentOS, Linux, Windows etc (having different OS on two machines is preferred for better understanding if you are planning to get two machines). These machines will play the role of Puppet Agents (nodes) in our architecture.
I used Ubuntu for one and the other CentOS for this blog purpose.
iii. Open SSH installed on all these machines.
For Ubuntu: sudo apt-get install openssh-server
For CentOS: sudo yum install openssh-server
Installing Puppet on Master Server [Ubuntu]:
To install Puppet execute below commands in the same order
Command

sudo dpkg -i puppetlabs-release-trusty.deb


Output

--2015-09-10 19:25:28-- https://apt.puppetlabs.com/puppetlabs-release-trusty.deb

Resolving apt.puppetlabs.com (apt.puppetlabs.com)... 192.155.89.90, 2600:3c03::f03c:91ff:fedb:6b1d

Connecting to apt.puppetlabs.com (apt.puppetlabs.com)|192.155.89.90|:443... connected.

HTTP request sent, awaiting response... 200 OK

Length: 7384 (7.2K) [application/x-debian-package]

Saving to: `puppetlabs-release-trusty.deb'

100%[================================================================================>] 7,384 --.-K/s in 0.002s

2015-09-10 19:25:29 (3.87 MB/s) - `puppetlabs-release-trusty.deb' saved [7384/7384]


Command

sudo dpkg -i puppetlabs-release-trusty.deb


Output

Selecting previously unselected package puppetlabs-release.

(Reading database ... 192532 files and directories currently installed.)

Unpacking puppetlabs-release (from puppetlabs-release-trusty.deb) ...

Setting up puppetlabs-release (1.0-11) ...


Command

sudo apt-get update


Output

Get:1 http://dl.google.com stable Release.gpg [198 B]

Hit http://dl.google.com stable Release.gpg

Get:2 http://archive.canonical.com precise Release.gpg [198 B]

Get:3 http://dl.google.com stable Release [1,347 B]

:

:

Hit http://archive.ubuntu.com precise-backports/universe Translation-en

Get:42 http://security.ubuntu.com precise-security/universe Translation-en [78.0 kB]

Fetched 3,333 kB in 15s (211 kB/s)

Reading package lists... Done


Command

sudo apt-get install puppetmaster


Output

Reading package lists... Done

Building dependency tree

Reading state information... Done

The following extra packages will be installed:

augeas-lenses debconf-utils facter hiera libaugeas-ruby libaugeas-ruby1.8 libaugeas0 libjson-ruby libruby libshadow-ruby1.8 puppet-common puppetmaster-common ruby-json virt-what

:

:

Setting up puppet-common (3.8.2-1puppetlabs1) ...

Setting up puppetmaster-common (3.8.2-1puppetlabs1) ...

* Starting puppet queue [ OK ]

Setting up puppetmaster (3.8.2-1puppetlabs1) ...

* Starting puppet master [ OK ]

Processing triggers for libc-bin ...

ldconfig deferred processing now taking place


Now you can check if Puppet is installed on the master and if it is running or not with the help of below commands:

Command

puppet -v


Output

3.8.2


Command

service puppetmaster status


Output

* master is running


You can stop, start and restart Puppet master with the help of below commands:

Command

service puppetmaster stop

service puppetmaster start

service puppetmaster restart


Installing Puppet on Agent Nodes [Ubuntu and CentOS]:

Login to the Agent Nodes. Sudo to root user. To install Puppet Agent execute below commands in the same order.
For Ubuntu:

sudo wget https://apt.puppetlabs.com/puppetlabs-release-trusty.deb

sudo dpkg -i puppetlabs-release-trusty.deb

sudo apt-get update

sudo apt-get install puppet


For CentOS:

sudo rpm -ivh http://yum.puppetlabs.com/puppetlabs-release-el-6.noarch.rpm

sudo yum install puppet


Check the Puppet Version with command: puppet -V

Managing hosts files on master and agents
Below are IP addresses of my machines
Puppet Master: 10.1.1.215
Puppet Agent (Ubuntu): 10.1.1.186
Puppet Agent (CentOS): 10.1.1.101

On master your /etc/hosts file should like this:

cat /etc/hosts

127.0.0.1 localhost

127.0.1.1 puppet-master

10.1.1.186 puppet-u-agent

10.1.1.101 puppet-c-agent


On agents your /etc/hosts file should like this:

Ubuntu Agent

cat /etc/hosts

127.0.0.1 localhost

127.0.1.1 puppet-u-agent

10.1.1.215 puppet-master


CentOS Agent

cat /etc/hosts

127.0.0.1 localhost

127.0.1.1 puppet-c-agent

10.1.1.215 puppet-master


Note: This /etc/hosts editing is only for exploring Puppet through this blog. At organisation level these settings will be done in private or intranet DNS

Editing Puppet files:
On the agents find the puppet configuration file “/etc/puppet/puppet.conf”. Make below changes as seen on line 17.

vi /etc/puppet/puppet.conf

[main]

logdir=/var/log/puppet

vardir=/var/lib/puppet

ssldir=/var/lib/puppet/ssl

rundir=/var/run/puppet

factpath=$vardir/lib/facter

#templatedir=$confdir/templates

server = puppet-master

#[master]

## These are needed when the puppetmaster is run by passenger

## and can safely be removed if webrick is used.

#ssl_client_header = SSL_CLIENT_S_DN

#ssl_client_verify_header = SSL_CLIENT_VERIFY


Then find one more file on agents to edit “/etc/default/puppet” and make below changes as seen on line 3.

[...]

START=YES

[...]


Restart puppet service on agents once you are done with these file edits.

Certificate exchange from Puppet master to Puppet agents
Each master or agent must have an identifying SSL certificate, and will examine their counterpart’s certificate to decide whether to allow an exchange of information between master and agent. When Puppet is first run on agent node, it will send a certificate signing request to master. In order to start communicating with agent nodes, master must sign individual agent node’s certificate. You can check the pending requests for master’s approval.
Execute below command on master server.

sudo puppet cert list


Output

"puppet-u-agent" (SHA256) 2B:BB:33:C5:5E:52:51:7A:6D:CE:84:9B:12:D4:55:36:.....CC:BA:70:BB:E2:C1:4A:98:7A

"puppet-c-agent" (SHA256) 7B:B8:62:8E:69:DA:69:0A:EE:99:F2:DC:34:30:89:A2:.....E9:38:6B:A7:31:61:FE:49:65


Note that there is no “+” sign in front of these certificates. This indicates that they have not been signed yet.

Signing a Request: Execute below command for every node from master server in order to sign a certificate for that particular node.

sudo puppet cert sign puppet-u-agent


Output

Notice: Signed certificate request for puppet-u-agent

Notice: Removing file Puppet::SSL::CertificateRequest puppet-u-agent at '/var/lib/puppet/ssl/ca/requests/puppet-u-agent.pem'


The above mentioned “puppet cert list” command will not show pending certificate for “puppet-u-agent” node as signing has been carried out by master for this node.

To view all the certificates signed by master along with the pending ones execute below command.

sudo puppet cert sign --all


Output

+ "puppet-master" (SHA256) 2B:D3:34:31:81:66:5B:FF:49:52:9D:8D:0B:13:CD:D4:.....87:BD:63:1D:99:EB:E0:9F:2D (alt names: "DNS:puppet", "DNS:puppet-master")

+ "puppet-u-agent" (SHA256) 27:6D:5E:9A:9F:FB:04:10:03:9B:4D:CE:D8:2F:62:91:.....91:D5:90:C1:07:AE:ED:FB:6C

"puppet-c-agent" (SHA256) 7B:B8:62:8E:69:DA:69:0A:EE:99:F2:DC:34:30:89:A2:.....E9:38:6B:A7:31:61:FE:49:65


Note that there is “+” sign in front of one certificate. The other certificate can also be signed with the above commands.

With successful signing process the communication gets established between master and all the nodes. With this we can say that our infrastructure is ready to play with puppet.
Puppet Language
When I say that Puppet configures all the machines in the infrastructure in a single click, I am actually talking about a Puppet Code. Puppet uses its own domain-specific language (DSL) to describe re-usable pieces of configuration. Puppet code is primarily consisted of resource declarations. This Puppet code is saved in files called manifests with “.pp” extensions, which are in turn, stored in structured directories called modules. Puppet also has its own repository of modules, it is called the Puppet Forge. Many users around the globe contribute to this library by sharing their self-written modules. For example, if we are in need of some module like say the Chrome browser module, then we can search for it in Puppet Forge and if found, can directly install it on puppet master. This later configures it with other required nodes.

Installing Module from Puppet Forge:
If you decide to download a module from Puppet Forge (frankly speaking, a good move, why reinvent the wheel when something is readily available) here is a way to install it. Let’s say you want to have an apache service running on all your nodes. You go to Puppet Forge, search for Apache module, then you select the one which is “Puppet Supported” (as they are fully tested) and execute the command mentioned there on Puppet Master.

image2015-9-11-16-14-591.png

Output

Notice: Preparing to install into /home/puppetm/.puppet/modules ...

Notice: Downloading from https://forgeapi.puppetlabs.com ...

Notice: Installing -- do not interrupt ...

/home/puppetm/.puppet/modules

└─┬ puppetlabs-apache (v1.6.0)

├── puppetlabs-concat (v1.2.4)

└── puppetlabs-stdlib (v4.9.0)


In this manner, with build-in “puppet module” command Puppet Forge modules can be quickly installed. Installing Apache module on master does not mean that Apache service is available on PM server. It means that modules for installing and maintaining Apache are available on PM server which can later be used for installed on PA nodes.

Writing Your Own Module with File Resource:
Below I have given an example of user defined module “add_plugin”. This piece of code will be present in “init.pp” file of this module. The tree-structure will look something like this:

# Below mentioned file defines class named add_plugin (same as module name). This module copies plugin jar file to the agent node(s)

  /etc/puppet/modules/add_plugin/manifests/init.pp


# Below mentioned is the file to be transferred to the agent node(s) with pre-defined location

  /etc/puppet/modules/add_plugin/files/myplugin_latest.jar


init.pp for this module file looks like this:

# Defining a class add_plugin which consists of a file resource

class add_plugin {

  # Below is the target location on agent node where latest plugin file needs to be copied

    file { '/home/myfolder/plugins/installed-plugins/myplugin.jar' :

    ensure => 'present',

    # Below is the path of the latest plugin file available on master server

    source => 'puppet:///modules/add_plugin/myplugin_latest.jar',

    owner => 'user123', # other optional attributes

    group => 'userpass',

    mode => '664',

  }

}


site.pp: This is a default main manifest file in Puppet and is located at “/etc/puppet/manifests/site.pp”. This file mentions the client nodes where configurations need to be applied by master. Every node definition consists of class declarations and resource definitions like users, files, packages etc. That is the class definition which we saw earlier in init.pp of our module will do nothing unless it gets declared in this site.pp file. Below is the manifest file which is available on my master and which configures the other two nodes which we set up earlier.

# Below default node definition applies to nodes that are not explicitly defined in this manifest file

node default {

}



# Below node definition applies to nodes puppet-u-agent and puppet-c-agent

node 'puppet-u-agent', 'puppet-c-agent' {

  # Below is the required file location on agent node.
   # For testing purpose make sure there is no hello file present under /tmp folder

  file { '/tmp/hello':

    ensure => "present", # This attribute makes sure the file exists

    # content attribute adds below content to hello file we are creating on agent

    content => "Ubuntu agent is able to connect to Puppet Master\n",

  }



  class { 'add_plugin': } # Declaring our self-defined class



  # Below we are declaring class Apache, which effectively adds resources under this class to the catalog so that Puppet can manage that resources's state on every request by agent node

  class { 'apache': }



  # define vhost resource which is a part of Apache class

  apache::vhost { 'example.com':

    port => '80', # passing parameters to the resource

    docroot => '/var/www/html'

    # Other attributes will take default values as mentioned in resource definition

  }



  # Declaring class which will install Chrome and Firefox on the agents

  include software::browsers::chrome

  include software::browsers::firefox



  # Here we are declaring mysql class which will install MySQL on agent if not already installed

  class { '::mysql::server':

    root_password => 'rootpass123',

    remove_default_accounts => true,

  }



  # Below declaration helps creating database 'testdb' if not present already on the MySQL database

  mysql::db { 'testdb':

    user => 'test', # User credentials can be configured in this declaration part

    password => 'test123',

    host => 'localhost',

  }

}


Applying a Manifest

Once we are ready with the site.pp manifest as mentioned above, we can manually apply this manifest using puppet agent command in below manner. If not manually, this manifest will anyways gets applied after 30 minutes, which is the default refresh time for Puppet Nodes.

sudo puppet agent --test


After applying this manifest, if everything is successfully executed, we should find both the PA nodes with below configuration:

New /tmp/hello file available with proper content
Apache service running
Browsers Chrome and Firefox available
MySQL installed and database ‘testdb’ is present
Now do a small test. In the site.pp file change the content of hello file in file resource definition like this:

# content of this file to be present on agent node

  content => "Hello file is updated in this configuration update.\n",


Once you are done with this change do nothing. Wait about 30-40 minutes and then check the PA nodes. You should find the hello file updated as expected on both the nodes. Also nothing else happens apart from this file update as all the nodes already are in desired state in terms of other configurations.

Open Source Puppet and Puppet Enterprise
Open Source Puppet lets you describe machine configurations in a declarative language, and then automates both creation and enforcement of your desired state. But this is suitable for simple level environments. Companies running the largest, most complex enterprise environments use Puppet Enterprise. Its next-generation architecture supports greater scalability and includes a comprehensive set of commercial-only capabilities and enterprise support. Open source Puppet is free whereas Puppet Enterprise is free on up to 10 agent nodes. Some of the features provided by Puppet Enterprise are beautiful user interface, event inspection on every node, role-based access control, puppet server reporting, puppet code manager, puppet configuration manager, automated provisioning, and full enterprise support.
I will cover Puppet Enterprise and it’s features in the next upcoming blog. Until then, please stay tuned.

References:
You may find below references useful for further reading on Puppet.
https://docs.puppetlabs.com/puppet/3.8/reference/architecture.html
https://docs.puppetlabs.com/puppet/3.8/reference/lang_summary.html
https://forge.puppetlabs.com
https://puppetlabs.com/puppet/enterprise-and-open-source