Puppet & Foreman : The scalable way

Oye there !

We’ve been struggling with a single Puppetmaster for around 250+ nodes in my company for the last months, and I decided to review the actual architecture toward something more evolved. Managing the nodes in site.pp didn’t seem natural for me and the file began to be unmaintanable.

So I’ve googled around for some alternatives and found something called ENC, standing for External Node Classifier, and more specifically Foreman.

I’ve also decided to add high availability and load balancing in my architecture, and I’ve come up with the following:

I’ll describe in this post how to setup the same stack on Ubuntu Server 12.04 nodes using Foreman 1.5.1 and Puppet 3.6.2.

Requirements

Nodes

As you can see in the scheme above, this setup is composed of 10 nodes.

You’ll need to setup the following nodes (I’ve specified IPs for example) :

• foreman.domain – 10.0.0.1
• foreman-enc.domain – 10.0.0.2
• foreman-reports.domain – 10.0.0.3
• foreman-db.domain – 10.0.0.4
• memcached.domain – 10.0.0.5
• puppetmaster-1.domain – 10.0.0.6
• puppetmaster-2.domain – 10.0.0.7
• puppet-ca.domain – 10.0.0.8
• puppet-lb.domain – 10.0.0.9
• puppet-lb-passive.domain – 10.0.0.10

A virtual IP will be required for the high available load balancer : 10.0.0.11.

Puppet

Ensure you have Puppet installed on each node:

$ wget https://apt.puppetlabs.com/puppetlabs-release-precise.deb
$ sudo dpkg -i puppetlabs-release-precise.deb
$ sudo apt-get update && sudo apt-get install puppet

SSL (optional)

In my company, we got our own SSL certificates for our internal services.

If you want to setup this kind of architecture with your certificates, ensure you have the following files on all the nodes:

• /etc/ssl/domain/certs/ca.pem
• /etc/ssl/domain/certs/domain.pem
• /etc/ssl/domain/private_keys/domain.pem

Of course you can skip this step, you’ll also need to skip the configurations steps for the SSL setups in the next parts.

Setup the Foreman block

Setup for the foreman block introduction.

MySQL server

The installation of a MySQL server is beyond the scope of this topic, ensure you have a MySQL instance running.

Then create a database for foreman and the required users:

CREATE DATABASE foreman CHARACTER SET utf8;
CREATE USER 'foreman'@'foreman.domain';
GRANT ALL PRIVILEGES ON foreman.* TO 'foreman'@'foreman.domain' IDENTIFIED BY 'foreman_password';
CREATE USER 'foreman'@'foreman-enc.wit.prod';
GRANT ALL PRIVILEGES ON foreman.* TO 'foreman'@'foreman-enc.domain' IDENTIFIED BY 'foreman_password';
CREATE USER 'foreman'@'foreman-reports.wit.prod';
GRANT ALL PRIVILEGES ON foreman.* TO 'foreman'@'foreman-reports.domain' IDENTIFIED BY 'foreman_password';

Memcached server

It’s quite easy to setup a memcached node. The package is available in the Ubuntu repositories.

$ sudo apt-get install memcached

Once installed, update the following values in configuration file /etc/memcached.conf:

-m 128
-l 0.0.0.0

Then restart the service.

$ sudo service memcached restart

Foreman (main instance)

Setup the foreman-installer from the debian repository:

$ wget -q http://deb.theforeman.org/foreman.asc -O- | sudo apt-key add -
$ echo "deb http://deb.theforeman.org/ $(lsb_release -sc) 1.5" | sudo tee -a /etc/apt/sources.list.d/foreman.list
$ echo "deb http://deb.theforeman.org/ plugins 1.5" | sudo tee -a /etc/apt/sources.list.d/foreman.list
$ sudo apt-get update && sudo apt-get install foreman-installer

Setup the foreman instance via foreman-installer:

$ sudo foreman-installer \
--no-enable-puppet \
--no-enable-foreman-plugin-bootdisk \
--no-enable-foreman-proxy \
--foreman-db-adapter=mysql2 \
--foreman-db-database=foreman \
--foreman-db-host=foreman-db.domain \
--foreman-db-manage=true \
--foreman-db-username=foreman \
--foreman-db-password='foreman_password' \
--foreman-db-port=3306 \
--foreman-db-type=mysql \
--foreman-server-ssl-ca=/etc/ssl/domain/certs/ca.pem \
--foreman-server-ssl-chain=/etc/ssl/domain/certs/ca.pem \
--foreman-server-ssl-cert=/etc/ssl/domain/certs/domain.pem \
--foreman-server-ssl-key=/etc/ssl/domain/private_keys/domain.pem

Now that the service is installed, we need to setup the memcached plugin:

$ echo "gem 'foreman_memcache'" | sudo tee -a /usr/share/foreman/bundler.d/Gemfile.local.rb
$ sudo chown foreman:foreman /usr/share/foreman/bundler.d/Gemfile.local.rb
$ cd ~foreman
$ sudo -u foreman bundle update foreman_memcached

Then configure it by appending the following content in /etc/foreman/settings.yaml:

:memcache:
 :hosts:
 - memcached.domain:11211
 :options:
 :namespace: foreman
 :expires_in: 86400
 :compress: true

And restart the Apache service:

$ sudo service apache2 restart

You can now access your foreman instance via http://foreman.domain, access it and use the default credentials to login : admin/changeme.

Once logged in, you’ll need to retrieve some specific values from the settings, but also setup a few ones. Go to Administer > Settings > Auth and retrieve the following settings:

• oauth_consumer_key
• oauth_consumer_secret

These settings will be used in the following setup procedures, you’ll need to replace the values MY_CONSUMER_KEY and MY_CONSUMER_SECRET with the values you’ve retrieved.

We also need to setup the following settings:

• require_ssl_puppetmasters=false
• trusted_puppetmaster_hosts=[puppet.domain, puppet-lb.domain, puppet-lb-passive.domain]

SSL (optional):

Update the following settings in Administer > Settings > Provisioning:

• ssl_ca_file = /etc/ssl/domain/certs/ca.pem
•  ssl_certificate = /etc/ssl/domain/certs/domain.pem
•  ssl_priv_key = /etc/ssl/domain/private_keys/domain.pem

Foreman ENC & Reports

Apply these configurations on each nodes.

Setup the foreman-installer from the debian repository:

$ wget -q http://deb.theforeman.org/foreman.asc -O- | sudo apt-key add -
$ echo "deb http://deb.theforeman.org/ $(lsb_release -sc) 1.5" | sudo tee -a /etc/apt/sources.list.d/foreman.list
$ echo "deb http://deb.theforeman.org/ plugins 1.5" | sudo tee -a /etc/apt/sources.list.d/foreman.list
$ sudo apt-get update && sudo apt-get install foreman-installer

Setup the foreman instance via foreman-installer:

$ sudo foreman-installer \
--no-enable-puppet \
--no-enable-foreman-plugin-bootdisk \
--no-enable-foreman-proxy \
--foreman-db-adapter=mysql2 \
--foreman-db-database=foreman \
--foreman-db-host=foreman-db.domain \
--foreman-db-manage=false \
--foreman-db-username=foreman \
--foreman-db-password='foreman_password' \
--foreman-db-port=3306 \
--foreman-db-type=mysql \
--foreman-server-ssl-ca=/etc/ssl/domain/certs/ca.pem \
--foreman-server-ssl-chain=/etc/ssl/domain/certs/ca.pem \
--foreman-server-ssl-cert=/etc/ssl/domain/certs/domain.pem \
--foreman-server-ssl-key=/etc/ssl/domain/private_keys/domain.pem \
--foreman-oauth-consumer-key=MY_CONSUMER_KEY \
--foreman-oauth-consumer-secret=MY_CONSUMER_SECRET

After the service is installed, we need to setup the memcached plugin like previously:

$ echo "gem 'foreman_memcache'" | sudo tee -a /usr/share/foreman/bundler.d/Gemfile.local.rb
$ sudo chown foreman:foreman /usr/share/foreman/bundler.d/Gemfile.local.rb
$ cd ~foreman
$ sudo -u foreman bundle update foreman_memcached

Then configure it by appending the following content in /etc/foreman/settings.yaml:

:memcache:
 :hosts:
 - memcached.domain:11211
 :options:
 :namespace: foreman
 :expires_in: 86400
 :compress: true

And restart the Apache service:

$ sudo service apache2 restart

Setup the Puppet block

Setup for the foreman block introduction.

Puppet CA

Setup the foreman-installer from the debian repository:

$ wget -q http://deb.theforeman.org/foreman.asc -O- | sudo apt-key add -
$ echo "deb http://deb.theforeman.org/ $(lsb_release -sc) 1.5" | sudo tee -a /etc/apt/sources.list.d/foreman.list
$ echo "deb http://deb.theforeman.org/ plugins 1.5" | sudo tee -a /etc/apt/sources.list.d/foreman.list
$ sudo apt-get update && sudo apt-get install foreman-installer

Clear your Puppet SSL folder before the installation:

$ sudo rm -rf /var/lib/puppet/ssl/*

Setup the puppetmaster instance via foreman-installer:

$ sudo foreman-installer \
--no-enable-foreman-plugin-bootdisk \
--no-enable-foreman-plugin-setup \
--no-enable-foreman \
--foreman-proxy-foreman-base-url=https://foreman.domain \
--foreman-proxy-tftp=false \
--foreman-proxy-ssl-ca=/etc/ssl/domain/certs/ca.pem \
--foreman-proxy-ssl-cert=/etc/ssl/domain/certs/wit.prod.pem \
--foreman-proxy-ssl-key=/etc/ssl/domain/private_keys/wit.prod.pem \
--foreman-proxy-oauth-consumer-key=MY_CONSUMER_KEY \
--foreman-proxy-oauth-consumer-secret=MY_CONSUMER_SECRET \

Once the service is installed, you’ll need to generate certificates for your puppet masters:

$ sudo puppet cert generate puppetmaster-1.domain --dns_alt_names=puppet,puppet.domain,puppetmaster-1.domain
$ sudo puppet cert generate puppetmaster-2.domain --dns_alt_names=puppet,puppet.domain,puppetmaster-2.domain

This will generate the following files:

• /var/lib/puppet/ssl/certs/puppetmaster-1.domain.pem
• /var/lib/puppet/ssl/certs/puppetmaster-2.domain.pem
• /var/lib/puppet/ssl/private_keys/puppetmaster-1.domain.pem
• /var/lib/puppet/ssl/private_keys/puppetmaster-2.domain.pem

Puppetmasters 1 & 2

Retrieve the certificates previously generated and put them in the same folder on the puppetmasters nodes:

• /var/lib/puppet/ssl/certs/
• /var/lib/puppet/ssl/private_keys/

Ensure the permissions are OK:

$ chown -R puppet:puppet /var/lib/puppet/ssl

Setup the foreman-installer from the debian repository:

$ wget -q http://deb.theforeman.org/foreman.asc -O- | sudo apt-key add -
$ echo "deb http://deb.theforeman.org/ $(lsb_release -sc) 1.5" | sudo tee -a /etc/apt/sources.list.d/foreman.list
$ echo "deb http://deb.theforeman.org/ plugins 1.5" | sudo tee -a /etc/apt/sources.list.d/foreman.list
$ sudo apt-get update && sudo apt-get install foreman-installer

Setup the puppetmaster instances using the foreman-installer:

$ sudo foreman-installer \
--no-enable-foreman-plugin-bootdisk \
--no-enable-foreman-plugin-setup \
--no-enable-foreman \
--puppet-ca-server=puppet.domain \
--puppet-server-ca=false \
--puppet-server-foreman-ssl-ca=/etc/ssl/domain/certs/ca.pem \
--puppet-server-foreman-ssl-cert=/etc/ssl/domain/certs/domain.pem \
--puppet-server-foreman-ssl-key=/etc/ssl/domain/private_keys/domain.pem \
--puppet-server-foreman-url=https://foreman.domain \
--foreman-proxy-foreman-base-url=https://foreman.domain \
--foreman-proxy-tftp=false \
--foreman-proxy-puppetca=false \
--foreman-proxy-ssl-ca=/etc/ssl/domain/certs/ca.pem \
--foreman-proxy-ssl-cert=/etc/ssl/domain/certs/domain.pem \
--foreman-proxy-ssl-key=/etc/ssl/domain/private_keys/domain.pem \
--foreman-proxy-oauth-consumer-key=MY_CONSUMER_KEY \
--foreman-proxy-oauth-consumer-secret=MY_CONSUMER_SECRET \

Now edit your Puppet configuration under /etc/puppet/puppet.conf and add the following lines under the appropriate sections:

[main]
ca_port = 8141
[master]
dns_alt_names = puppet,puppet.domain,puppetmaster-X.domain

Note: replace puppetmaster-X.domain by the fqdn of the instance you setup (either puppetmaster-1.domain or puppetmaster-2.domain).

After that, we’ll need to tell the puppetmaster to use the foreman-enc node as the ENC. Edit the foreman URL in the file /etc/puppet/node.rb to point foreman-enc.domain.

Next, we’ll need to tell the puppetmaster to use the foreman-reports node for the reports. So edit the foreman URL in the file /usr/lib/ruby/vendor_ruby/puppet/reports/foreman.rb to point foreman-reports.domain.

Restart apache and you’re done with the puppetmaster setup :

$ sudo service apache2 restart

HAProxy

You’ll need to setup haproxy and keepalived on the following nodes:

• puppet-lb.domain
• puppet-lb-passive.domain

Install haproxy and keepalived on both nodes:

$ sudo apt-get install haproxy keepalived

Configure keepalived

On the active node (puppet-lb.domain), create a file /etc/keepalived/keepalived.conf with the following content:

global_defs {
  lvs_id lb_internal_KA
}

vrrp_script check_haproxy {
  script "killall -0 haproxy"
  interval 2
  weight 2
}

vrrp_instance VIP_01 {
  state MASTER
  interface eth0
  virtual_router_id 57
  priority 101

  virtual_ipaddress {
    10.0.0.11
  }

  track_script {
    check_haproxy
  }
}

On the passive node (puppet-lb-passive.domain), the content of this file will be different:

global_defs {
  lvs_id lb_internal_KA_passive
}

vrrp_script check_haproxy {
  script "killall -0 haproxy"
  interval 2
  weight 2
}

vrrp_instance VIP_01 {
  state SLAVE
  interface eth0
  virtual_router_id 57
  priority 100

  virtual_ipaddress {
    10.0.0.11
  }

  track_script {
    check_haproxy
  }
}

NOTE: The directive virtual_router_id NEEDS to have a unique value, if another keepalived cluster exists on the same network/vlan an error will occur if you set the same ID.

Configure haproxy

Here is a sample configuration for haproxy, use it on both nodes (puppet-lb.domain & puppet-lb-passive.domain) :

global
        log 127.0.0.1   local1
        maxconn 4096
        user haproxy
        group haproxy
        daemon

defaults
        log     global
        option  dontlognull
        retries 3
        option redispatch
        maxconn 2000
        contimeout      300000
        clitimeout      300000
        srvtimeout      300000

listen  stats *:1936
        mode http
        stats enable
        stats hide-version
        stats realm Haproxy\ Statistics
        stats uri /


listen puppet    *:8140
       mode      tcp
       option    tcplog
       option    ssl-hello-chk
       server    puppetmaster-1        puppetmaster-1.domain:8140     check inter 5000 fall 3 rise 2
       server    puppetmaster-2        puppetmaster-2.domain:8140     check inter 5000 fall 3 rise 2

listen  puppetca  *:8141
        mode      tcp
        option    tcplog
        option    ssl-hello-chk
        option    abortonclose
        server    puppetca-1            puppet-ca-1.domain:8140              check inter 5000 fall 3 rise 2

Now, restart your haproxy service :

$ sudo service haproxy restart

And that’s it ! Your Foreman/Puppet model is ready !

The node configuration

You will need to add the following properties under the [agent] section in your Puppet node configuration file in /etc/puppet/puppet.conf:

[agent]
    report        = true
    masterport    = 8140
    server        = puppet.domain
    ca_port       = 8141

Suggestions and evolutions

Modules synchronisation between the Puppet masters

I recommend the use of r10k to easily synchronize your modules between the Puppet masters.

More information here: https://github.com/puppetlabs/r10k

The main SPOF: the Foreman ENC

Actually, the main SPOF in this architecture is the Foreman ENC box. I guess you could add an Apache service in front of the Foreman services to achieve a passive/active Foreman ENC. You can also update the URL of the ENC on the Puppet masters with the URL of the Foreman reports box while trying to diagnosis your problem with the ENC.

High availability on the database

Of course, the other SPOF in this architecture is MySQL. There is many existing solutions for MySQL high availability, you could use another HAProxy service in front of your MySQL instances for an active/passive cluster and setup a replication architecture.

High availability on the Puppet CA

About the Puppet CA, I do not consider it as a SPOF because if the service goes down, the model will still be working. You’ll just not be able to add new nodes.

You could have a highly available service by adding another Puppet CA to the load balancer (a passive CA for example) but you would need a way to synchronize your certificates between your Puppet CA servers. Maybe using rsync or a NFS mount point.

Special thanks and feedbacks

Feel free to ask any questions or comment on this blog, or you can catch me on Freenode IRC, @deviantony on channel #theforeman.

Finally I’d like to thanks the people who helped me to setup this architecture: @elobato, @dominic and @gwmngilfen from the channel #theforeman. Do not hesitate to ask any questions over there, the channel is pretty active and the Foreman team members are really reactive !

Advertisement