Apply Software Development Practice to Application Configuration

Ops teams should always strive towards implementing the best application configuration strategy and infrastructure as code solution that will work best for each scenario. However, Puppet, Chef, Ansible, and other similar products may not be the right solution for every project and team. Depending on the DevOps maturity of the team, flexible delivery capabilities needed, feature plugin issues, or off-release work cycles, building a custom solution could be the steps needed get a team from manually editing configurations in production to a fully automated continuous integration pipeline. We want to avoid the scenario where “when you’ve got a hammer, every problem looks like a nail”.

In my example below, I’m going to assume a few things to get you to this point:

Hand editing configurations in production is like using a rock to drive a nail into a board. If you’ve got the right kind of rock and enough focus and some luck, you’ll be able to drive that thing in there. So what does the equivalent look like when you’re dealing with a real life production application? To me, that’s hand editing configuration right in production to support an application. So I’ve got Jenkins and apache running default out of the box, with Jenkins on port 8080. In the default instance, I can drop any .conf file into /etc/httpd/conf.d/ and it will be picked up by apache. Here’s “manual-config-example.conf”

# /etc/httpd/conf.d/manual-config-example.conf
# WHOOPSIES

ProxyPass /jenkins http://localhost:8080/jenknis
ProxyPassReverse /jenkins http://localhost:8080/jenknissss
ProxyRequests Off

# WAIT, WHICH PORT?

Order deny,allow
Allow from all

So we have lovely little configuration that will not cause any errors on apache, but clearly will not work for a couple of different reasons. Instead of continuing to hand edit these changes and testing to see if each one works, let’s check this into a repo and start thinking about a process that allows us to move towards a development pipeline for this configuration. Instead of a rock, now you’re starting to use a hammer to drive that nail in. So I did the following:

git clone git@github.com:tomcudd/apache-config-examples.git
cp /etc/httpd/conf.d/manual-config-example.conf apache-config-examples/
git add .
git commit -am "check in my initial configuration that's not so great"
git push

Then we just start going through the process of modifying the configuration file in source control to track all of our changes. See the history for what I did at https://github.com/tomcudd/apache-config-examples/commits/master. From that point forward, I started building out, stage by stage, a Jenkins deployment pipeline with various stages. You’re moving from a hammer to a nail gun to drive these in. You can find the code at https://github.com/tomcudd/jenkins-pipeline-scripts and listed below here:

node {
stage('git clone') {
git url: 'git@github.com:tomcudd/apache-config-examples.git'
}

stage(‘move configs around’) {
sh ”’#!/bin/bash
if [ -f /etc/httpd/conf.d/manual-config-example.conf ]; then
sudo mv /etc/httpd/conf.d/manual-config-example.conf /etc/httpd/conf.d/manual-config-example.bak
fi
sudo cp *.conf /etc/httpd/conf.d/
”’
}

stage(‘test config’) {
sh ”’#!/bin/bash
/usr/sbin/apachectl configtest
”’
}

stage(‘restart apache’) {
sh ”’#!/bin/bash
sudo /bin/systemctl restart httpd.service
”’
}

stage(‘test apache’) {
sh ”’#!/bin/bash
wget -O- http://centos7/jenkins
”’
}
}

The goal here was to move the manual configuration out of the way, “deploy” the new configuration, do a syntax check with apachectl, restart the service, and then follow it up with a “unit test” that ensures our restarting the service actually allows us to hit the local URL for the instance we were trying to make accessible through the apache proxy to Jenkins. Whereas you were using a nail gun, now this is more like an assembly line with a robotic arm doing all the work for you.