Revision control

Sometimes you need to make changes to these files, but making a mistake can lead to trouble varying from a program that does not work correctly to a system that will not boot! So if you make changes to these configuration files, you should always keep a copy of the original. Now, this is something that is easy to forget. That is why I keep my configuration files under revision control.

This is not meant as en exhaustive treatise on revision control. Enough of that is available on the internet. But a short introduction is in order to make you understand why it is worthwhile to use this.

A revision control system will record different versions of a file. This action is called ‘checking in’ or ‘committing’ a file. So if you edit a file, and you discover that you have made a mistake, you can always return to any of the previous version that you have saved. Even better, these systems can also show you the difference the current file and previous versions. A check-in or commit is usually accompanied by a short message that you type in describing the changes. You can list these messages to see how a file has developed over time.

My revision control system of choice is git, for various reasons that I shall not go into. But for configuration files something simple like rcs would do as well. For FreeBSD users, the latter is part of the FreeBSD base system.

Using a separate archive location

System configuration files are usually stored in /etc, or under /var (or, on FreeBSD at least, also /usr/local/etc for applications that are not part of the base system but in ports). Usually only the root user can edit files in these directories. But since root has complete access to the whole system, a mistaken command there can have dire consequences.

For that reason I generally use another location to keep edited configuration files, a directory in my home-directory that I call setup. Since I have multiple machines running FreeBSD, each machine gets its own subdirectory. In my case, this subdirectory also serves as the so-called ‘repository’ for the revision control system.

But some revision control systems (like cvs or subversion) have separate repositories, e.g. stored in databases. Personally I think that is overkill in this case.

Workflow

When I started using a separate setup directory, the following workflow grew

1. Edit a configuration file in my private directory.
2. Copy it to its proper location.
3. Test the changes. If problems occur, go back to step 1.
4. Commit the changes to the revision control system.
5. Copy the committed files to their proper location.

The second step was something that needed automation; I had to keep track of where every file was supposed to be and what ownership and permissions were needed. Some files even need a program to be run after a new version is installed.

Sometimes when I upgrade my OS to a newer version, that introduces changes as well. So I needed a way to check if actually installed files differed from the version I had in the setup directory.

Automation with perl scripts

To accomplish both things, I wrote a couple of perl scripts that could check which configuration files had changed, and could install them in the right places. These scripts are located in the setup directory. Both of these scripts use a data file that tells them the files that need to be checked, what their permissions should be, their proper location and optionally the commands that need to be executed after they have been installed. Given the above mentioned differences between system and user files, two different file lists are used, depending on if the scripts are called by a normal user or by root. Below is an excerpt of the list for root, filelist.root. The other file should be called filelist.USER, where USER is your username.

etc/hosts          640 /etc/hosts
etc/locate.rc      640 /etc/locate.rc
etc/login.access   640 /etc/login.access
etc/login.conf     640 /etc/login.conf    cap_mkdb /etc/login.conf
etc/make.conf      644 /etc/make.conf
etc/master.passwd  600 /etc/master.passwd pwd_mkdb -p /etc/master.passwd
etc/mergemaster.rc 644 /etc/mergemaster.rc
etc/motd           644 /etc/motd
etc/newsyslog.conf 640 /etc/newsyslog.conf
etc/ntp.conf       644 /etc/ntp.conf      /etc/rc.d/ntpd restart
etc/periodic.conf  644 /etc/periodic.conf
etc/pf.conf        640 /etc/pf.conf       /etc/rc.d/pf reload

The configuration files are listed one on every line. The first column is the path of the file relative to the setup directory. The second column is the permissions that the file should have when installed at the target location. The third column is the target location where the file should be installed. Any further text on a line is used as commands to be executed after the files have been installed.

This works pretty well, except for my emacs configuration file .emacs.elc file, which needs to be byte-compiled with emacs before installation. So I still have a separate install script for that. I’ve toyed with the idea to extend the scripts, but in my opinion the efforts outweigh the gains.

check.pl

The first script is called check.pl, and checks for differences between the files in the setup directory and the installed files. The check is done by comparing MD5 checksums of the two files.

The normal mode of operation for check.pl is to just list the files that differ. If the -d (for ‘details’) flag is given when check.pl is invoked, it also prints a unified diff between the two files. The -l (for ‘list’) flag lists all files and whether they are changed or not. The -h option merely produces a short help message and then exits the script.

#!/usr/bin/perl
# Check for changes in files according to instructions in 'filelist.$USER'.
# Time-stamp: <2010-08-13 15:20:51 rsmith>
# $Id: 2088790e9bde79f4476a003adc6c9b58dc8f1aa5 $
#
# To the extent possible under law, Roland Smith has waived all copyright and
# related or neighboring rights to check.pl. This work is published from
# Netherlands. See https://creativecommons.org/publicdomain/zero/1.0/

use Getopt::Std;

getopts('ldh');

if ($opt_h == 1) {
    print "Usage: ./check.pl [-hld]\n";
    print "-l\t(long); lists all files, not just the different ones.\n";
    print "-d\t(diff); Print the diff(1) output for different files.\n";
    exit 0;
}

$name = `id -u -n`;

open(RF, "filelist.$name") || die "Cannot open list file 'filelist.$name'.";

while(<RF>) {
    chomp;
    if (/^#/) {next}; # skip comment lines.
    if (/^[ \t]*$/) {next}; # skip empty lines.
    ($src,$perm,$dest) = split;
# For debugging.
#    printf("src: \'%s\', perm: %d, dest: \'%s\'\n", $src,$perm,$dest);
    if (-r $dest) {
        $msrc = `md5 -q $src`;
        $mdest = `md5 -q $dest`;
        if ($msrc ne $mdest) {
            printf "\e[31m%s differs from %s\n\e[0m", $src, $dest;
            if ($opt_d == 1) {
                system "diff", "-u", $dest, $src;
            }
        } elsif ($opt_l == 1) {
            printf "\e[32m%s matches %s\n\e[0m", $src, $dest;
        }
    } else {
        printf "\e[41m\e[30m%s cannot be read.\n\e[0m", $dest;
    }
}

close(RF);

Running this script tells me which files differ from the ones in the repository. I can then take appropriate action; merge any changes into the repository, or call install.pl to restore the version from the repository.

install.pl

After using check.pl to check if there are no system changes to be merged into the repository, the following install.pl is used to carry out the installation. It only installs files that are different, not just everything. If post-install commands are provided, they are run;

#!/usr/bin/perl
# Install files according to instructions in 'filelist.$USER'.
# Time-stamp: <2010-08-13 15:19:41 rsmith>
# $Id: 38a7b30e9198dd1c548ac90f7a554e8feaa1a682 $
#
# To the extent possible under law, Roland Smith has waived all copyright and
# related or neighboring rights to install.pl. This work is published from
# Netherlands. See https://creativecommons.org/publicdomain/zero/1.0/

$name = `id -u -n`;

open(RF, "filelist.$name") || die "Cannot open list file 'filelist.$name'.";

while(<RF>) {
    chomp;
    if (/^#/) {next}; # Skip comment lines.
    if (/^[ \t]*$/) {next}; # skip empty lines.
    @cmds = split;
    $src = shift @cmds;
    $perm = shift @cmds;
    $dest = shift @cmds;
    # Skip identical files.
    if (-e $dest) {
        $msrc = `md5 -q $src`;
        $mdest = `md5 -q $dest`;
        if ($msrc eq $mdest) {next};
    }
    # Create directories if necessary
    $tdir = `dirname $dest`;
    chomp $tdir;
    if (! -d $tdir) {
        print "\e[31m";
        $rv = system "install", "-d", $tdir;
        $rv>>8;
        if ($rv == 0) {
            printf "\e[32mCreated %s.\n\e[39m", $tdir;
        } else {print "\e[39m";}
    }
    # Install the file
    print "\e[31m";
    $rv = system "install", "-p", "-m", $perm, $src, $dest;
    $rv>>8;
    if ($rv == 0) {
        printf "\e[32mInstalled %s as %s.\n\e[39m", $src, $dest;
        # Execute post-install commands.
        if (@cmds) {
            $rv = system @cmds;
            $rv>>8;
            if ($rv != 0) {
                printf "\e[31mPost-install commands for %s failed.\n\e[39m",
                $src;
            }
        }
    } else {print "\e[39";}
}

close(RF);

Improved workflow

With the scripts and the revision control, my workflow is now any of these;

1. Run ./check.pl -d to see if any system files have changed e.g. after the base system or any of the ports have been updated.
2. If so, merge changes into my setup repository and commit them.

Or

1. Make my changes by editing the files in my repository.
2. Run ./install.pl to roll out the changes.
3. Test the changes. In case of trouble, go back to 1.
4. Commit the changes to the repository.

At any stage it is now easy for me to check what has changed, and why.

Example

Suppose I want my system to be a gateway. One way of doing this (on FreeBSD) is to set gateway_enable="YES" in /etc/rc.conf. So I go to my local repository of config files, and I edit rc.conf. After editing, I review the changes that I’ve made by comparing them to the version that is in the git revision control system;

> git diff
diff --git a/etc/rc.conf b/etc/rc.conf

index fa025c1..9c523ad 100644
--- a/etc/rc.conf
+++ b/etc/rc.conf
@@ -1,6 +1,6 @@
 # file: /etc/rc.conf  - System configuration information
 # host: slackbox.erewhon.net
-# Time-stamp: <2010-08-15 23:01:28 rsmith>
+# Time-stamp: <2010-08-19 22:36:43 rsmith>
 # $Id$

 hostname="slackbox.erewhon.net"
@@ -16,6 +16,7 @@ devfs_system_ruleset="slackbox_usb"
 ifconfig_lo0="inet 127.0.0.1"
 ifconfig_age0="inet 10.0.0.150/24 polling media 100baseTX mediaopt full-duplex"
 ifconfig_rl0="inet 192.168.0.1/24 polling media 100baseTX mediaopt full-duplex"
+gateway_enable="YES"
 router_enable="YES"
 defaultrouter="10.0.0.138"
 tcp_extensions="NO"

This looks OK. The Time-stamp was changed by my editor (emacs) when I saved the file. Since this is a really simple change, I can commit it without testing.

> git commit -am "Enable gateway in rc.conf"
Checking out etc/rc.conf to update Id.
[master 485ba6b] Enable gateway in rc.conf
 1 files changed, 2 insertions(+), 1 deletions(-)

With the command git log I can see a short summary of my changes;

commit 485ba6b8b208311fb8e6a4debe8879088cf28567
Author: Roland Smith <rsmith@xs4all.nl>
Date:   Thu Aug 19 22:47:36 2010 +0200

    Enable gateway in rc.conf

commit 93d13f6aacbefc872ca980ac228064c2d2894038
Author: Roland Smith <rsmith@xs4all.nl>
Date:   Wed Aug 18 23:54:16 2010 +0200

    Fix typo in dhcpd.conf.
...

Now I go and check the difference with the real version in /etc by running check.pl

# ./check.pl -d
etc/rc.conf differs from /etc/rc.conf
--- /etc/rc.conf    2010-08-16 00:00:19.000000000 +0200
+++ etc/rc.conf     2010-08-19 22:47:36.000000000 +0200
@@ -1,7 +1,7 @@
 # file: /etc/rc.conf  - System configuration information
 # host: slackbox.erewhon.net
-# Time-stamp: <2010-08-15 23:01:28 rsmith>
-# $Id: fa025c16642a4aed686603a3824644fbdaf03f42 $
+# Time-stamp: <2010-08-19 22:36:43 rsmith>
+# $Id: 9c523adb9d067f37393efc540096b06b1c816f8c $

 hostname="slackbox.erewhon.net"

@@ -16,6 +16,7 @@
 ifconfig_lo0="inet 127.0.0.1"
 ifconfig_age0="inet 10.0.0.150/24 polling media 100baseTX mediaopt full-duplex"
 ifconfig_rl0="inet 192.168.0.1/24 polling media 100baseTX mediaopt full-duplex"
+gateway_enable="YES"
 router_enable="YES"
 defaultrouter="10.0.0.138"
 tcp_extensions="NO"

This is also the correct change. The $Id was updated by git when I chacked in the file. Now I can install the file in its proper location;

# ./install.pl
Installed etc/rc.conf as /etc/rc.conf.

That’s all folks!