Roland's UNIX miscellanea page

Introduction

This page is a place where I gather worthwhile information that doesn’t really fit anywhere else. Large or important pieces will eventually get their own page. To keep things simple, this is just a newest-first list of items on a single page. No blogging engine or database in sight. Comments? Just send me an e-mail. :-) If an item is specific to FreeBSD, it is marked as such.

Conventions

Configuration data, shell commands or shell scripts are shown as preformatted text in green on a light-gray background. Shell commands start with a greater-than sign (‘>’) for commands that should be started as a normal user or a hash mark (‘#’) for commands that should be run as root. Sometimes configuration files also contain hash marks to start a comment. I trust that it is sufficiently clear from context that these are not commands. :-)

2010

Converting PostScript and PDF images to SVG format

2010–08–28

As a long-time UNIX user I’m used to programs generating PostScript output, and I’m fluent enough in that language to produce graphics in it myself. Unfortunately PostScript is hard to display e.g. on the Web. Luckily another vector format, SVG has arisen that is supported by modern browsers.

To convert PostScript images into SVG while keeping the figures’ bounds intact, I use the following programs;

pstopdf

A Ruby script from the TeXLive distribution. Converts the PostScript picture to PDF format while keeping the bounding box intact. Note that this is not one of the commercial programs that go by the same name.

pdf2svg

To do the actual conversion to SVG format.

Note that the ps2pdf converter that comes with Ghostscript doesn’t work well in this case as it renders the PostScript file as a full page. For including SVG pictures in webpages it is better to use the <embed> tag than to use the <img> tag, at least in Mozilla Firefox. Only the embed tag actually renders the picture.

It is assumed that the PostScript file has a correct BoundingBox set. If not, the proper DSC comment for a file foo.ps can be generated with the command

gs -q -sDEVICE=bbox foo.ps -c quit 

The resulting BoundingBox line should be added or replaced on the second line of foo.ps. The workflow to convert it to SVG is;

pstopdf foo.ps
pdf2svg foo.pdf foo.svg
rm -f foo.pdf

Writing speed on FreeBSD 8.1-RELEASE amd64

2010–08–22

Recently the question came up how fast the write speed of a disk drive on FreeBSD should be. I decided to test the hardware that I have at my disposal.

The test is writing approximately 10 GB data from /dev/zero to /tmp/foo;

dd if=/dev/zero of=/tmp/foo bs=10M count=1000

The tests are done on otherwise idle machines. In both cases, the partition layout is similar.

desktop

Partition layout;

/dev/ad4s1a        484M    102M    343M    23%    /
/dev/ad4s1g.eli    373G    140G    204G    41%    /home
/dev/ad4s1e         48G     38K     45G     0%    /tmp
/dev/ad4s1f         19G    6.4G     11G    36%    /usr
/dev/ad4s1d        1.9G    254M    1.5G    14%    /var

Write caching enabled

Output of ‘atacontrol cap ad4’

Protocol              SATA revision 2.x
device model          WDC WD5001ABYS-01YNA0
serial number         WD-WCAS87154115
firmware revision     59.01D01
cylinders             16383
heads                 16
sectors/track         63
lba supported         268435455 sectors
lba48 supported       976773168 sectors
dma supported
overlap not supported

Feature                      Support  Enable    Value           Vendor
write cache                    yes      yes
read ahead                     yes      yes
Native Command Queuing (NCQ)   yes       -      31/0x1F
Tagged Command Queuing (TCQ)   no       no      31/0x1F
SMART                          yes      yes
microcode download             yes      yes
security                       yes      no
power management               yes      yes
advanced power management      no       no      0/0x00
automatic acoustic management  yes      yes     254/0xFE        128/0x80

Test results;

10485760000 bytes transferred in 138.304953 secs (75816229 bytes/sec)
10485760000 bytes transferred in 139.125501 secs (75369073 bytes/sec)
10485760000 bytes transferred in 136.149871 secs (77016305 bytes/sec)

Write caching disabled

The write caching was disabled by putting hw.ata.wc="0" in /boot/loader.conf and rebooting the machine.

Protocol              SATA revision 2.x
device model          WDC WD5001ABYS-01YNA0
serial number         WD-WCAS87154115
...
Feature                      Support  Enable    Value           Vendor
write cache                    yes      no

Test results;

10485760000 bytes transferred in 811.677303 secs (12918632 bytes/sec)
10485760000 bytes transferred in 811.628748 secs (12919404 bytes/sec)

laptop

Partition layout;

/dev/ada0s1a        496M     99M    357M    22%    /
/dev/ada0s1g.eli    240G     91G    130G    41%    /home
/dev/ada0s1e         19G     10K     18G     0%    /tmp
/dev/ada0s1f         19G    4.6G     13G    26%    /usr
/dev/ada0s1d        1.9G     35M    1.7G     2%    /var

Output of ‘camcontrol identify ada0’

pass0: <ST9320423AS 0002SDM1> ATA-8 SATA 2.x device
pass0: 300.000MB/s transfers (SATA 2.x, UDMA6, PIO 8192bytes)

protocol              ATA/ATAPI-8 SATA 2.x
device model          ST9320423AS
firmware revision     0002SDM1
serial number         5VH0322Y
WWN                   5000c50017f4f8d9
cylinders             16383
heads                 16
sectors/track         63
sector size           logical 512, physical 512, offset 0
LBA supported         268435455 sectors
LBA48 supported       625142448 sectors
PIO supported         PIO4
DMA supported         WDMA2 UDMA6 
media RPM             7200

Feature                      Support  Enable    Value           Vendor
read ahead                     yes      yes
write cache                    yes      yes
flush cache                    yes      yes
overlap                        no
Tagged Command Queuing (TCQ)   no       no
Native Command Queuing (NCQ)   yes              32 tags
SMART                          yes      yes
microcode download             yes      yes
security                       yes      no
power management               yes      yes
advanced power management      yes      yes     32896/0x8080
automatic acoustic management  yes      yes     208/0xD0        208/0xD0
media status notification      no       no
power-up in Standby            no       no
write-read-verify              yes      no      0/0x0
unload                         yes      yes
free-fall                      no       no
data set management (TRIM)     no

Test results;

10485760000 bytes transferred in 122.625997 secs (85510090 bytes/sec)
10485760000 bytes transferred in 126.081170 secs (83166741 bytes/sec)
10485760000 bytes transferred in 126.101845 secs (83153105 bytes/sec)

Conclusion

The effect of disabling the write cache is quite big. Sustained writes go from 75 MiB/s to 13 MiB/s on the desktop. Still, in daily use this doesn’t really show. And data integrity means more to me than speed.

Installing Plone via the unified installer

2010–08–12

After having installed the plone CMS from ports and playing with it for a while, I found out that extensions can only really be added via the so-called buildout mechanism, which was kind of hard to understand if you don’t use it in the first place! So I decided to ditch the install via ports, and go for a buildout in my server jail. The first step was to download the latest unified installer for the latest stable release (at the time of wrting, Plone-3.3.5-UnifiedInstaller.tgz) from www.plone.org.

For reasons mentioned in the following links, I decided to install as root, and to use the ZEO client/server setup. I did decide to use the libz library that is part of the base system, and the jpeg port that I already have installed instead of having the installer build separate copies.

The installation was pretty easy;

server# cd tmp
server# tar xf Plone-3.3.5-UnifiedInstaller.tgz
server# cd Plone-3.3.5-UnifiedInstaller/
server# ./install.sh zeo --libz=no --libjpeg=no
Plone successfully installed at /usr/local/Plone
See /usr/local/Plone/zeocluster/README.txt
for startup instructions

Use the account information below to log into the Zope Management Interface
The account has full 'Manager' privileges.

  Username: admin
  Password: nAjPbZge
...

It is important to note this password. You’ll need it to log in. As per instructions, I read the /usr/local/Plone/zeocluster/README.txt file. The first thing it advised me was to check /usr/local/Plone/zeocluster/buildout.cfg to see if I wanted to change anything. Since my server jail runs on a local IP address (192.168.0.100), I edited buildout.cfg to change the zeo-address under the [buildout] heading from 127.0.0.1:8100 to 192.168.0.100:8100. Having read the documentation, I decided to also add the Products.CacheSetup [egg][] get CacheFu to make plone faster. These changes made it necessary to run ./bin/buildout, which automatically downloaded any needed components.

The installation process created a plone user and group on my system. Since the ZEO server wanted to run under its own user, I added a zeo user as well. I changed the entries in /etc/master.password so that they looked like this:

plone:*:1003:1003::0:0:Plone Admin:/var/empty:/usr/sbin/nologin
zeo:*:1004:1004::0:0:Zeo Server:/var/empty:/usr/sbin/nologin

(As always after making changes to /etc/master.password, it is necessary to run the command pwd_mkdb -p /etc/master.passwd, see the pwd_mkdb manual page.)

It turned out the server (running as user zeo didn’t have access to the directory /usr/local/Plone/zeocluster/var/zeoserver, where it wanted to put a PID-file and a logfile, because that was owned by root. Ditto for the /usr/local/Plone/zeocluster/var/filestorage directory. So I fixed that;

server# chown zeo /usr/local/Plone/zeocluster/var/zeoserver
server# chown zeo /usr/local/Plone/zeocluster/var/filestorage

Now I could start the server for the first time;

server# /usr/local/Plone/zeocluster/bin/plonectl start
zeoserver: . daemon process started, pid=39058
This is the first start of this instance.
Creating Data.fs and a Plone site.
We only need to do this once, but it takes some time.
Creating Plone site at /Plone in ZODB...
Installed Plone standard content
Finished adding Plone site
client1: . daemon process started, pid=39062
client2: . daemon process started, pid=39066

The site creation is only run once, so at the next start, you will only see the notifications of the server and clients startup.

To go to the site, I pointed my browser to http://server.erewhon.net:8080/Plone/.

Stopping the server goes as follows;

server# /usr/local/Plone/zeocluster/bin/plonectl stop
zeoserver: . . . . . . . . . . daemon process stopped
client1: . . . . . daemon process stopped
client2: . . . . daemon process stopped

I’ll be experimenting with building a site for documentation next.

Using nullfs and unionfs for the ports tree in a jail

2010–08–10

When running a virtual server in a jail, one would like to avoid replication of files as much as practical. One candidate for sharing would be the ports tree in /usr/ports. One could just make a symbolic link from /usr/ports in the jail’s tree to the one on the host. But what if multiple jails were to be run sharing the same /usr/ports? You could end up with multiple jails trying to rebuild the same port at the same time. That would probably not work as intended.

So we use a combination of a nullfs and a unionfs to create copies of /usr/ports for each jail without writing in ports tree on the host.

The mount_nullfs command is used to create a virtual copy of /usr/ports in the filesystem of the jail:

slackbox# cd /usr/local/var/jails/192.168.0.100/usr
slackbox# mount_nullfs /usr/ports/ ports/

Now we create an empty directory in the filesystem tree of the jail, and use that as an overlay for the virtual ports tree using mount_unionfs;

slackbox# mkdir tmp/foo
slackbox# mount_unionfs -o noatime tmp/foo ports/

The affect of this is that whenever a file is written in the overlayed copy of the ports tree, that file is actually written in the tree under /usr/local/var/jails/192.168.0.100/usr/tmp/foo! So you can compile ports in the ports tree in the jail without writing files in the host’s ports tree!

To make use of this, the four commands above should be run before starting a jail. The start-up sequence of the jail now becomes;

slackbox# cd /usr/local/var/jails/192.168.0.100/usr
slackbox# mount_nullfs /usr/ports/ ports/
slackbox# mkdir tmp/foo
slackbox# mount_unionfs -o noatime tmp/foo ports/
slackbox# /etc/rc.d/jail onestart server
Configuring jails:.
Starting jails: server.erewhon.net.

When you are finished with the jail, you should do the following to take everything down;

slackbox# /etc/rc.d/jail onestop server
slackbox# umount /usr/local/var/jails/192.168.0.100/usr/ports
slackbox# umount /usr/local/var/jails/192.168.0.100/usr/ports

And yes, the umount command does need to be run twice: once for the unionfs, and once for the nullfs!

To save space, one can now remove the contents of the unionfs;

slackbox# cd /usr/local/var/jails/192.168.0.100/usr
slackbox# rm -rf tmp/foo/*

Updating the ports in the jail

Start the jail with the nullfs and unionfs mounts in place.

slackbox# cd /usr/local/var/jails/192.168.0.100/usr
slackbox# mkdir tmp/foo
slackbox# mount_nullfs /usr/ports/ ports/
slackbox# mount_unionfs -o noatime tmp/foo ports/
slackbox# /etc/rc.d/jail onestart server
Configuring jails:.
Starting jails: server.erewhon.net.
slackbox# exit
slackbox:~> ssh server
server:~> su
server# portmaster -a -B -d

Afterwards, close the jail as usual.

Perl program to add a header to my webpages

2010–07–30

To give all of my pages a consistent look, I used a standard header on all of my pages, displaying the page title and subtitle. Unfortunately this did not work well with the table-of contents that multimarkdown can generate. This is not surprising, since this ToC is placed at the top of the body text, before my header.

After looking at the xslt files and deciding I really didn’t want to hack XSLT, I decided to whip up a Perl script to insert the needed header. Perl is still my favorite language for munging text files, even though I prefer Lua for doing calculations. What it does is very simple. It reads the lines of the XHTML file produced by multimarkdown one by one, and copies them to the output file. After encountering the <body> tag, it inserts some HTML code that creates the header. After that, it just copies the rest of the line, stripping out single-line comments.

The complete source code is shown below;

use strict;
my $prefix = $0;
$prefix =~ s/\/tools.*//;
my $infile = @ARGV[0] || die "You must supply an input file name.";
my $outfile = @ARGV[1] || die "You must supply an output file name.";
$infile ne $outfile || die "Input and output may not be the same file.";
open(IN, $infile) || die "Cannot open $infile: $!";
open(OUT, ">$outfile") || die "Cannot open $outfile: $!";
my $title = "Roland's Homepage";
my $subtitle = "";

while (<IN>) {
    print OUT $_;
    if (/(<title>)(.*)(<\/title>)/) {$title= $2;}
    elsif (/(^.*<meta name="Subtitle" content=")(.*)(".*)/) {$subtitle= $2;}
    elsif (/^<body>/) {
    print OUT "<table width=\"100%\" class=\"header\"><tbody><tr>\n";
    print OUT "   <td rowspan=\"1\" colspan=\"1\">\n";
    print OUT "   <h1 style=\"font-size: 250%;\">$title</h1>\n";
    print OUT "   <p>$subtitle</p></td>\n";
    print OUT "   <td rowspan=\"1\" colspan=\"1\">\n";
    print OUT "   <img src=\"$prefix/pics/face.jpg\" width=\"105\" height=\"141\" />\n";
    print OUT "   </td></tr>\n";
    print OUT "</tbody></table>\n";
    last;
    }
}
# Now just copy the rest of the lines through.
while (<IN>) {
    # skip single line comments.
    if (/^\s*<!--.*-->\s*/) {next;}
    print OUT $_;
}

XZ compression in FreeBSD 8.1-RELEASE

2010–07–28

With the release of 8.1, the xz compression program has been imported into FreeBSD. I decided to do some testing to compare it to the bzip2 format. In this test I’ve used the standard non-multithreaded implementations of these compression programs.

Tests

First I compressed my procmail logfile (a plain text file) with both bzip2 and xz. The compression comands used were:

> time bzip2 -c procmail.log >procmail.log.bz2
9.401u 0.074s 0:09.56 99.0% 37+1512k 237+16io 0pf+0w
> time xz -c procmail.log >procmail.log.xz
15.638u 0.112s 0:15.75 99.9% 63+1474k 0+15io 0pf+0w

As shown, xz takes longer to compress the file. The resulting xv compressed file is somewhat smaller than the bzip2 compressed file, but not spectacularly so;

Decompression was significantly faster for xz;

> time xz -dc procmail.log.xz >/dev/null
0.256u 0.015s 0:00.28 92.8% 66+1534k 0+0io 2pf+0w
> time bzip2 -dc procmail.log.bz2 >/dev/null
0.866u 0.014s 0:00.88 98.8% 37+1529k 0+0io 0pf+0w

Next up were a couple of UFS2 dumps, which are a mixture of binary and text data.

> time bzip2 -c root-0-20100724.dump >root-0-20100724.dump.bz2
14.605u 0.105s 0:14.91 98.5% 37+1503k 784+238io 0pf+0w
> time xz -c root-0-20100724.dump > root-0-20100724.dump.xz
70.884u 0.240s 1:11.14 99.9% 63+1473k 0+159io 0pf+0w
> time bzip2 -c usr-0-20100724.dump > usr-0-20100724.dump.bz2
1484.101u 12.137s 25:27.87 97.9% 37+1499k 66059+26227io 0pf+0w
> time xz -c usr-0-20100724.dump > usr-0-20100724.dump.xz
4510.041u 19.445s 1:15:37.69 99.8% 63+1474k 66067+21319io 3pf+0w
> time bzip2 -c var-0-20100724.dump >var-0-20100724.dump.bz2
51.057u 0.390s 0:52.57 97.8% 37+1500k 2032+707io 1pf+0w
> time xz -c var-0-20100724.dump > var-0-20100724.dump.xz
149.619u 0.593s 2:30.28 99.9 %63+1474k 1+685io 0pf+0w

Here again xz takes significantly longer to compress, but outperforms bzip2 in terms of compressed size, especially for root–0–20100724.dump, where the difference is really spectacular.

Decompression times are given below. Again xz is much faster than bzip2 in decompressing data.

> time bzip2 -dc root-0-20100724.dump.bz2 >/dev/null
4.985u 0.052s 0:05.04 99.8% 37+1502k 242+0io 0pf+0w
> time xz -dc root-0-20100724.dump.xz >/dev/null
2.615u 0.052s 0:02.73 97.4% 64+1482k 165+0io 3pf+0w
> time bzip2 -dc usr-0-20100724.dump.bz2 >/dev/null
449.188u 4.511s 7:36.99 99.2% 37+1500k 26356+0io 0pf+0w
> time xz -dc usr-0-20100724.dump.xz > /dev/null
197.313u 3.179s 3:21.98 99.2% 63+1474k 21408+0io 1pf+0w
> time bzip2 -dc var-0-20100724.dump.bz2 >/dev/null
12.407u 0.105s 0:12.51 99.9% 37+1500k 0+0io 0pf+0w
> time xz -dc var-0-20100724.dump.xz >/dev/null
8.077u 0.075s 0:08.15 99.8% 63+1475k 0+0io 0pf+0w

Conclusion

• bzip2 compresses faster than xz
• xz compresses better than bzip2, sometimes spectacularly so
• xz decompresses significantly faster than bzip2

Updating to FreeBSD 8.1-RELEASE

2010–07–25

Updating my desktop to 8.1-RELEASE was pretty straightforward. I used the following steps

• Made a backup and verified that it can be restored. I cannot stress enough how important that is!

• Using csup I updated my kernel sources, using the following supfile;

  *default host=cvsup.nl.FreeBSD.org *default base=/var/db *default prefix=/usr *default release=cvs tag=RELENG_8_1_0_RELEASE *default delete use-rel-suffix *default compress

  src-all

• Updated my kernel configuration. Based on the changes in the GENERIC kernel from 8.0-RELEASE to 8.1-RELEASE, I changed options COMPAT_IA32 to options COMPAT_FREEBSD32 in my kernel configuration. Additionally, I added options INCLUDE_CONFIG_FILE and device vlan to my kernel configuration.

• Ran make buildworld and make kernel as root from /usr/src.

• Rebooted into single user mode.

• Mounted /usr, /var and /tmp since these are seperate partitions in my setup. Do not forget to make the root filesystem writeable with mount -uw /.

• Ran mergemaster -p.

• Ran make installworld from /usr/src.

• Ran mergemaster -i -U.

• Rebooted into the the newly upgraded system.

Enlarging networking buffers

2010–07–06

In the hope of increasing networking performance, I’ve set the following sysctls in /etc/sysctl.conf;

# Increase send/receive buffer maximums from 256KB to 16MB.
# FreeBSD 7.x and later will auto-tune the size, but only up to the max.
net.inet.tcp.sendbuf_max=16777216
net.inet.tcp.recvbuf_max=16777216

# Double send/receive TCP datagram memory allocation.  This defines the
# amount of memory taken up by default *per socket*.
net.inet.tcp.sendspace=65536
net.inet.tcp.recvspace=131072

# Enlarge buffers for BPF device.
sysctl net.bpf.bufsize=65536
sysctl net.bpf.maxbufsize=524288

Defining command completions for tcsh

2010–06–21

The default interactive shell is [tcsh(1)][]. Recently I discovered that this shell also has the ability to complete your command-line by using the built-in complete command. After some experimentation, I’ve added the following lines to my /etc/csh.cshrc file:

complete cd 'p/1/d/'
complete man 'p/1/c/'
complete git 'p/1/(add am checkout commit diff format-patch gc init log mv rm revert status)/'
complete openssl 'p/1/(enc rand base64)/' 'p/2/(-base64 -aes-256-cbc -idea-cbc -bf-cbc)/'
complete portsnap 'p/1/(fetch)/' 'p/2/(update)/'
complete portmaster 'p/1/(-a)/' 'p/2/(-B)/' 'p/3/(-d)/'
complete shutdown 'p/1/(-p)/' 'p/2/(now)/'
complete df 'p/1/(-h)/'
complete du 'p/1/(-csm)/'

This means e.g. that if I type cd followed by the TAB key, the shell will list all available subdirectories. For another example, if I type portsnap and press the TAB key twice, it expands to portsnap fetch update. Especially with programs like openssl and git which have a ton of available commands, this comes in very handy.

Creating a restart command file for CDrecord

2010–05–17

During an update of CDrecord I spotted the following in the release notes for cdrtools–2.01.01a54.

Default Transfer Size reverted from 126 kB to 63 kB. Sorry FreeBSD guys - it seems that FreeBSD is the only OS that correctly deals with larger DMA sizes.

FreeBSD people may add a line with CDR_TRANSFERSIZE=126k in /etc/default/cdrecord to raise the default.

In FreeBSD, this file is located in /usr/local/etc/cdrecord since /usr/local/ is the place where ports should put this. I didn’t have it, so I had a look at my devices with cdrecord -scanbus

> cdrecord -scanbus
Cdrecord-ProDVD-ProBD-Clone 2.01.01a79 (amd64-unknown-freebsd8.0) Copyright (C) 1995-2010 Jörg Schilling
Using libscg version 'schily-0.9'.
scsibus0:
    0,0,0     0) 'Optiarc ' 'DVD RW AD-7203A ' '1.01' Removable CD-ROM

[snip]

scsibus1:
    1,0,0   100) *
    1,1,0   101) 'TSSTcorp' 'DVD-ROM SH-D163B' 'SB01' Removable CD-ROM

This tells me that device 0,0,0 is my DVD burner. This has lead me to create the following /usr/local/etc/cdrecord file

# file: /usr/local/etc/cdrecord
# host: slackbox.erewhon.net
# Time-stamp: <2010-05-17 09:13:50 rsmith>
# $Id: c68c0656331f1322b55c308416c116fd553aae40 $

CDR_DEVICE=Optiarc
CDR_TRANSFERSIZE=126k

# drive name    device  speed   fifosize        driveropts      transfersize
Optiarc=           0,0,0   -1      -1                 burnfree

Using the rc scripts to start a virtual server in a jail(8)

2010–05–09

In the previous item, we saw how to build, start and stop a jail. The rc scripts allow us to easily start and stop a jail.

Starting the jail (from the host):

slackbox# /etc/rc.d/jail onestart server
Configuring jails:.
Starting jails: server.erewhon.net.

Then log into the jail via ssh;

slackbox# ssh 192.168.0.100

Closing the jail (again from the host);

slackbox# /etc/rc.d/jail onestop server

For this to work, the following has to be set in /etc/rc.conf:

# For jails
# Do not start automatically
jail_enable="NO"
# General jail options
jail_list="server"
jail_interface="rl0"
jail_devfs_ruleset="devfsrules_jail"
jail_devfs_enable="YES"
# Specific options for the jail 'server'.
jail_server_rootdir="/home/jails/192.168.0.100"
jail_server_hostname="server.erewhon.net"
jail_server_ip="192.168.0.100"

See /etc/defaults/rc.conf for an overview of all the settings that can be used.

Creating a virtual server on FreeBSD with a jail(8)

2010–05–09

After Virtualbox crashed my machine, I decided to try my hand at building a virtual server using FreeBSD’s jail facility.

First, I created a directory to serve as the root directory for my jail;

slackbox# mkdir -p /usr/local/var/jails/192.168.0.100

Then I built the FreeBSD base system, so I could install it in the jail. Were I to build multiple jails, I’d probably use ezjail.

slackbox# cd /usr/src
slackbox# make buildworld
slackbox# mount -u -o exec /tmp
slackbox# make installworld DESTDIR=/usr/local/var/jails/192.168.0.100
slackbox# make distribution DESTDIR=/usr/local/var/jails/192.168.0.100
slackbox# du -csm /usr/local/var/jails/192.168.0.100/
184 /usr/local/var/jails/192.168.0.100/

An empty /etc/fstab file is needed because the filesystems on the host are already mounted. Mounting filessystems in jails is disallowed by default. This can be changed by supplying the parameter allow.mount=1to the jail command.

slackbox# touch /usr/local/var/jails/192.168.0.100/etc/fstab

The jail will need some device nodes. The following is a way to create them.

slackbox# mount -t devfs devfs /usr/local/var/jails/192.168.0.100/dev
slackbox# devfs -m /usr/local/var/jails/192.168.0.100/dev ruleset 4
slackbox# devfs -m /usr/local/var/jails/192.168.0.100/dev rule applyset

Also you don’t want to have a actual kernel, so just link it to /dev/null.

slackbox# cd /usr/local/var/jails/192.168.0.100/boot/kernel; ln -sf ../../dev/null kernel

Some files need to be set up in the jail;

 slackbox# cat /usr/local/var/jails/192.168.0.100/etc/rc.conf 
 # /etc/rc.conf
 # Local configuration for server.erewhon.net
 # Hostname and ip-adres are set by the jail.
 # Only expose the basic necessary devices in a jail.
 devfs_system_ruleset="devfsrules_jail"
 # Quell warnings about network interfaces.
 network_interfaces=""
 # Run the secure shell daemon.
 sshd_enable="YES"
 # Do not run sendmail
 sendmail_enable="NO"
 # Do not run the port mapper.
 rpcbind_enable="NO"

 slackbox# cat /usr/local/var/jails/192.168.0.100/etc/resolv.conf 
 search erewhon.net
 nameserver 10.0.0.150

Now it is time to start the jail for the first time.

slackbox# ifconfig rl0 inet alias 192.168.0.100/32
slackbox# cd /usr/local/var/jails/192.168.0.100
slackbox# jail /usr/local/var/jails/192.168.0.100 server.erewhon.net 192.168.0.100 /bin/csh

In the jail, sysinstall(8) is used to set the root password. Additionally, I’ve added a user named ‘rsmith’ as a member of the wheel group, and with /bin/tcsh as default shell.

After use, the jail is destroyed by logging out of the started program. To completely remove everything associated with the jail, unmount the devfs instance and remove the alias from the network interface.

To start up a virtual server in the jail, run;

slackbox# ifconfig rl0 inet alias 192.168.0.100/32
slackbox# mount -t devfs devfs /usr/local/var/jails/192.168.1.1/dev
slackbox# cd /usr/local/var/jails/192.168.0.100
slackbox# jail /usr/local/var/jails/192.168.0.100 server.erewhon.net 192.168.0.100 /bin/sh /etc/rc

To close a jail, use jail -r to kill all processes in the jail. Then unmount the devfs instance used in the jail, and remove the alias from the network device.

Breaking and reparing Xorg

2010–05–01

By accident I managed to break X11. The symptoms were that Xorg could not open new programs, and after restarting it complained that it could not save a compiled keymap, so it unloaded the keayboard and mouse modules, leaving X unusable. :-(

After going into single-user mode, I ran fsck on all filesystems, which came up clean. I then used my backup from the day before to check if any important files had been changed. After some further investigation it turned out that the permissions on /tmp had gone haywire. This is important because Xorg keeps some sockets in hidden directories under /tmp. These sockets are used by programs to talk to the Xorg server, so they are kind of necessary. :-)

I’m not sure what caused the strange permissions on /tmp. But I’d been playing with running make installworld for a jail, and I had to remove the noexec option from the /tmp filesystem to do that. I probably screwed something up there. Anyway, running

# chmod 1777 /tmp

fixed things.

Downloading photos from a Canon digital IXUS 970 IS

2010–04–25

To download all pictures and videos on the camera to the current directory, simply use:

> gphoto2 -P

This gives you a number of files, with their names in upper case. So I use a shell-script to convert them to lower case;

> tolower *

The contents of this script are:

#!/bin/sh
if [ $# -eq 0 ]; then
echo "Usage: tolower <files>"
fi
for f in $*; do
    n=`echo $f|tr [:upper:] [:lower:]`
    #echo "moving $f to $n"
    mv $f $n
done

Next, I add my copyright notice, using the mogrify program from the ImageMagick suite;

> mogrify -comment "Copyright © 2010, R.F. Smith <rsmith@xs4all.nl>" *.jpg

You can also use this program to add a visible copyright notice (e.g. using a mask image).

After that, I use the jhead program to set the file time to the time the picture was taken:

> jhead -ft *.jpg

The files are made read-only, so they cannot be accidentally deleted;

> chmod 444 *.jpg

Using Matroška containers for videos

2010–04–25

I’ve been using ffmpeg2theora to encode my movies with the Theora video codec and the Vorbis audio codec. (see previous installment) But up till now I’ve always used the Ogg .ogv container format. I’ve had some problems with that, in the sense that seeking in a Theora encoded video in an .ovg container doesn’t seem to work properly with my favorite video player, mplayer.

Recently I discovered the Matroška container format, so I installed the mkvtoolnix toolset. Note that Matroška is just a container. It doesn’t much care what kind of codecs you use to actually encode the video and audio streams. When transforming files from the .ogv format to the .mkv format, the problems with seeking disappeared, and the files are somewhat smaller to boot.

An example of the commands used to convert a DVD is given below;

Find the largest track (the movie) using lsdvd.

> lsdvd /dev/cd1

Dump the track (number 1 in this example) to an mpeg file .

> mplayer dvd://1 -dumpstream -dumpfile enigma.mpg

Copy the video data, but convert the AC3 soundtrack to a smaller stereo stream. In this case, mencoder is used to convert a AC3 sound stream into stereo mp3 format, because it is smaller and I can’t hear the difference with my speakers anyway. Mencoder can also be used to e.g. crop the image to remove black borders.

> mencoder -ovc copy -oac mp3lame -idx -aid 129 -o enigma.avi enigma.mpg

Convert to theora video and vorbis audio, with slightly higher than default quality.

> ffmpeg2theora --sync -v 7 -a 2 enigma.avi

Convert to Matroška container format.

> mkvmerge --title "Enigma" -o enigma.mkv enigma.ogv
> du -m enigma.*
4920    enigma.avi
5348    enigma.mpg
933     enigma.ogv
925     enigma.mkv
> rm enigma.avi enigma.mpg enigma.ogv

It should be noted that in this conversion the pixel size of the video has not been changed. And yet the size of the resulting .mkv file is just 17% of the original .mpg file. Of course, a portion of that original size are different soundtracks and subtitles, which I’ve discarded. Still, not bad for a 720×576 movie of 1:53:33! Scaling down the movie somewhat would have yielded a significantly smaller file.

Updating TeXLive to version 2009

2010–01–10

Last week I got my TeXLive 2009 CD in the mail. Time to upgrade :-)

Installing the packages

From previous years I know that the install-tl script needs a large terminal. So I stretch my terminal window under the X Window System to 80 columns and 45 lines.

This release has some new prequisites that I did not have installed; wget and xzdec. So I installed their ports first;

# cd /usr/ports/archivers/xz/
# make install clean
# cd /usr/ports/ftp/wget
# make install clean

Now the installation of TeXLive can start;

# mount /cdrom1
# cd /cdrom1/texlive/
# ./install-tl
Installing TeX Live 2009 from: /cdrom1/texlive
Platform: amd64-freebsd => 'x86_64 with FreeBSD'
Distribution: live (uncompressed)
Directory for temporary files: /tmp
Loading /cdrom1/texlive/tlpkg/texlive.tlpdb

An old installation of TeX Live has been found in /usr/local/texlive/2008

If you want the selection of collections and various options being taken
over press `y', otherwise anything else.

Import settings from previous TeX Live installation: (y/n): y

======================> TeX Live installation procedure <=====================

=======> Note: Letters/digits in <angle brackets> indicate menu items <=======
=======>       for commands or configurable options                   <=======

 Detected platform: x86_64 with FreeBSD

*** WARNING: No binaries for your platform found.  
 <B> binary systems: 0 out of 13

 <S> Installation scheme (scheme-custom)
     28 collections out of 85, disk space required: 1311 MB

 Customizing installation scheme:
   <C> standard collections
   <L> language collections

 <D> directories:
   TEXDIR (the main TeX directory):
     /usr/local/texlive/2009
   TEXMFLOCAL (directory for site-wide local files):
     /usr/local/texlive/texmf-local
   TEXMFSYSVAR (directory for variable and automatically generated data):
     /usr/local/texlive/2009/texmf-var
   TEXMFSYSCONFIG (directory for local config):
     /usr/local/texlive/2009/texmf-config
   TEXMFHOME (directory for user-specific files):
     ~/texmf

 <O> options:
   [ ] use letter size instead of A4 by default
   [X] create all format files
   [X] install macro/font doc tree
   [X] install macro/font source tree
   [ ] create symlinks to standard directories

 <V> set up for running from DVD

Other actions:
 <I> start installation to hard disk
 <H> help
 <Q> quit

Enter command: I

This installed 1473 packages, using 1303 MiB.

Building and installing TeXLive 2009 binaries for FreeBSD 8.0-RELEASE-p2 amd64

This edition of TeXLive does not come with binaries for my abovementioned platform, so I had to compile it myself. Luckily this turned out to be relatively straightforward. I used the standard build script that uses some libraries from the texlive package rather than from the system, because it turned out that that the texlive source did not like some of the libraries I had installed (e.g. poppler). So most of these binaries only depend on the system libc.so.7 and libm.so.5. Of course, programs that use the X Window system depend on its libraries, and XeTeX and friends depends on libz.so.5 and libfontconfig.so.1, libiconv.so.3, libexpat.so.6 and libfreetype.so.9.

> cd ~/tmp
> cp /cdrom1/texlive/source/texlive-20091011-source.tar.xz .
> cd src/
> tar xvf ../texlive-20091011-source.tar.xz
> cd texlive-20091011-source/

> env TL_MAKE=gmake ./Build
...
make world done.
      882.52 real       622.82 user       154.51 sys
+ echo 0

./Build:      306 executables in /home/rsmith/tmp/src/texlive-20091011-source/inst/bin.
done Sat Jan 9 21:57:46 CET 2010

Next, I had to install the binaries left in inst/bin in their proper location. Note that I used the same binary directory name as TeXLive 2008.

> su
# mkdir -p /usr/local/texlive/2009/bin/amd64-freebsd
# mv inst/bin/x86_64-unknown-freebsd8.0/* /usr/local/texlive/2009/bin/amd64-freebsd/

For those who want them, these binaries are available in the archive texlive2009-freebsd8-amd64.tar.bz2. Just go to /usr/local/texlive/2009, and extract the archive there.

Updating my config files to use the TeXLive 2009 installation

As explained on my configfiles page, I keep my configuration files in a git repository in my home directory for easy management. So it was quite straightforward to update them for TeXLive 2009. First, I checked where the string texlive/2008 is used in my configuration files;

> cd ~/setup/desktop
> grep -lR texlive .
./emacs/emacs.el
./etc/csh.cshrc
./etc/cshrc.root
./etc/login.conf
./etc/manpath.config
./etc/profile
./filelist.root
./pdflib/pdflib.upr
./user/local.html

Then these files are edited in place using sed. Since I’m using the standard FreeBSD tcsh, this command uses its syntax, and is somewhat different from its Bourne shell equivalent;

foreach f (emacs/emacs.el etc/csh.cshrc etc/cshrc.root etc/login.conf etc/manpath.config etc/profile filelist.root pdflib/pdflib.upr user/local.html) 
    sed -i .bak -e 's|texlive\/2008|texlive\/2009|g' $f
end

After making sure (with git diff) that all changes are correct, I removed the *.bak files, and installed the changed configuration files. To have these settings take effect, you have to log out completely and then log in again. A little bit of testing confirmed that TeXLive was working correctly.

Update to FreeBSD 8.0-RELEASE-p2

2010–01–07

On 2010–01–06, three security advisories and one errata notice were published for FreeBSD. After updating the source, most of the updates went without problems.

However, the update procedure for ‘FreeBSD-SA–10:02.ntpd’ failed;

# cd /usr/src/usr.sbin/ntp/ntpd
# make obj && make depend && make && make install
don't know how to make /usr/obj/usr/src/usr.sbin/ntp/ntpd/../libparse/libparse.a. Stop

Thanks to a tip from the freebsd-security mailing list, I changed it;

# cd /usr/src/usr.sbin/ntp
# make obj && make depend && make && make install

This will build the required libraries before ntpd.

Encoding videos with Ogg Vorbis/Theora

2010–01–02

Most high-performance video codecs are encumbered with patents. Not so long ago, a new version of the Theora video codec was released with improved performance. I tested it with version 0.25 of ffmpeg2theora, which creates files with Vorbis sound and Theora video.

The MPG file was produced with ‘mplayer dvd://1 -dumpstream -dumpfile movie.mpg’. Converting to AVI format was done with ‘mencoder -profile hqmovie -o movie.avi movie.mpg’. (This profile is covered further down on this page.) The OGV file was created with the command ‘ffmpeg2theora --sync --aspect 16:9 --croptop 8, --cropbottom 8 -v 7 -c2 movie.mpg’. The difference in size between the AVI and the OGV file is impressive, especially since I cannot see much difference in quality. The only snag is that mencoder seems to do a better job of creating a better sounding stereo sound from e.g. a AC3 stream. Therefore I tend to do a quick resample of only the sound with ‘mencoder -ovc copy -oac mp3lame -idx -o movie-int.avi movie.mpg’ .

I cannot see or hear much of a difference between the H.264+MP3 encoded movie and the one encoded with Theora+Vorbis. The H.264 AVI file made by mencoder is 70% of the size of the original MPG file. The OGV files are 30% of the size of their original MPG files, and are in my eyes of equal quality to the H.264 encodd AVI files. And since Ogg/Theora is not encumbered with patents like H.264, it makes me think that theora+vorbis is a better choice for encoding video now. I’ve encoded other movies as well, and constantly find that the size of the theora+vorbis encoded film is around 30–40% of the size of a H.264+MP3 encoding.

For feature films, file size in this OGV format range from 1051 MiB for a 97 minute standard format movie, via 1750 MiB for a 147 minute widescreen movie to 2988 MiB for a 207 minute black/white movie.

One thing to take into account is that on most DVDs the sound is encoded as AC3 which has six sound channels, and ffmpeg2theora cannot downsample that. So it is best to resample the sound to plain stereo MP3 with mencoder, and then use ffmpeg2theora. It is then often necessary to use the --sync option during theore encoding to prevent sound/video mismatches. Widescreen DVDs are encoded at 720x576 pixels, with an aspect ratio of 16:9 (≈1.78:1). This means that mplayer will resize the movies to 1024x576. Usually, there are black bands about 75 pixels high on top and bottom of these movies. We want to get rid of those. It is important to remember that for effective encoding, width and hight should be a multiple of sixteen. Apperently this holds for a lot of video codecs, not just theora. So instead of removing 75 pixels on top and bottom, we remove 72, because (576–2×72)=432, which is exactly 27×16. Remember we still want the player to enlarge the width of the image to 1024. The aspect ratio then becomes 1024/432 = 2.37. This is approximately 24:10, so that is what I use. So the complete procedure to convert a widescreen movie to Ogg Theora+Vorbis is;

> mplayer dvd://1 -dumpstream -dumpfile movie.mpg
> mencoder -ovc copy -oac mp3lame -idx -o movie.avi movie.mpg
> ffmpeg2theora --sync --aspect 24:10 --croptop 72, --cropbottom 72 -v 7 -c 2 movie.avi
> rm movie.avi

2009

Upgrade emacs to 23.1 on FreeBSD

2009–12–21

For some time I have been using the emacs-devel port (version 23.0.95), while some problems were worked out to fit the new version 23.1 into FreeBSD’s ports system. To this end I had placed the line

EMACS_PORT_NAME=emacs-devel

in /etc/make.conf. For the new 23.1 port I had to remove that. Updating the to the new port was done as follows.

# pkg_delete auctex-emacs-devel-11.85_1
# pkg_delete nxml-mode-emacs-devel-20041004_3
# pkg_delete emacs-23.0.95_1
# cd /usr/ports/editors/emacs
# make install clean
# cd /usr/ports/print/auctex/
# make install clean

nXML-mode is now part of emacs, so I didn’t have to install that. In fact, I had to remove the lines that refer to it from my .emacs.elc

Switching X11 from the radeonhd to ati driver

2009–12–21

After having experienced some X11 lockups with the xf86-video-radeonhd driver in full hardware acceleration mode, I switched back to the xf86-video-ati driver. The device section for my graphics card now looks like this;

Section "Device"
        Identifier      "Card0"
        Driver          "radeon"
        VendorName      "Sapphire"
        BoardName       "Radeon X 1650 Pro"
        BusID           "PCI:1:0:0"
        Option          "AccelMethod"   "EXA"
EndSection

There is no ‘Module’ section in my /etc/X11/xorg.conf file anymore; the needed modules were all loaded by default anyway.

Note that I’ve built my X server without the HAL daemon, so I must use

Option "AutoAddDevices"  "off"

to the ServerLayout section in my /etc/X11/xorg.conf.

Starting website rewrite in multimarkdown

2009–12–17

Up till recently I’ve been editing my webpages by hand in emacs using the excellent (validating) nxml-mode. But the long URI’s tend to make the text less than pretty to edit, and adding tags (even in Emacs) is tedious.

So I started looking for alternatives. Beforehand I must admit that I love editing plain text in emacs. It is fast and doesn’t distract me with frills of a What You See Is Somewhat Like You Get “wordprocessor”. This is also why I love using pdfTeX for correspondence.

Recently I came across the markdown format when I discovered multimarkdown. At that time I was looking for a way to use math formulas on a web page without having to hand-edit mathml (shudder). The simple syntax suits me, and coupled with the markdown-mode emacs extension, it rocks. :-) I’m starting to rewrite my webpages in it. And I’m starting this blog.

Installing multimarkdown

2009–12–15

Since multimarkdown isn’t available as a FreeBSD port, I installed it myself. This was pretty easy. Below is a summary of the commands I used to install multimarkdown version 2.0.b6.

> tar xf fletcher-MultiMarkdown-7b87653.tar.gz
> cd fletcher-MultiMarkdown-7b87653
# mkdir /usr/local/share/multimarkdown
# mv XSLT MultiMarkdownXSLTMathML Utilities bin /usr/local/share/multimarkdown/
# mkdir /usr/local/share/doc/multimarkdown
# mv README.txt Documentation/* /usr/local/share/doc/multimarkdown/
# cd /usr/local/bin
# ln -s /usr/local/share/multimarkdown/bin/mmd2XHTML.pl mmd2XHTML

In the mmd2XHTML.pl script, some paths were hardcoded to /usr/share/multimarkdown. I changed those to /usr/local/share/multimarkdown. There are also scripts for converting multimarkdown to other formats, but I didn’t bother with those since I don’t plan on using them.

Modifying Emacs’ scrolling behavior

2009–12–13

With thanks to an article on the emacs-fu blog, I’ve add the following to my emacs.el;

(set-scroll-bar-mode nil) ;; No scrollbar.
(setq 
 scroll-margin 0                  
 scroll-conservatively 100000
 scroll-preserve-screen-position 1)

Finding leftover old binaries and libraries after ‘make installworld’

2009–12–09

Adding options to /etc/src.conf does not remove old binaries, libraries or manpages! It just prevents the system from building newer ones. The best way to deal with this is to use find(1) to locate binaries and libraries that are older than the most recent installworld. It might be a good idea to make a backup first, in case you screw up the system! Now, supposing the latest installworld (with the new options in /etc/src.conf) was December 3rd. Running the following command will reveal all older files;

find /bin -type f -and -not -newermt 'Dec 3' -ls

After checking that you’ve really only found old binaries, replace -ls with -delete to remove them. Next, repeat this for the directories /sbin, /usr/bin, /usr/sbin, /lib, /libexec, /usr/lib, /usr/libexec, /usr/share/man/man* and /rescue.

Of course, one can make the list more fancy with a command like;

find /bin -type f -and -not -newermt 'Dec 3' -ls | \
awk '{print $11, "\t", $8, " ", $9;}'

Upgrading FreeBSD to 8.0-RELEASE

2009–11–29

After using FreeBSD 7.2 since its release, I switched to the PRERELEASE of 8.0, because it added support for the Attansic/Atheros L1 network chip on my desktops motherboard using the age(4) driver. After the official release, I updated again.

Update procedure

The process is documented in detail in §24.7 of the FreeBSD Handbook. This is just my short and sweet version.

First, I use csup(1) to update the source code. (Note that the name ‘csup’ is not a typo! This program is a rewrite in C of the cvsup program from ports. It has been part of the base system since 6.2.)

> cat releng80-supfile
*default host=cvsup.nl.FreeBSD.org
*default base=/var/db
*default prefix=/usr
*default release=cvs tag=RELENG_8_0
*default delete use-rel-suffix
*default compress
## Main Source Tree & Documentation.
src-all

Then the system software and kernel is rebuilt. The kernel is also installed, and the system is rebooted.

# csup releng80-supfile
# rm -rf /usr/obj/*
# cd /usr/src
# make buildworld
...
# make kernel
...
# reboot

During the boot process, I choose single user mode. I log in using the simple default shell, and mount the necessary filesystems. When mounting /tmp, the exec option is used, because my default set-up is to mount /tmp so that it doesn’t allow programs to be started from within /tmp. Since the root directory is mounted read-only in single user mode, that has to be updated as well.

# mount /usr
# mount /var
# mount -o exec /tmp
# mount -u -w /

Now the update process can be completed by installing the userland programs, and updating the configuration files.

# mergemaster -p
...
# cd /usr/src
# make installworld
...
# mergemaster -i -U
# reboot

After rebooting into multi-user mode, I check if any changes in configuration files need to be merged into my repository. This is documented on my configfiles page.

Changes to the kernel and userland

There have been some changes in the kernel configuration.

• The options PRINTF_BUFR_SIZE=128 statement prevents mashed up printouts on a SMP kernel
• The device sio has been replaced by device uart for serial ports.
• The age(4) device has become available.
• The device uscanner has been removed.
• Most USB devices are now accessed via libusb. An implementation of that API has been written for FreeBSD, and has become part of the base system in 8-CURRENT since 2009–03–09.

Configuration changes

• The ttyd devices have been renamed to ttyu. This affects /etc/ttys
• The kern.polling.enable sysctl has been deprecated.
• Two new attributes have been added to the default profile in /etc/login.conf; :swapuse=unlimited: and :pseudoterminals=unlimited:.
• In /etc/devfs.rules, the line for the uscanner device was removed. Since USB devices now live in their own directory, the path for USB devices was changed from 'usb*' to 'usb/*'.

Re-encoding DVDs with mencoder

2009–09–02

DVDs tend to come with all kinds of crap at the beginning which you can’t skip when playing the movie in a standalone DVD player. I find that most annoying, especially since I buy DVDs instead of downloading them! Therefore I generally re-encode the contents to harddisk, so I can skip that crap. This also generates a backup for the movies I buy. I like watching movies on my PC anyway, since the resolution of my monitor is a lot better than that of my TV!

Because the MPEG2 encoding used on DVDs is not very space efficient, I prefer to re-encode video to H.264 format with MP3 sound with the help of mencoder. After extensive reading of documentation and searching for and trying of encoding parameters, I arrived at a set of parameters that seem to work well.

Mencoder enables you to group configuration options into “profiles”, that can be stored in ~/.mplayer/mencoder.conf. Below is the profile that I use for re-encoding DVDs;

[hqmovie]
profile-desc="High-quality movie encoding."
ovc=x264=1
oac=mp3lame=1
x264encopts=subq=6:partitions=all:8x8dct:me=umh:frameref=5:bframes=3
x264encopts=b_pyramid:weight_b:qp=18:threads=auto
idx=1

First, the MPEG2 stream is dumped to disk;

> mplayer dvd://1 -v -dumpstream -dumpfile movie.mpg

On a DVD, the movie is usually chapter one, hence the one in the command above. But not always, so try playing different chapters to see which one is the real movie.

Before you re-encode it, you need to know which soundtrack and optionally subtitle you want. The -aid and -sid options let you choose an audio track and subtitle respectively. When you perform the dump, you’ll usually see which tracks and subtitles are available if you use the -v option;

> mplayer dvd://6 -v -dumpstream -dumpfile movie.mpg
URL: dvd://6
Reading disc structure, please wait...
There are 17 titles on this DVD.
There are 8 chapters in this DVD title.
There are 1 angles in this DVD title.
DVD successfully opened.
audio stream: 0 format: ac3 (stereo) language: en aid: 128.
number of audio channels on disk: 1.
subtitle ( sid ): 0 language: nl
subtitle ( sid ): 1 language: fr
number of subtitles on disk: 2

One thing that cannot be easily put into the profile is the cropping needed to remove any black bands above and below the movie. To check how you should crop the movie, first play it with the ‘cropdetect’ filter;

> mplayer -vf cropdetect movie.mpg

This will put out suitable arguments for cropping. Stop with control-c when you have the correct cropping arguments. Output looks something like;

[CROP] Crop area: X: 1..719  Y: 70..505  (-vf crop=704:432:10:72). 

Suppose you want audio track 1, subtitle 0, and the movie should be cropped to the information gained with cropdetect;

> mencoder -profile hqmovie -vf crop=704:432:10:72 -o movie.avi movie.mpg

The resulting AVI file would normally be about half the size of the original MPG file. But it can vary between 20% and 80% of the original filesize, depending on the content of the movie.

Adding $Id$ keywords to files controlled with git

2009–08–22

After some experimentation, I discovered that git can use keywords like $Id:$, see gitattributes. But these are only updated if a file is checked out! To make that happen automatically, I created a post-commit script and removes the files that have been changed and have an $Id in them, and checks them out again.

#!/bin/sh
# Script to be executed after a successfull commit.
# Time-stamp: <2009-08-22 16:09:46 rsmith>
# $Id: 3a91728b079ffa48971420b899b1f38745573c89 $
# 
# Looks for changed files that contain the '$Id' keyword, removes those files
# and checks them out again to refresh the keywords.


files=`git show HEAD | grep '^+++ b' | sed 's/.*b\///'`

for f in $files; do
    if grep -q '$Id' $f; then
    echo "Checking out $f to update Id."
    rm -f $f
    git checkout $f
    else
    echo "No Id found in $f"
    fi
done

This script is moved to .git/hooks/post-commit, and made executable. To make this keyword expansion happen, place the following lines in .git/info/attributes;

* ident
text/no-id -ident

As an example, the file text/no-id is excluded from keyword expansion.

Generating random passwords

2009–05–25

Most UNIX-like operating systems have a /dev/random device that produces semi-random data. So a command like

dd if=/dev/random bs=8 count=1

would produce a string of eight semi-random bytes. But since these byte sequences can contain characters that are not printable, this output is not directly usable as a password. My solution is to encode the random data with the base64 algorithm, rendering it readable;

> dd if=/dev/random bs=48 count=1 | b64encode -m -
begin-base64 644 -
1+0 records in
1+0 records out
48 bytes transferred in 0.000055 secs (871544 bytes/sec)
5sk/J0BHleay1NlcBrebZ0qR2/4mbJl4LMmC64O2YrrMURZwEmd6tTM2FeI13NBp
====

This yields a string of 65 characters. Using that as a random password for e.g. online services should make your passwords a tough nut to crack. You have to store this password in a secure place, though. I do not imagine that many people can memorize them. :-)

If you don’t have a /dev/random device, you can use the rand(1) command from openssl;

> openssl rand -base64 48
7yOb44lpNQQhs23IwIBndT+odhznaGQ8JF3S/MPX2BJirTK9SjhsEx5c7l7G1rC8

Adding tabbar-mode to emacs

2009–05–19

Where I found it I cannot remember, but I was sure glad when I found tab-bar mode. Tabbed browsing has been a favorite of mine since it appeared in FireFox, and I’m very glad to have it available in emacs.

Installing it is easy;

> emacs -batch --eval '(byte-compile-file "tabbar.el")'
# install -m 644 tabbar.elc /usr/local/share/emacs/site-lisp

Activating it requires adding the following to my emacs.el;

(load "tabbar.elc")
(tabbar-mode t)
(require 'uniquify)
(setq uniquify-buffer-name-style 'forward)

The last two lines make buffer names unique. I discovered this little gem on the excellent emacs-fu blog. Above is shown how Emacs looks like with the tab bar.

Backing up to another machine using dump and netcat

2009–05–17

If you are cramped for disk space, and you still want to make backups, here is a possible solution. It does require that you have the machine to be backed up and another machine with plenty of disk space connected to a network. The second machine does not need to run FreeBSD, but it must be able to run nc (a.k.a. netcat).

Suppose the machine to be backed up is named foo, while the receiving machine has the hostname bar. On foo, you need to be logged in as the root user, because you are using dump. Start by running the following command on bar;

> nc -l 65000 |bzip2 -c >root.dump.bz2

After this command is started, run the following command on foo;

# dump -0 -a -C 8 -L -u -f - / | nc bar 65000

What happens is that on bar the netcat process is listening on port 65000. You can safely use pretty much any number over 30000. The dump process on foo writes its output to stdout, wich is then captured by netcat and sent to the netcat process listening on the host bar.

On the receiving end, I’m piping the output from netcat through bzip2 to compress it. That could also have been done on the sending side, but in this case the receiving side has more computing power, and both machines are connected by a 100 Mbit/s link, so bandwidth isn’t an issue.

If you want to move big pieces of data over the network as fast as possible, netcat is the tool for the job. When just moving big files, and if it doesn’t have to wait on other programs, I’ve seen it saturate a 100 Mbit/s link.

2008

Using ispell in batch mode

2008–09–06

Running ispell on a file interactively can be quite a drag. It is easy to make a mistake when correcting things, and even when you specify a format (like -t for (La)TeX), ispell is still prone to check things that you don’t want to have checked. So I prefer to run it in batch mode using the -l option;

> ispell -t -l -W 4 -C -d nederlands <dutch.tex | sort | uniq | less

This example is of a Dutch text, so I’ve specified the Dutch dictionary (-d nederlands) and the -C option is used because running words together is common in Dutch as well.

The source for this webpage has been checked with the following command:

> ispell -l -W 4 -C <misc.md | sort | uniq | less

This produces a list of words that ispell thinks are wrong. I can then scan this list and fish out the real spelling mistakes. E.g, ispell rightly marks the word ‘mencoder’ and ‘sysctl’ as misspelled because they are not English words. They are however a names of existing programs.

ERP systems

2008–07–31

In my 15 years at $BIGCORP we went from a homebuilt system to MFG/Pro to Baan and now SAP. Every move was a turn for the worse, IMO. Every new “solution” was more expensive and more time-consuming than the previous one. Stay away from the big vendors, especially if you have a small business, as they will try to suck you dry.

What I learned from these processes was:

1. Keep It Simple! Avoid the temptation to build a singing and dancing do-it-all system. Get a system that is modular so you can add things later.
2. Think it through before you implement. What do you need, and what would you like to have? Chart your processes. How do you want to track them?
3. Implementation is key. All ERP systems are basically capable of doing the same things. But how you set them up makes the difference between a good solution and a can of worms.
4. Most EPR systems can be extensively customized to suit your needs. This is very expensive, though.

Some open-source systems to look at:

• Open ERP
• compiere
• Openbravo

Installing TeXLive 2008 on FreeBSD

2008–01–22

This release of TeXLive includes FreeBSD binaries even for amd64, yay! So the basic install is a question of mounting the DVD, then;

# cd /cdrom1/texlive/
# ./install-tl

This will take you through a set of menus. I set the main TeX directory to /usr/local/texlive/2008 and chose which packages to install et cetera. The install itself went extremely smooth.

I did have to make some additional changes in my various configuration files;

• Change the path for the info files in emacs.el to include /usr/local/texlive/2008/texmf/doc/info.
• I like to be able to use TeX fonts in Ghostscript, so I had to change /etc/profile and /etc/csh.cshrc to point the variable GS_FONTPATH to point to the new paths for the tex-gyre and latin modern (lm) fonts.
• In /etc/login.conf, I had to change the path setting for the default user to point to the directory for the new binaries, /usr/local/texlive/2008/bin/amd64-freebsd.
• In /etc/manpath.config I had to add an OPTIONAL_MANPATH pointing to the manual pages for the TeXLive program in /usr/local/texlive/2008/texmf/doc/man. Additionally, I added a MANPATH_MAP to link binaries in /usr/local/texlive/2008/bin/amd64-freebsd to the aforementioned MANPATH.

After installing the various updated configuration files, logging out completely and logging in again to get the new path, TeX worked fine.

2007

Modifying the auctex port to work with TeXLive on FreeBSD

2007–12–23

Since TeXLive is not in the FreeBSD ports tree, installing auctex pulls in teTeX. This is not what I wanted, so I had to patch /usr/ports/print/auctex/Makefile;

--- Makefile.orig       2007-12-23 14:48:03.000000000 +0100
+++ Makefile    2007-12-23 14:50:31.000000000 +0100
@@ -16,8 +16,8 @@
 MAINTAINER=    hrs@FreeBSD.org
 COMMENT=       Integrated environment for writing LaTeX using GNU Emacs

-BUILD_DEPENDS= ${MKTEXLSR}:${PORTSDIR}/print/teTeX-base
-RUN_DEPENDS=   ${MKTEXLSR}:${PORTSDIR}/print/teTeX-base
+#BUILD_DEPENDS=        ${MKTEXLSR}:${PORTSDIR}/print/teTeX-base
+#RUN_DEPENDS=  ${MKTEXLSR}:${PORTSDIR}/print/teTeX-base

 USE_GHOSTSCRIPT=yes
 GNU_CONFIGURE= yes
@@ -33,8 +33,8 @@
                MKTEXLSR=${MKTEXLSR}
 INFO=          auctex preview-latex

-TEXMFDIR=      share/texmf
-MKTEXLSR=      ${LOCALBASE}/bin/mktexlsr
+TEXMFDIR=      texlive/2007/texmf
+MKTEXLSR=      ${LOCALBASE}/texlive/2007/bin/x86_64-unknown-freebsd7.0/mktexlsr

 NOT_FOR_ARCHS= ia64

The variables TEXMFDIR and MKTEXLSR should be adapted to suit the TeXLive install.

Using a Kensington Expert Mouse trackball with Xorg

2007–07–20

Instead of a mouse, I prefer using a trackball. But a lot of trackballs are not usable for left-handed people like myself. The one I’m currently using is a Kensington Expert Mouse. I like it because it has four buttons and a scroll-ring.

To use this trackball with the Xorg server, I put the following in /etc/X11/xorg.conf;

Section "InputDevice"
        Identifier  "ExpertMouse"
        Driver      "mouse"
        Option      "Protocol"          "Auto"
        Option      "Device"            "/dev/psm0"
    Option      "Buttons"           "6"
    Option      "ZAxisMapping"      "4 5"
    Option      "ButtonMapping"     "3 2 1"
        Option      "CorePointer"       "on"
EndSection

The device /dev/psm0 is specific for FreeBSD. Linux users will want to use /dev/mouse.

The ButtonMapping option is used to switch buttons one and three because I’m left-handed. With this option, the button on the lower-right is button one, lower-left is button three and top-left is button two. Rotating the scroll-ring clockwise is button five, anti-clockwise is button four.

Using the Redglass cursor theme in the X Window System

2007–06–24

In ~/.xinitrc I had to add the following line;

export XCURSOR_THEME=redglass

In ~/.Xresources;

Xcursor.theme: redglass

At one point I noticed that the wait cursor in Firefox was broken. I fixed that with the following commands;

# cd /usr/local/lib/X11/icons/redglass/cursors
# ln -s left_ptr_watch 08e8e1c95fe2fc01f976f1e063a24ccd

Encrypting my /home partition with FreeBSD

2007–06–16

After having a computer stolen in a burglary, I decided that I should encrypt my data on-disk, to keep other people from poking their noses in my data should something like this ever happen again. Since my data is concentrated in the /home directory which I always put on a separate partition, I only have to encrypt a single partition.

First thing to do was make a backup! That is the only way to prevent data loss in case you screw something up. I mounted a UFS2 formatted USB connected harddisk on /mnt/root, and used the following command to perform the backup;

# dump -0 -a -C 16 -f - -L /home |gzip -c >/mnt/root/home.gz

This took about two hours to finish. The file home.gz is 63786 MiB. The data on the /home is 69167 MiB, so the compression saved 7.7%. I might not bother with compressing it next time.

Before going further, I tested that I could recover data from the backup with the restore command in interactive mode;

# gzip -c /mnt/root/dumps/home.gz|restore -if -

Having veryfied that the backup was OK, I unmounted the partition, and filled it with random garbage so that the encrypted data that is written later will not be easily recognizable.

# umount /home
# dd if=/dev/random of=/dev/ar0s1g bs=32k

Writing the random data took about 65 minutes for 120 GiB.

Now the encrypted device is to be created and attached. This requires that that the options GEOM_ELI is compiled into the kernel (together with the device crypto), or that the kernel modules geom_eli.ko is loaded. The latter can be done by hand with the kldload(8) command, but it is more convenient to also add the following line to /boot/loader.conf:

geom_eli_load="YES"

This ensures that the module is loaded on the next boot. That being done, the geli(8) command is used to initialize and attach the encrypted device:

# geli init -l 256 /dev/ar0s1g
# geli attach /dev/ar0s1g

Both commands will ask for a passphrase that is used to derive the encryption key from. After the ‘geli attach’ command, a new device will appear; /dev/ar0s1g.eli. A filesystem is created on this device, and the previously made backup is restored.

# newfs -U /dev/ar0s1g.eli
# mount /dev/ar0s1g.eli /home
# cd /home
# gzip -c /mnt/root/dumps/home.gz|restore -rf -

The last thing to do is to update /etc/fstab, and change the device for /home from /dev/ar0s1g to /dev/ar0s1g.eli. FreeBSD will recognize the encrypted partition and ask for the passphrase before mounting the filesystem.

Making a backup of CDs

2007–06–08

For details of how a CD stores data, see the CD-ROM page on Wikipedia. A data CD-ROM has sectors that are effectively 2048 bytes. So a good way to backup a CD-ROM is to use dd(1);

> dd if=/dev/cd0 of=my_cd.iso bs=2k

The input/output block size (bs) can also be chosen a multiple of 2k to increase reading speed, but it cannot be made smaller.

For backups of music CDs, I tend to use cdparanoia, and then convert the resulting WAV files to FLAC format. These are about ½ the size of the original.

2006

Giving CUPS access to the printer on FreeBSD

2006–12–02

When using CUPS like I am, its programs need to have access to the printer device. In my case connected to the parallel port;

# chown root:cups /dev/lpt0
# chmod g+rw /dev/lpt0

To make this change permanent, you can add the following rules to /etc/devfs.conf;

# Give cups printer access
own     lpt0    root:cups
perm    lpt0    0660

This will make sure that the relevant group and permissions will be installed after the next reboot.

In case you have a USB connected printer, you should add a rule to devfs.rules for a ulpt(4) device.

The printer now works fine with CUPS. One caveat I came across, is that you need to use the command-line tool lpadmin(8) to install a printer with a PPD file. I used the following command to install my printer in CUPS:

# /usr/local/sbin/lpadmin -E -p clj2550 -v parallel:/dev/lpt0 -P clj2550.ppd

Converting video-CD to DVD

2006–06–25

I have some old video CDs that I cannot play in my standalone DVD player. The following commands were used to convert them to DVD.

First the two VCDs are ripped to disk using vcdxrip from the vcdimager suite;

> #(Put the first disk in the drive)
> vcdxrip --nofiles -C /dev/cd1
> rm videocd.xml
> mv avseq01.mpg part1.mpg
> #(Put the second disk in the drive)
> vcdxrip --nofiles -C /dev/cd1
> rm videocd.xml
> mv avseq01.mpg part2.mpg

Combine the two MPEG files into a single file in DVD format using mencoder;

> mencoder -oac lavc -ovc lavc -of mpeg -mpegopts format=dvd \
-vf scale=720:576,harddup \
-srate 48000 -af lavcresample=48000 \
-lavcopts vcodec=mpeg2video:vrc_buf_size=1835:vrc_maxrate=9800:\
vbitrate=5000:keyint=15:aspect=4/3:mbd=1:\
acodec=ac3:abitrate=192 -ofps 25 -o dvd.mpg part1.mpg part2.mpg
> rm part1.mpg part2.mpg

Then the dvdauthor program is used to create a DVD structure, which is then transferred to DVD with growisofs;

> cat >dvd.xml <<EOF
<dvdauthor>
  <vmgm />
    <titleset>
      <titles>
        <pgc>
          <vob file="dvd.mpg" />
        </pgc>
      </titles>
    </titleset>
</dvdauthor>
EOF
> dvdauthor -o dvd -x dvd.xml
> growisofs -dvd-compat -Z /dev/cd1 -dvd-video dvd/
> rm -rf dvd/
> rm dvd.xml

postscript 2009–12–28: To create a video using the compact H.264 video codec, use the following command;

> mencoder -profile hqmovie -vf scale -zoom -xy 720 -idx \
-o red_planet.avi part1.mpg part2.mpg

The hqmovie profile is the described in the section 2009–09–02: Re-encoding DVDs with mencoder.

Using a Logitech Trackman Marble with Xorg

2006–05–13

Using a mouse was giving me wrist trouble, so I bought a Logitech Trackman Marble (also known as the Marble Mouse) trackball;

To use this with Xorg, I use the following InputDevice section in xorg.conf;

Section "InputDevice"
        Identifier  "MarbleMouse"
        Driver      "mouse"
        Option      "Protocol"          "Auto"
        Option      "Device"            "/dev/psm0"
        Option      "Buttons"           "5"
        Option      "ZAxisMapping"      "4 5"
        Option      "ButtonMapping"     "3 2 1 6 7"
        Option      "CorePointer"       "on"
EndSection

(For Linux, you probably need to change the mouse device node.) Additionally, I changed ~/.Xmodnap to get the scroll buttons to work properly;

pointer = 1 2 3 6 7 4 5 8 9 10 11

Setting up Xorg to use UTF–8

2006–05–03

To get Xorg to use UTF–8, I added the following to ~/.xinitrc;

export LANG=en_US.UTF-8

And I removed the command setting the standard keyboard map to ‘us’.

Additionally, I had to add the following to ~/.Xresources to get xterm to use a unicode font;

XTerm*utf8: 2
XTerm*font: -misc-fixed-medium-r-normal--14-130-75-75-c-70-iso10646-1

A similar change had to be made to the configuration file for the mutt e-mail client, ~/.muttrc, changing

set charset="iso-8859-15"

to

set charset="utf-8"

Changing from subversion to git

2006–02–18

By default, git doesn’t use keywords like rcs and cvs and subversion. So when I converted my projects from subversion to git, I replaced the $Id$ tags by the ‘Time-stamp’ that emacs can generate;

find . -type f|xargs sed -i "" -e 's|\$Id.*\$|Time-stamp: <2006-02-18 21:37:00 rsmith>|g'

Adding a copyright notice to photos

2006–01–13

For clarification I like to add a copyright notice to my pictures, especially the ones that I might publish. To this end I use the mogrify command from the ImageMagick suite;

> chmod 644 *.jpg
> mogrify -comment "Copyright © 2009 R. Smith <rsmith@xs4all.nl>" *.jpg
> jhead -ft *.jpg
> chmod 444 *.jpg

The jhead command modifies the file time to match the time that the picture was taken, according to the exif data in the file. Note that not all picture formats allow comments to be attached, but JPEG and PNG both work.

This copyright notice is not visible when you look at the picture. But it is embedded in the file. The ImageMagick annotation tutorial shows you how to add visible annotations to pictures.

Postscript 2009–12–24: There is a program called ExifTool that can edit EXIF, IPTC and XMP tags. I’ve downloaded it but haven’t used it yet.

2004

Making a PDF from scanned B/W pages

2004–12–08

First, scan the pages at 300 dpi. Save the scanned images in PNM format. Convert them to a multipage TIFF file with ImageMagick:

convert -adjoin -density 300x300 -colors 2 `ls *.pnm` output.tiff

Convert the TIFF file to PDF format:

tiff2pdf -r300 -o output.pdf output.tiff

2003

Printing labels with LaTeX

2003–11–16

I bought a packet of 500 labels, see below. On the left is the front of the package. A page with four labels is shown in the middle.

These labels were probably originally intended for a matrix printer since the pages are strung together. The page boundary is perforated, so they are easy to separate. I can print them just fine on my laser printer.

The LaTeX code for typesetting the labels shown on the right in the previous picture is shown below..

% -*- latex -*-
% LaTeX sourcecode file for printing 89x36 mm stickers (four on a page).
% Written by R.F. Smith <rsmith@xs4all.nl> in 2003 and placed in the 
% public domain
%%%%%%%%%% Normally no changes required below here %%%%%%%%%%
\documentclass[a4paper,12pt]{article}
\usepackage[utf8]{inputenc}
\usepackage[T1]{fontenc}
\usepackage{tgpagella} % Default fonts
\usepackage{calligra} % Font for the addresses
\usepackage[newdimens]{labels}
\LabelGridtrue
% Settings for the label package; 4 89x36 stickers.
\LabelCols=1
\LabelRows=4
\LeftPageMargin=60.5mm
\RightPageMargin=60.5mm
\TopPageMargin=1mm
\BottomPageMargin=150mm
\InterLabelRow=3mm
% Create an empty label; for if the label has already been used
\newcommand{\emptylabel}{\rule{0mm}{22mm}}
%%%%%%%%%%%%%%%%%%%%%% Start of the document %%%%%%%%%%%%%%%%%%%%%%%%
\begin{document}
\addresslabel{%
  \texttt{Mr S.\,Holmes\\
  221b Baker Street\\
  London NW1\,6XE\\
  England}
}
\addresslabel{%
  \begin{tabular}{ll}% l,c,r
    Hostname: & \texttt{guardian}\\
    Function: & \texttt{router \& firewall}\\
    IP-address: & \texttt{192.168.2.173}\\
  \end{tabular}
}
\addresslabel{\centering{\calligra\Huge Dry Beans}\\(phaseolus vulgaris)}
\addresslabel{\centering\Huge \LaTeX{} rules!}
%%%%%%%%%%%%%% End of the document contents %%%%%%%%%%%%%%%%%%%
\end{document}

The typeset label is shown on the right in the previous picture. The frames around the individual labels are drawn because the \LabelGridtrue command was used to indicate the size of the labels. For printing one would normally not want to have these frames, and the line containing that command should be removed or commented out by putting a % in front of it.

—– Copyright © 2010, Roland Smith rsmith@xs4all.nl

This <span xmlns:dc=“http://purl.org/dc/elements/1.1/” href=“http://purl.org/dc/dcmitype/Text” rel=“dc:type”>work is licensed under a Creative Commons Attribution 3.0 Unported License