Roland's homepage

My random knot in the Web

Attempting a conky replacement in Python (part 1)

My window manager (i3) has a small bar on the bottom of the screen. I use conky to put a single line of (text) information there.

It looks like this:

Ext: 34.1KiB/910B  Int: 0B/0B  | Mail: 51 | RAM: 35% | CPU: 4%, 49°C | Mon 2019-07-01 21:20:17

This is refreshed every second. Conky is a pretty substantial program that can do a lot more than I ask to do. Recently it added Lua as an embedded programming language. So my question was; can I write a simple replacement in Python 3?

This program uses FreeBSD specific commands like sysctl and netstat. (Well, OS X is based on a FreeBSD userland. so they might exist there as well. But you probably won’t be running i3 on OS X. :-)

Simple implementation

My first implementation is pretty straightforward. It calls a number of functions, and joins the outputs with |.

items = [network(), mail(), memory(), cpu(), date()]
print(' | '.join(items))

Let’s see how fast this runs using IPython.

Network

In [1]: import os
...: import statistics as stat
...: import subprocess as sp
...: import time

In [2]: # Global data
...: netdata = {'age0': (0, 0, 0), 'rl0': (0, 0, 0)}
...: lastmail = {'count': -1, 'time': 0}
...: cpusage = {'used': 0, 'total': 0}
Out[2]: {'used': 0, 'total': 0}

In [3]: def netstat(interface):
...:     """Return network statistics for named interface."""
...:     lines = sp.check_output(['netstat', '-b', '-n', '-I', interface],
...:                             encoding='ascii').splitlines()
...:     tm = time.time()
...:     if len(lines) == 1:
...:         return None
...:     items = lines[1].split()
...:     ibytes = int(items[7])
...:     obytes = int(items[10])
...:     return ibytes, obytes, tm
...:

In [4]: %timeit netstat('age0')
79.9 ms ± 488 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)

In [5]: def network():
...:     global netdata
...:     ifaces = {'Ext': 'age0', 'Int': 'rl0'}
...:     items = []
...:     for desc, i in ifaces.items():
...:         data = netstat(i)
...:         if data:
...:             dt = data[2] - netdata[i][2]
...:             delta_in = int((data[0] - netdata[i][0])/dt)
...:             delta_out = int((data[1] - netdata[i][1])/dt)
...:             netdata[i] = data
...:             items.append(f'{desc}: {delta_in}B/{delta_out}B')
...:         else:
...:             items.append(f'{desc}: 0B/0B')
...:     return ' '.join(items)
...:

In [6]: %timeit network()
160 ms ± 638 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)

It is clear that almost all of the time in network() is spent in netstat(). I suspect that most of that time is spent calling the subprocess. Let’s check that.

In [7]: %timeit sp.check_output(['netstat', '-b', '-n', '-I', 'age0'], encoding='ascii').splitlines()
79.3 ms ± 403 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)

Almost all of the time in netstat is indeed spent running the subprocess.

Mail

Initially I used regular expressions for parsing the mbox file. This turned out to be very slow. So I quickly changed that to looking at the beginning of each string.

In [8]: def mail():
...:     mboxname = '/home/rsmith/Mail/received'
...:     global lastmail
...:     newtime = os.stat(mboxname).st_mtime
...:     if newtime > lastmail['time']:
...:         with open(mboxname) as mbox:
...:             lines = mbox.readlines()
...:         # This is faster than regex; see above.
...:         read = len([ln for ln in lines if ln.startswith('Status: R')])
...:         total = len([ln for ln in lines if ln.startswith('From ')])
...:         mailcount = total - read
...:         lastmail = {'count': mailcount, 'time': newtime}
...:     return f'Mail: {mailcount}'
...:

In [9]: %timeit -n 1 -r 1 mail()
333 ms ± 0 ns per loop (mean ± std. dev. of 1 run, 1 loop each)

Since this takes around 1/3 of a second, we only do this if the modification time of the mail file is newer than the last time we checked. Still, this is pretty slow. I should probably look at using mmap.

Memory

In [10]: def vmstats():
    ...:     """Return FreeBSD's vm stats."""
    ...:     rv = {}
    ...:     for ln in sp.check_output(['sysctl', 'vm.stats.vm'], encoding='ascii').splitlines():
    ...:         name, count = ln.split()
    ...:         rv[name[14:-1]] = int(count)
    ...:     return rv
    ...:

In [11]: %timeit vmstats()
80.5 ms ± 278 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)

In [12]: def memory():
    ...:     stats = vmstats()
    ...:     memmax = stats['page_count']
    ...:     mem = (memmax - stats['free_count'] - stats['inactive_count'] - stats['cache_count'])
    ...:     free = int(100 * mem / memmax)
    ...:     return f'RAM: {free}%'
    ...:

In [13]: %timeit memory()
81.9 ms ± 620 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)

In [14]: %timeit sp.check_output(['sysctl', 'vm.stats.vm'], encoding='ascii').splitlines()
80.3 ms ± 83.7 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)

As expected based on the previous items, most of the time is spent in the subprocess.check_output call.

CPU load and temperature

In [15]: def cpu():
    ...:     global cpusage
    ...:     items = [f'dev.cpu.{n}.temperature' for n in range(4)] + ['kern.cp_time']
    ...:     lines = sp.check_output(['sysctl'] + items, encoding='ascii').splitlines()
    ...:     temps = [float(ln.split()[1][:-1]) for ln in lines[:-1]]
    ...:     T = round(stat.mean(temps))
    ...:     states = [int(j) for j in lines[-1].split()[1:]]
    ...:     # According to /usr/include/sys/resource.h, these are:
    ...:     # USER, NICE, SYS, INT, IDLE
    ...:     total = sum(states)
    ...:     used = total - states[-1]
    ...:     frac = int((used - cpusage['used']) / (total - cpusage['total']) * 100)
    ...:     cpusage = {'used': used, 'total': total}
    ...:     return f'CPU: {frac}%, {T}°C'
    ...:

In [16]: %timeit cpu()
82.2 ms ± 169 µs per loop (mean ± std. dev. of 7 runs, 10 loops each)

In [18]: %timeit sp.check_output(['sysctl'] + items, encoding='ascii').splitlines()
83.4 ms ± 2.92 ms per loop (mean ± std. dev. of 7 runs, 10 loops each)

Here again we see that most of the time is spent calling the subprocess.

Date and time

This is trivial, and very fast.

In [19]: def date():
    ...:     return time.strftime('%a %Y-%m-%d %H:%M:%S')
    ...:

In [20]: %timeit date()
4.79 µs ± 19 ns per loop (mean ± std. dev. of 7 runs, 100000 loops each)

Total

Let’s run the complete program and see how long it takes.

> python3 statusline-i3.py
Ext: 4B/0B Int: 0B/0B | Mail: 51 | RAM: 36% | CPU: 4%, 54°C | Mon 2019-07-01 22:05:45
DEBUG: runtime = 0.62 s
Ext: 0B/0B Int: 0B/0B | Mail: 51 | RAM: 36% | CPU: 10%, 53°C | Mon 2019-07-01 22:05:46
DEBUG: runtime = 0.29 s
Ext: 0B/0B Int: 0B/0B | Mail: 51 | RAM: 36% | CPU: 7%, 54°C | Mon 2019-07-01 22:05:47
DEBUG: runtime = 0.29 s
Ext: 0B/0B Int: 0B/0B | Mail: 51 | RAM: 36% | CPU: 6%, 53°C | Mon 2019-07-01 22:05:48
DEBUG: runtime = 0.28 s
Ext: 0B/0B Int: 0B/0B | Mail: 51 | RAM: 36% | CPU: 7%, 53°C | Mon 2019-07-01 22:05:49
DEBUG: runtime = 0.29 s
Ext: 0B/0B Int: 0B/0B | Mail: 51 | RAM: 36% | CPU: 7%, 53°C | Mon 2019-07-01 22:05:50
DEBUG: runtime = 0.29 s
Ext: 0B/0B Int: 0B/0B | Mail: 51 | RAM: 36% | CPU: 7%, 53°C | Mon 2019-07-01 22:05:51
DEBUG: runtime = 0.29 s
Ext: 0B/0B Int: 0B/0B | Mail: 51 | RAM: 36% | CPU: 6%, 52°C | Mon 2019-07-01 22:05:52
DEBUG: runtime = 0.29 s

Here, “runtime” is the (clock, not CPU) time that one iteration takes. After startup, the program stabilizes at approximately 0.3 s per iteration. The rest of the time (up to the 1 second refresh time) it sleeps. One could say that this program is fast enough, since it has more than enough time to do its job.

The CPU usage is too high for my taste, though. A simple status line should not require 7% of CPU time!

Conclusions for now

Parsing the mbox file takes the most time by far. But this is only done intermittently.

The rest of the time is mostly spent calling external applications lite netstat and sysctl.

In part 2 we’re going to make this faster.


←  SHA256 in pure Python Attempting a conky replacement in Python (part 2)  →