Updating Python to 3.5

This article documents the process I followed to update Python 3.4 to 3.5 on my FreeBSD machines.

Introduction

The goal is to replace Python 3.4 and all the packages that depend on it by Python 3.5. Normally I would use portmaster for updating but at the time of writing portmaster can’t deal with installing a port for two different Python versions. During an update, it will e.g. suggest to update py34-nose-1.3.6 to py27-nose-1.3.7, probably because Python 2.7 is still the default version. This is however not what I want.

So fo the moment my solution is to do the updates by hand, because then I can specifiy I want to build a port for Python 3.5 by setting the PYTHON_VERSION variable in the make command.

Note

In the following code fragments, an ellipsis “…” denotes that some lines of output have been removed for brevity.

Investigate and backup

First priority was to see which Python 3.4 packages were installed:

> pkg info -x py34
py34-Jinja2-2.8
py34-MarkupSafe-0.23
py34-Wand-0.4.1
py34-cairo-1.10.0_3
py34-certifi-2015.04.28
py34-gobject3-3.16.2_1
py34-ipython-3.2.1
py34-jsonschema-2.4.0
py34-mistune-0.5
py34-nose-1.3.7
py34-numpy-1.9.2,1
py34-pep8-1.5.7
py34-pexpect-3.3
py34-pillow-2.9.0
py34-pip-7.0.3
py34-pycrypto-2.6.1_1
py34-pytz-2015.4,1
py34-pyzmq-14.7.0
py34-requests-2.7.0
py34-setuptools34-17.0
py34-sqlite3-3.4.3_6
py34-tornado-4.2.1

Just to be sure, backup packages for all these ports were created:

> pkg create py34-Jinja2-2.8 py34-MarkupSafe-0.23 \
py34-Wand-0.4.1 py34-cairo-1.10.0_3 py34-certifi-2015.04.28 \
py34-gobject3-3.16.2_1 py34-ipython-3.2.1 py34-jsonschema-2.4.0 \
py34-mistune-0.5 py34-nose-1.3.7 py34-numpy-1.9.2,1 py34-pep8-1.5.7 \
py34-pexpect-3.3 py34-pillow-2.9.0 py34-pip-7.0.3 py34-pycrypto-2.6.1_1 \
py34-pytz-2015.4,1 py34-pyzmq-14.7.0 py34-requests-2.7.0 \
py34-setuptools34-17.0 py34-sqlite3-3.4.3_6 py34-tornado-4.2.1
Creating package for py34-Jinja2-2.8
…
Creating package for py34-tornado-4.2.1

Looking at the Makefiles of all of these ports, I checked that they are compatible with Python 3.5. Upon concluding that, I removed the 3.4 packages:

# pkg delete -f py34-Jinja2-2.8 py34-MarkupSafe-0.23 \
?     py34-Wand-0.4.1 py34-cairo-1.10.0_3 py34-certifi-2015.04.28 \
?     py34-gobject3-3.16.2_1 py34-ipython-3.2.1 py34-jsonschema-2.4.0 \
?     py34-mistune-0.5 py34-nose-1.3.7 py34-numpy-1.9.2,1 py34-pep8-1.5.7 \
?     py34-pexpect-3.3 py34-pillow-2.9.0 py34-pip-7.0.3 py34-pycrypto-2.6.1_1 \
?     py34-pytz-2015.4,1 py34-pyzmq-14.7.0 py34-requests-2.7.0 \
?     py34-setuptools34-17.0 py34-sqlite3-3.4.3_6 py34-tornado-4.2.1
Updating database digests format: 100%
Checking integrity... done (0 conflicting)
Deinstallation has been requested for the following 22 packages (of 0 packages in the universe):

Installed packages to be REMOVED:
    py34-Jinja2-2.8
    …
    py34-tornado-4.2.1

The operation will free 69 MiB.

Proceed with deinstalling packages? [y/N]: y
…

At this point I used pip-3.4 list to check which Python modules were installed outside of the ports tree:

> pip-3.4 list
You are using pip version 7.0.3, however version 7.1.2 is available.
You should consider upgrading via the 'pip install --upgrade pip' command.
icalendar (3.9.1)
line-profiler (1.0)
python-magic (0.4.6)

As an aside, Trying to run the “upgrade” command as suggested by pip failed. Don’t try that!

Configuration

In /etc/make.conf, I changed the default Python 3 version to 3.5:

DEFAULT_VERSIONS= ruby=2.1 python3=3.5

In my vimrc, I set the paths for the python checkers:

let g:syntastic_python_python_exec='/usr/local/bin/python3'
let g:syntastic_python_pep8_exec='/usr/local/bin/pep8'

Note that I removed the Python2 version of pep8.

Installing Python 3.5

First I installed the new Python 3.5:

# cd /usr/ports/lang/python35/
# make PYTHON_VERSION=python3.5 install clean

The build process make me choose options:

┌────────────────────────────── python35-3.5.0 ────────────────────────────────┐
│ ┌──────────────────────────────────────────────────────────────────────────┐ │
│ │+[ ] DEBUG     Build with debugging support                               │ │
│ │+[x] IPV6      IPv6 protocol support                                      │ │
│ │+[x] LIBFFI    Use libffi from ports instead of bundled version           │ │
│ │+[x] NLS       Enable gettext support for the locale module               │ │
│ │+[x] PYMALLOC  Enable specialized mallocs                                 │ │
│ │+[x] THREADS   Threading support                                          │ │
│ │+[x] TSC       Enable processor timestamp counter profiling               │ │
│ │───────────────────────── Hash Algorithm (PEP-456) ───────────────────────│ │
│ │+( ) FNV       Modified Fowler-Noll-Vo Algorithm                          │ │
│ │+(*) SIPHASH   SipHash24 Algorithm                                        │ │
│ └──────────────────────────────────────────────────────────────────────────┘ │
├──────────────────────────────────────────────────────────────────────────────┤
│                       <  OK  >            <Cancel>                           │
└──────────────────────────────────────────────────────────────────────────────┘

The only non-default options I chose were TSC and SIPHASH.

After that the build and install proceeded normally. The only thing that struck me was a significant amount of “unreachable code” warnings when compiling. The program seemed to work normally.

After that I deleted the python34-3.4.3_1 program:

# pkg delete python34-3.4.3_1
Updating database digests format: 100%
Checking integrity... done (0 conflicting)
Deinstallation has been requested for the following 2 packages (of 0 packages in the universe):

Installed packages to be REMOVED:
    python34-3.4.3_1
    python3-3_3

The operation will free 78 MiB.

Proceed with deinstalling packages? [y/N]: y
[1/2] Deinstalling python3-3_3...
[1/2] Deleting files for python3-3_3: 100%
[2/2] Deinstalling python34-3.4.3_1...
[2/2] Deleting files for python34-3.4.3_1: 100%

There were some manually installed modules left so I had to remove the Python 3.4 directory by hand:

# rm -rf /usr/local/lib/python3.4

And I re-installed the lang/python3 port:

# cd /usr/ports/lang/python3
# make install clean
===>   python3-3_3 depends on file: /usr/local/sbin/pkg - found
===> Fetching all distfiles required by python3-3_3 for building
===>  Extracting for python3-3_3
===>  Patching for python3-3_3
===>  Configuring for python3-3_3
===>  Staging for python3-3_3
===>   python3-3_3 depends on file: /usr/local/bin/python3.5 - found
===>   python3-3_3 depends on file: /usr/local/bin/python3.5 - found
===>   Generating temporary packing list
/bin/ln -sf python3.5  /usr/ports/lang/python3/work/stage/usr/local/bin/python3
/bin/ln -sf 2to3-3.5  /usr/ports/lang/python3/work/stage/usr/local/bin/2to3-3
/bin/ln -sf idle3.5  /usr/ports/lang/python3/work/stage/usr/local/bin/idle3
/bin/ln -sf pydoc3.5  /usr/ports/lang/python3/work/stage/usr/local/bin/pydoc3
/bin/ln -sf python3.5-config  /usr/ports/lang/python3/work/stage/usr/local/bin/python3-config
/bin/ln -sf python-3.5.pc  /usr/ports/lang/python3/work/stage/usr/local/libdata/pkgconfig/python3.pc
====> Compressing man pages (compress-man)
===>  Installing for python3-3_3
===>  Checking if python3 already installed
===>   Registering installation for python3-3_3
Installing python3-3_3...
===>  Cleaning for python3-3_3

Installing the Python 3 ports

Starting with the most important ones:

# cd /usr/ports/math/py-numpy/
# make PYTHON_VERSION=python3.5 install clean

# cd /usr/ports/graphics/py-wand/
# make PYTHON_VERSION=python3.5 install clean

cd /usr/ports/databases/py-sqlite3
# make PYTHON_VERSION=python3.5 install clean

# cd /usr/ports/devel/py-nose
# make PYTHON_VERSION=python3.5 install clean

# cd /usr/ports/devel/py-pytz
# make PYTHON_VERSION=python3.5 install clean

# cd /usr/ports/devel/pep8
# make PYTHON_VERSION=python3.5 install clean

# cd /usr/ports/www/py-requests
# make PYTHON_VERSION=python3.5 install clean


# cd /usr/ports/devel/ipython/
# make PYTHON_VERSION=python3.5 install clean

Installing IPython also takes care of pexpect, tornado, certifi, Jinja2, MarkupSafe, jsonschema, mistune and pyzmq.

More ports:

# cd /usr/ports/graphics/py-pillow/
# make PYTHON_VERSION=python3.5 install clean

# cd /usr/ports/security/py-pycrypto
# make PYTHON_VERSION=python3.5 install clean

# cd /usr/ports/devel/py-pip
# make PYTHON_VERSION=python3.5 install clean

The graphics/py3-cairo port produces an error, so I’ll skip that for now. Also I haven’t used gobject, so I’ll skip that as well.

Python 3 modules that are not available from ports

Some of the modules I use are not in the FreeBSD ports tree. Those I installed by hand. For this I used pip because these are pure Python modules.

# pip-3.5 install python-magic

# pip-3.5 install line_profiler

# pip-3.5 install icalendar

For comments, please send me an e-mail.

← Python bindings for libmagic Replacing openssl with libressl for ports →

Roland's homepage

Updating Python to 3.5

Introduction

Investigate and backup

Configuration

Installing Python 3.5

Installing the Python 3 ports

Python 3 modules that are not available from ports

Related articles