Tuesday, May 29, 2018

Building Python 2.7.15

On this post I'll document how to build the (up to this writing) latest version of Python version 2 series. But instead of doing it under a 64-bits Solaris 11.3 GA, this time I'll do it under a 32-bits Solaris 11 Express SRU-14. I'll do so because Python has become an important technology upon which many other depend and because running 32-bits software is still important in the 3rd world where most of people remain on a deprived situation and there's not enough discarded 64-bits hardware for them to reuse. Running a 32-bits Solaris on these legacy hardware is pure gold them. This way deprived people have a chance of getting latest Python on legacy hardware they can afford. It's true one could argue, for instance, that running FreeBSD or even Linux could be more appealing to this scenario, but that's not Solaris, which IMO still is overall better than both of them.

$ python
Python 2.7.15 (default, May 27 2018, 21:38:19)
[GCC 3.4.3 (csl-sol210-3_4-20050802)] on sunos5
Type "help", "copyright", "credits" or "license" for more information.
>>>
 


$ cat /etc/release
            Oracle Solaris 11 Express snv_151a X86
     Copyright (c) 2010, Oracle ...  All rights reserved.
                  Assembled 04 November 2010


$ pkg info entire |grep Summary |sed 's/.*[\(]\(.*\)[\)].*/\1/'
Oracle Solaris 11 Express 2010.11 SRU 14


$ isainfo -v
32-bit i386 applications
        ssse3 ahf sse3 sse2 sse fxsr mmx cmov sep cx8 tsc fpu
 


Solaris 11 Express is a transitional step between Solaris 10 and 11. Although it lacks many features and goodies that would only be finally released under Solaris 11 it's already was much better than Solaris 10. In terms what is more relevant to this post, Solaris 11 Express lacks some necessary IPS packages, headers and still is attached to the legacy C++98 with GCC 3.4.3:

$ gcc --version
gcc (GCC) 3.4.3 (csl-sol210-3_4-20050802)
Copyright (C) 2004 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.  

There is NO warranty; not even for ...

In order to successfully accomplish this little mission, there's a few undocumented requirements and bugs that must be noticed and appropriately circumvented, all of which are not quite straightforward. They took me more than negligible effort to identify and address, but after documenting them I could it's easy to upgrade Python under Solaris 11 Express:

  • It's "necessary" or easier to just duplicate (by simple copying) the header file /usr/include/sys/audioio.h from a newer Solaris such as from an 11.3 GA. Although the basic interface audio(7I) is already in place, Solaris 11 Express lacks a lot of audio packages, it has only 14 against the 31 found on Solaris 11.3 GA as per the following IPS query:

    $ pkg search -r -H -o pkg.shortfmri \
      *:set:pkg.fmri:*audio* |sort -u


    Fortunately, the missing header is just a thin-wrapper of ioccom.h and this issue can be easily overcome by manually copying the missing header. This will allow configure to properly define HAVE_SYS_AUDIOIO_H as 1 in pyconfig.h:771, which in turn will define SOLARIS, which is need on ./Modules/sunaudiodev.c lines 8 and 17.
     
  • It's "necessary" to install the latest version 0.29.2 (up to this writing) of the pkg-config tool from FreeeDesktop.org, which is missing on Solaris 11 Express, but present on Solaris 11.3 GA under an IPS package of the same name for version 0.23.
      
  • Solaris 11 Express offers an IPS package for libffi version 3.0.9 which should work fine and is virtually the same found on Solaris 11.3 GA. One possible explanation for not having a more updated version under Solaris 11.3 GA could be related to some bug which prevents a successful build of newer minor versions such as 3.1 (bundled with Python 2.7.15) or 3.2.1. Or not, because I could build version 3.2.1 on an Solaris 11.3 GA with updated GNU tools and the default GCC 4.8.2 compiler. By the way, the latest minor version of branch 3.0 series, version 3.0.13. seems to work fine too! So I'll use this "latest" version 3.0.13 which I manually build and install beforehand. By the way, libffi is required by the Python's _ctypes module.
     
  • On Solaris 11 Express it will be necessary (at least for the options I use on configure) to make sure the package header-math is installed.
      
  • Perhaps it may be necessary to use gdbm instead of ndbm. If so it will be necessary to install the gdbm IPS package, fortunately on Solaris 11 Express.

NOTE
Even with all the previous tweaks the Python build will emit the following:

... build finished, but ... bits to build these ... not found:
_bsddb     bsddb185     linuxaudiodev
     ossaudiodev
..., look in setup.py in detect_modules() ...


From that I wonder if ossaudiodev could have been successfully built as well because some related documentation (audio_engine_start (9e)) is already available on Solaris 11 Express, in spite of the necessary header files being missing under /usr/include/sys/audio/. Most of these headers are just thin-wrappers, except for audio_driver.h which may reference some lacking or undocumented stub or alpha/beta implementation. But I don't pursue this any further because I'm not so interested on OSS audio under Solaris 11 Express. Anyway, I would expect Python's OSS module build fine under Solaris 11.3 GA. Ultimately, there seems to be alternatives: i) using a legacy SVr4 package for OSS 4.2 from OpenSound (it appears to be 32-bits and compatible with Solaris 11 as well according to its release notes) or ii) building from the source from 4Front Technologies.

Other points to consider in case you succeed with this much, is to redo everything after manually (building from source) updating Tcl/Tk and SQLite.  Sometimes it's a little disturbing to realize how Tcl/Tk, Perl and things like that are used in so many other stuff. On such cases, it may be necessary to consider configure options --with-tcltk-includes='-I... ' and --with-tcltk-libs='-L... ' and --with-libs='lib1 ...'.

Now for the build, again, the basic building strategy and general assumptions have been detailed on a previous post: Staged Building.

 $ pwd
/stage/build

$ ./gnu-build-preparation ../source/.../Python-2.7.15.tgz
...

$ cd Python/Python-2.7.15-32

$ source ../setenv-2.7.15 32

Configuration file [version 4]: /stage/build/Python/ld.config32 
  Platform:     32-bit LSB 80386
  Default Library Path (ELF):   /opt/gnu/lib:/opt/usr/lib:/lib:/usr/lib
  Trusted Directories (ELF):    /lib/secure:/usr/lib/secure  (system default)

Command line:
  crle -c /stage/build/Python/ld.config32 \

       -l /opt/gnu/lib:/opt/usr/lib:/lib:/usr/lib

LD_CONFIG_32=/stage/build/Python/ld.config32

CONFIG_SHELL=

CC=/usr/bin/gcc CFLAGS=-m32 -march=nocona -mtune=nocona -std=gnu89

CXX=/usr/bin/g++ CXXFLAGS=-m32 -march=nocona -mtune=nocona -std=gnu++98

LD=/usr/bin/ld LDFLAGS=-m32 -march=nocona -mtune=nocona

PATH=/opt/gnu/bin:/opt/usr/bin:/usr/gnu/bin:/usr/bin:/usr/sbin

PKG_CONFIG_PATH=/opt/usr/lib/pkgconfig

Suggested build sequence:

Fine-tune/fix config.h.in, Makefile.in and others...

$ ./configure \
    --build=i386-pc-solaris2.11 \
    --prefix=/opt/... \
    ...

$ gmake -j3

For IPS package:

$ su 

# source ../setenv-2.7.15 32
# gmake DESTDIR=/stage/prototype/Python/2.7.15/32 install

For immediate use:

$ su

# source ../setenv-2.7.15 32
# gmake install
# zfs snapshot .../opt/...@Python-2.7.15


NOTE

According to my personal preference, I'm assuming the existence of a .../opt/usr ZFS dataset mounted as /opt/usr. And they will be exclusively 32-bits for this post which is about a 32-bits Solaris 11 Express system. Furthermore, multiple bitness would only make sense for 64-bits systems on which 32-bits software can always be run.

$ ./configure \
  --build=i386-pc-solaris2.11 \
  --prefix=/opt/usr \
  --enable-shared \
  --disable-ipv6 \
  --with-signal-module \
  --with-threads \
  --with-pth \
  --without-pymalloc \
  --with-doc-strings \
  --enable-optimizations
  --with-system-ffi \

  --enable-big-digits \
  --with-wctype-functions \
  --with-computed-gotos


I would suggest slight tweaks to the Makefile generated by configure. On line 46, the flag -fprofile-correction isn't supported by GCC 3.4.3, so remove it. Furthermore, the flags -fprofile-generate and -fprofile-use on the same and nearby lines seems great for newer GCC versions. But for an old GCC version such as 3.4.3, this most probably means trouble. Attempting to use these flags will make the build take hours(!) instead of minutes and fail at last with error messages about "overflowing" the .gcda files. Better, getting away with these flags as well as (just in case; paranoia) the LLVM_PROF_MERGER value of true on the nearby line. In that case, it may make sense on line 74 to remove the -g option from the OPT make variable.

$ gmake -j2
...
Run tests sequentially
0:00:00 load avg: 1.12 [  1/399] test_grammar
0:00:00 load avg: 1.12 [  2/399] test_opcodes
0:00:00 load avg: 1.12 [  3/399] test_dict
0:00:00 load avg: 1.11 [  4/399] test_builtin
0:00:00 load avg: 1.11 [  5/399] test_exceptions
0:00:00 load avg: 1.11 [  6/399] test_types
0:00:00 load avg: 1.11 [  7/399] test_unittest
0:00:01 load avg: 1.11 [  8/399] test_doctest
0:00:02 load avg: 1.11 [  9/399] test_doctest2
0:00:02 load avg: 1.11 [ 10/399] test_MimeWriter
0:00:02 load avg: 1.11 [ 11/399] test_SimpleHTTPServer
0:00:02 load avg: 1.11 [ 12/399] test_StringIO
0:00:04 load avg: 1.11 [ 13/399] test___all__
0:00:08 load avg: 1.11 [ 14/399] test___future__
0:00:08 load avg: 1.11 [ 15/399] test__locale

...
Total duration: 11 min 53 sec
Tests result:
FAILURE
(maybe due not running as root or so)
...

Rebuilding with profile guided optimizations:
...
Python build finished, but ... bits to build these modules were not found:
_bsddb     bsddb185     linuxaudiodev     ossaudiodev
To find ..., look in setup.py in detect_modules() for the module's name.

...



NOTE
Do NOT use sudo to perform the gmake install.
It will disastrously loose all environment settings and ruin the build!

Sometimes, as in this paritular case, it may be subtle to detect this problem, because libffi-3.0.9 is part of the default image, but I'm linking to libffi-3.0.13 which I manually build on another library directory as per the crle settings on LD_CONFIG_32 which should point to /opt/gnu/lib and /opt/usr/lib in the first place, where libffi-3.0.13 will be found.

Use a simple su without - , reset the environment variables,
and then finally issue the  gmake install.

$ su
# source ../setenv-2.7.15 32
# gmake install
...

# zfs snapshot -r .../opt/usr@python-2.7.15

And happy Python programming on 32-bits Solaris 11 Express!