 |
| OUT OF DATE |
So far, for some of these experiments I took some extra time to document a few of my findings:
Neither the previous posts are fully satisfying to me, but I could explore and evolve with each of them. Some open-source software is fully compatible with the GNU building system. Others are at most partially compatible. And yet others aren't compatible at all. Few software offer a certain degree of compatibility with native compilers instead of GNU GCC. In general, by using GCC one increases the chances of a successful build. But again and in many cases, Solaris doesn't provide a sufficiently up-to-date GNU GCC compiler, which, in most cases, set a tombstone for the project. A naive thinking could argue that it would suffice to build a very updated (if not state-of-the-art) GNU GCC before attempting any other build. But following this approach one quickly realizes that a lot dependencies must be built beforehand and this can only be done with the outdated tools currently available.
Another difficulty is packaging the resulting software artifacts so that it can be more easily managed throughout other instances of the OS. The ultimate solution is building an IPS package. Although powerful and very desirable on a daily basis, it's not that trivial to build a good IPS package. There are lots of details involved as described on the man page pkg(5). Furthermore, there are other related techniques which are necessary to better integrate, organize and manage the building and distribution of software: a few of them I could grasp on the Oracle Solaris How-To Articles. The general workflow is frequently mixed with some particular case, which sometimes causes some confusion.
Because of all these difficulties my blog entries reveals my hurdles and evolution towards learning the best, I mean, the simplest and most resilient (robust), way (architecture) which takes advantage of the greatest Solaris features for building and running popular open-source software.
At the present time I'd like to think I was able to get to a reasonable start, which was sort of summarized in GNU - Build preparation. But looking forward to IPS packaging in subsequent yet to come blog posts I suspect it has a flaw related to the designated IPS staging-area (/software/prototype/...) and the GNU build-configuration (--prefix=...). Up to now I was setting --prefix=/software/prototype/..., but instead I now realize I should:
- Set --prefix to where I intend to be the final resting place of the software artifacts. For instance, somewhere under /opt. Of course, these shall not be the IPS staging-area.
- Set the DESTDIR environment variable according to the GNU Coding Standards on the same line of the install command. (sudo gmake DESTDIR=/stage/prototype/... install).
- Carefully assess an appropriate RLE configuration. This means setting the LD_CONFIG_32 or LD_CONFIG_64 environment variable or even the default system configuration.
I must review part of the gnu-build-preparation script I presented earlier on GNU - Build preparation, performing the following changes:
- Replacing every occurrence of DS_SOFTWARE with DS_STAGE.
- Replacing every occurennce of /software with /stage.
I must review the general building workflow somewhat as follows:
(assuming 64-bits build and final resting place somewhere under /opt)
- ./configure --prefix=/opt/... ...
- gmake -j$(( $(psrinfo |wc -l) - 1 ))
- sudo gmake DESTDIR=/stage/prototype/.../64 install
I must prepend via an appropriate RLE configuration the following directories (in my particular case) to the default (64-bits in general) library search paths:
- /opt/gnu/lib
- /opt/usr/lib
WARNING
A word of caution! When doing a staged build looking forward to build a system (IPS) package by using DESTDIR pointing to a prototype area in gmake install, it may be necessary to execute some post deployment actions which aren't possible when delivering to the prototype area. For instance, when building GNU libsigsegv you'll get a warning remembering to run "libtool --finish ${prefix}". This would have to be accomplished by some IPS "post-processing" rule.
I also take the opportunity to experiment with getting away with gnu32 and gnu64 in favor of just 32 and 64, adjusting the artifact subtree in prototype and hopefully better naming a few variables and functions. Similarly to the previous version, the revised script assumes the following ZFS delegations are already in place:
$ zfs allow rpool/stage/build
---- Permissions on rpool/stage/build ------------------
Permission sets:
@descendent clone,compression,destroy,promote,quota,
readonly,rename,reservation,share,sharenfs
@generic create,diff,hold,mount,receive,
release,rollback,send,snapshot,userprop
Descendent permissions:
user user1 @descendent
Local+Descendent permissions:
user user1 @generic
$ zfs allow rpool/stage/prototype
---- Permissions on rpool/stage/prototype --------------
Permission sets:
@descendent clone,compression,destroy,promote,quota,
readonly,rename,reservation,share,sharenfs
@generic create,diff,hold,mount,receive,
release,rollback,send,snapshot,userprop
Descendent permissions:
user user1 @descendent
Local+Descendent permissions:
user user1 @generic
For convenience, I list the revised gnu-build-preparation script below:
#!/bin/bash -
################################################################
if [[ -z "$1" ]];
then
echo
echo "Usage: $0
echo
exit 1
else
if ! [[ -f "$1" ]];
then
echo
echo "$1 not found."
echo
exit 1
fi
fi
################################################################
# This can be potentially better suited elsewhere
# as long as it doesn't break previous assumptions.
DS_BASE=rpool
DS_STAGE=${DS_BASE:+$DS_BASE/}stage
DS_BUILD=build
DS_PROTOTYPE=prototype
################################################################
#
# legacy gnu-coreutils IPS package lacks realpath
# TAR=$(realpath "$1")
#
TAR=$(readlink -f "$1")
A=$(basename $1 |\
sed 's/\(\w\+\([-.]\w\+\)*\)-.*/\1/')
V=$(basename $1 |\
sed 's/\w\+\([-.]\w\+\)*-\([0-9]\+\([_.-][0-9]\+\)*[a-z]*\).*/\2/')
echo
echo Processing $TAR
echo
echo ------------------------------------
echo "App: $A"
echo "Ver: $V"
echo ------------------------------------
echo
echo In the process, the following ZFS datasets will be created:
echo
echo " $DS_STAGE/$DS_BUILD/$A"
echo " $DS_STAGE/$DS_BUILD/$A/$A-$V"
echo " $DS_STAGE/$DS_BUILD/$A/$A-$V-32"
echo " $DS_STAGE/$DS_BUILD/$A/$A-$V-64"
echo
echo " $DS_STAGE/$DS_PROTOTYPE/$A"
echo " $DS_STAGE/$DS_PROTOTYPE/$A/$A-$V"
echo " $DS_STAGE/$DS_PROTOTYPE/$A/$A-$V/32"
echo " $DS_STAGE/$DS_PROTOTYPE/$A/$A-$V/64"
echo
read -p "Enter \"y\" to proceed: " -r
if ! [[ "$REPLY" = "y" ]] && ! [[ "$REPLY" = "Y" ]];
then
echo "Exiting..."
exit 1
fi
################################################################
function dataset-exists()
{
zfs list -H "$1" >/dev/null 2>&1
local rv=$?
echo $rv
}
################################################################
function get-mountpoint()
{
echo $(zfs get -H -o value mountpoint "$1" 2>/dev/null)
}
################################################################
function dataset-assert-mounted()
{
if [[ -z $(get-mountpoint "$1") ]]; then
echo "Unexpected: \"$1\" doesn't seem to be mounted."
exit 1
fi
}
dataset-assert-mounted "$DS_STAGE"
dataset-assert-mounted "$DS_STAGE/$DS_BUILD"
dataset-assert-mounted "$DS_STAGE/$DS_PROTOTYPE"
################################################################
function create-source-area()
{
if [[ $(dataset-exists "$1") -eq 1 ]]; then
echo "Creating $1..."
zfs create "$1"
else
local MP=$(get-mountpoint "$1")
if [[ $(basename "$MP") == "$A" ]]; then
echo "No need to create $1."
else
echo
echo "$1 mounted as $MP."
echo "No further action can be safely taken."
echo "Exiting..."
exit 1
fi
fi
}
echo
create-source-area "$DS_STAGE/$DS_BUILD/$A"
create-source-area "$DS_STAGE/$DS_BUILD/$A/$A-$V"
################################################################
echo
DS_SOURCE="$DS_STAGE/$DS_BUILD/$A"
MP_SOURCE=$(get-mountpoint "$DS_SOURCE")
DS_SOURCE_VERSION="$DS_SOURCE/$A-$V"
MP_SOURCE_VERSION=$(get-mountpoint "$DS_SOURCE_VERSION")
if [[ $(ls -A "$MP_SOURCE_VERSION" 2>/dev/null) ]]; then
echo
echo "Directory $MP_SOURCE_VERSION not empty!"
echo "No further action can be safely taken."
echo "Exiting..."
exit 1
fi
(cd "$MP_SOURCE_VERSION/.."; gtar xf "$TAR")
if ! [[ $(ls -A "$MP_SOURCE_VERSION" 2>/dev/null) ]]; then
echo
echo "Directory $MP_SOURCE_VERSION shouldn't be empty!"
echo "No further action can be safely taken."
echo "Exiting..."
exit 1
fi
zfs snapshot "$DS_SOURCE_VERSION@source"
zfs set readonly=on "$DS_SOURCE_VERSION"
################################################################
function set-target()
{
local T=$1
# This may be inappropriate in some cases.
# Some "packages" require/prefer just an empty dataset.
zfs clone "$DS_SOURCE_VERSION"@source \
"$DS_SOURCE_VERSION-$T"
# This may be too soon.
# The "package" may need some manual fix before @start
zfs snapshot "$DS_SOURCE_VERSION-$T"@start
}
set-target 32
set-target 64
################################################################
zfs list -H -r -t all -o name "$DS_SOURCE"
################################################################
#
# Create the final resting place of the results.
# Use an independent ZFS pool/dataset, not rpool/VARSHARE.
# At end consider snapshoting, just in case.
# Could be a staging area for packaging.
#
DS_ARTIFACT="$DS_STAGE/$DS_PROTOTYPE/$A"
DS_ARTIFACT_VERSION="$DS_ARTIFACT/$V"
echo
echo "Creating "$DS_ARTIFACT" subtree."
echo
function create-staging-area()
{
if [[ $(dataset-exists "$1") -eq 1 ]]; then
zfs create -p "$1"
fi
}
create-staging-area "$DS_ARTIFACT_VERSION"
create-staging-area "$DS_ARTIFACT_VERSION/32"
create-staging-area "$DS_ARTIFACT_VERSION/64"
zfs list -H -r -t all -o name "$DS_ARTIFACT"
MP_PROTOTYPE=$(get-mountpoint "$DS_STAGE/$DS_PROTOTYPE")
################################################################
echo
echo "Creating pre-configuration script."
cat > "$MP_SOURCE/setenv-$V" <<EOF
#
# HOSTTYPE=$HOSTTYPE
# OSTYPE=$OSTYPE
# MACHTYPE=$MACHTYPE
#
# Up to Solaris 11.3 the default is 32-bits.
# MACHTYPE is typically i386-pc-solaris2.11
#
# Since Solaris 11.4 Beta the default is 64-bits.
# MACHTYPE is x86_64-pc-solaris2.11
#
# It's advisable not to override the above env vars
# buy you experiment with the --build or --target options
# to the standard GNU's configuration script: configure.
# (they are important to properly organize the built artifacts)
#
# Bottom line it seems that the -m option to the flags
# CFLAGS, CXXFLAGS and LDFLAGS always prevail after all.
# (in terms of the actual bitness of the built artifacts)
#
if [[ -z "\$1" ]];
then
echo "Usage: source setenv
return 1
else
if [[ "\$1" -ne 32 ]] && [[ "\$1" -ne 64 ]];
then
echo "Valid bit-targets: 32 or 64."
return 1
fi
fi
# Despite setting bitness as shown next
# DON'T override HOSTTYPE and MACHTYPE
# Just remind the proper --build option
BITS=\$1
echo
function add-path()
{
local COMPONENT="\$1"
if [[ -d "\$COMPONENT" ]] ;
then
! [[ "\$PATH" =~ "\$COMPONENT" ]] && \
export PATH="\$COMPONENT:\$PATH"
fi
}
function add-pkgconfig-path()
{
local COMPONENT="\$1"
local TAIL="\${PKG_CONFIG_PATH:+:\$PKG_CONFIG_PATH}"
if [[ -d "\$COMPONENT" ]] ;
then
! [[ "\$TAIL" =~ "\$COMPONENT" ]] && \
export PKG_CONFIG_PATH="\$COMPONENT\$TAIL"
fi
}
function extend-env()
{
# In this revision I'm only focusing on 64-bits.
# Hence, I'm omitting the bitness subfolder.
# local BASE="\$1/\$BITS"
local BASE="\$1"
add-path "\$BASE/bin"
add-pkgconfig-path "\$BASE/lib/pkgconfig"
}
add-path /usr/gnu/bin
#
# Other PATH and PKG_CONFIG_PATH settings.
# Put entries in reverse order of dependency.
# (following my PATH building suggestion)
# The samples below are pre-Solaris 11.4 Beta.
#
# ...
# extend-env /opt/app
# extend-env /opt/usr
# extend-env /opt/gnu
#
# RLE configuration
#
LIBBITS=\$([[ \$BITS -eq 32 ]] && echo "" || echo "/64")
RLEBITS=\$([[ \$BITS -eq 32 ]] && echo "" || echo "-64")
RLECFG="$MP_SOURCE/ld.config-\$BITS"
#
# /opt/gnu/lib and /opt/usr/lib are fixed for simplification here
# but could mimic the system /usr/gnu/lib and /usr/lib
#
crle \$RLEBITS \\
-c "\$RLECFG" \\
-l /opt/gnu/lib:/opt/usr/lib \\
-l /lib\$LIBBITS:/usr/lib\$LIBBITS
crle \$RLEBITS -c "\$RLECFG"
export LD_CONFIG_\$BITS="\$RLECFG"
echo LD_CONFIG_\$BITS="\$RLECFG"
echo
# In general, not a good idea; evaluate.
# export CONFIG_SHELL=/bin/bash
echo CONFIG_SHELL=\$CONFIG_SHELL
echo
#
# The following compilers and linker overrides
# may be troublesome, specially the -m and -std options
# as it may be very source dependent; evaluate.
# The -std bar minimums on Solaris 11 Express is gnu89 and gnu++98.
# The -std bar minimums on Solaris 11.3 is gnu89 and gnu++03.
# On Solaris 11.4 Beta gnu11 and gnu++11 seem Ok.
#
FLAGS="-m\$BITS -march=core2 -mtune=core2"
export CC=/usr/bin/gcc
export CFLAGS="\$FLAGS -std=gnu89"
echo CC=\$CC CFLAGS=\$CFLAGS
echo
export CXX=/usr/bin/g++
export CXXFLAGS="\$FLAGS -std=gnu++03"
echo CXX=\$CXX CXXFLAGS=\$CXXFLAGS
echo
export LD=/usr/bin/ld
export LDFLAGS="\$FLAGS"
echo LD=\$LD LDFLAGS=\$LDFLAGS
echo
echo PATH=\$PATH
echo
echo PKG_CONFIG_PATH=\$PKG_CONFIG_PATH
unset FLAGS
unset LIBBITS
unset RLEBITS
unset RLECFG
function get-target()
{
case \$BITS in
32)
echo i386-pc-solaris2.11
;;
64)
echo x86_64-pc-solaris2.11
;;
esac
}
echo
echo Suggested build sequence:
echo
echo Fine-tune/fix config.h.in, Makefile.in and others...
echo
echo \$ ./configure \\\\
echo " "--build=\$(get-target \$BITS) \\\\
echo " "--prefix=/opt/... \\\\
echo " "...
echo
echo \$ gmake -j\$(( \$(psrinfo |wc -l) - 1 ))
echo
echo \$ su
echo \# source ../setenv-$V \$BITS
echo
echo For IPS package:
echo
echo \# gmake DESTDIR=$MP_PROTOTYPE/$A/$V/\$BITS install
echo
echo For immediate use:
echo
echo \# gmake install echo \# zfs snapshot -r .../opt/...@$A-$V
EOF
cat > "$MP_SOURCE/unsetenv-$V" <<EOF
function cleanup-env()
{
[[ -f "\$LD_CONFIG_32" ]] && rm "\$LD_CONFIG_32"
unset LD_CONFIG_32
[[ -f "\$LD_CONFIG_64" ]] && rm "\$LD_CONFIG_64"
unset LD_CONFIG_64
unset CONFIG_SHELL
unset CC
unset CFLAGS
unset CXX
unset CXXFLAGS
unset LD
unset LDFLAGS
unset PKG_CONFIG_PATH
export PATH=/usr/bin:/usr/sbin
unset -f add-path
unset -f add-pkgconfig-path
unset -f extend-env
unset -f cleanup-env
}
cleanup-env
EOF
touch "$MP_SOURCE/NOTES"
I would also suggest creating 3 additional datasets with somewhat similar structures at the same level as build and prototype. They would be:
- source (gather all tarballs for every desired version)
- package (gather final results or pre-built binaries tarballs)
- test (scratch area with quite obvious purposes)
The whole staging area first children would be:
$ zfs list -r -d 1 -o name rpool/stage
NAME
rpool/stage
rpool/stage/build
rpool/stage/package
rpool/stage/prototype
rpool/stage/source
rpool/stage/test
$ l /stage
total 31
drwxrwxrwt 4 root root ... build
drwxrwxrwt 10 root root ... package
drwxrwxrwt 4 root root ... prototype
drwxrwxrwt 6 root root ... source
drwxrwxrwt 3 root root ... test
