Tuesday, November 6, 2012

IPS repository creation

This post is about the first steps on establishing a a local IPS repository.
But once this is done establishing access to it is covered on a separate post.
In addition, after these previous tasks are complete, it may be necessary to update it.

For instance, the best way to install Oracle Solaris Studio is to set up a local IPS repository.
The procedure is reasonably well and fairly documented, but let's review it anyway.

What may not be very clear is that by default a local repository would be assumed to rest in /var/pkg/repo, even though it doesn't exist by default. This means that a dedicated ZFS dataset could or should have its mountpoint property set to this default, although it could be better to use symbolic links yielding something similar to /var/pkg/solaris/11.x/release and /var/pkg/solaris/11.x/sru-# and so on... But below, as dealing with Studio packages instead of OS packages, I've followed the crowd and created something completely apart. Anyway, note that in addition to compression (compression=on), access time could be turned off (atime=off) to buy some performance:
  
# zfs create rpool/export/depot
# zfs create -o compression=on rpool/export/depot/studio
 
# zfs get -r compression rpool/export/depot
NAME                       PROPERTY     VALUE     SOURCE
rpool/export/depot         compression  off       default
rpool/export/depot/studio  compression  on        local
   
# pkgrepo create /export/depot/studio
# ll /export/depot/studio

...
-rw-r--r--   1 root root   78 Nov  6 10:45 pkg5.repository

I'll use the support version of the repository which requires an Oracle support plan.
With no support plan the release version can be used according to terms and conditions.
For both versions it's necessary to get certificate(s) at https://pkg-register.oracle.com:
  
# cd /var/pkg/ssl
# ll Studio_Support.* 
...
-rw-------  1 root  root  753 ... Studio_Support.cert.pem
-rw-------  1 root  root  887 ... Studio_Support.key.pem

I'll use an script to populate and later update the local repository. If behind a proxy, set http_proxy and https_proxy accordingly. But sometimes a proxy/firewall can block the several TCP connections from the remote port 443 to random ports and the symptoms are stalls or ugly messages. In case of difficulties check your command line, your certificates and then the proxy/firewall settings. Perhaps you may need to temporary alter your wired connection and temporarily bypass something.

# cat update-studio
#!/bin/sh -

SSL=/var/pkg/ssl
PRODUCT=Studio_Support
 
KEY=$SSL/$PRODUCT.key.pem
CERT=$SSL/$PRODUCT.cert.pem
 
SOURCE=https://pkg.oracle.com/solarisstudio/support/
TARGET=/export/depot/studio
 
pkgrecv -s $SOURCE -d $TARGET --key $KEY --cert $CERT '*' 
[ $? -eq 0 ] && pkgrepo refresh -s $TARGET



But IPS repositories are not only useful for Solaris Studio.
In fact, it's crucial for Solaris 11 itself.

Start with just the initial release version of the repository.
Later, it can turn into a multiple support release updates repository.
The big picture is rather simple:
  1. Create a dedicated file system;
  2. Obtain just the release packages.
It follows a real example of the overall process.
For obtaining the release packages, two methods are available.

As always, begin by creating a dedicated ZFS file system.
Ideally, set compression and atime properties as already described.

# zfs create rpool/export/depot/solaris

The release packages can be found at http://pkg.oracle.com/solaris/release.
Alternatively, download and concatenate a split ISO file.
I'm not sure which is the most efficient.

I do know that burning the ISO file, instead of lofs-mounting it is bad idea.
It'll take much longer to receive all the packages.
Underlying seek in optical media isn't efficient.
The manifests processing is specially awful.

Getting the release packages.
Choose one of the following options.

OPTION 1:

# pkgrepo create /export/depot/solaris

# URI=http://pkg.oracle.com/solaris/release
# pkgrecv -s $URI -d /export/depot/solaris '*' 
Processing packages for publisher solaris ...
Retrieving and evaluating 4401 package(s)...
Download Manifests ( 369/4401) -
... 
  
# pkgrepo refresh -s /export/depot/solaris

OPTION 2:

# lofiadm -a /export/archive/sol-11_1-repo-full.iso
/dev/lofi/1
 
# mount -F hsfs /dev/lofi/1 /mnt
# ll /mnt
total 29
drwxr-xr-x   3 root  root    2.0K ... repo
-rwxr-xr-x   1 root  root    1.3K ... NOTICES
-rw-r--r--   1 root  root    3.2K ... COPYRIGHT
-rw-r--r--   1 root  root    7.4K ... README
  
(on an Intel Core 2 Quad Q6600 2.40 GHz with slow single disk)
# time rsync -q -aP /mnt/repo/ /export/depot/solaris

real    8m23.000s
user    1m20.248s
sys     3m43.634s

NOTE
Under VirtualBox that could be ridiculously more time consuming:
real    173m51.306s
user    1m19.884s
sys     3m14.512s
# umount /mnt
# lofiadm -d /dev/lofi/1

# zfs get referenced,compressratio rpool/export/depot/solaris
NAME                        PROPERTY       VALUE  SOURCE
rpool/export/depot/solaris  referenced     6.07G  -
rpool/export/depot/solaris  compressratio  1.15x  -
 

# pkgrepo rebuild -s /export/depot/solaris
Initiating repository rebuild.

Up to this point the task is finished.
But for good practice and also for later incremental updates, snapshot it:

# zfs snapshot rpool/export/depot/solaris@release

NOTE
On this post I've been using rpool/export/depot.
In fact what I prefer is to create dedicated pools.
That is, one for export and other for depot.

$ zpool list
NAME     SIZE  ALLOC   FREE  CAP  DEDUP  HEALTH  ALTROOT
depot    149G  38.1G   111G  25%  1.00x  ONLINE  -
export   464G   339G   125G  73%  1.00x  ONLINE  -
rpool   55.5G  14.4G  41.1G  25%  1.00x  ONLINE  -


Let me show a basic scheme that's been useful for years:

# zfs list -o name,mountpoint -t all -r depot | grep -v bk
NAME                              MOUNTPOINT
depot                             /depot
depot/software                    /depot/software
depot/solaris                     /depot/solaris
depot/solaris/11.2                /depot/solaris/11.2
depot/solaris/11.2@release        -
depot/solaris/11.3                /depot/solaris/11.3
depot/solaris/11.3@release        -
depot/studio                      /depot/studio
depot/studio/12                   /depot/studio/12
depot/studio/12@release           -


# ll /depot/solaris/ /depot/studio/
/depot/solaris/:
total 9
drwxr-xr-x   3 root root ... 11.2
drwxr-xr-x   3 root root ... 11.3
lrwxrwxrwx   1 root root ... current -> 11.3

/depot/studio/:
total 5
drwxr-xr-x   3 root root ... 12
lrwxrwxrwx   1 root root ... current -> 12