GNU/Solaris

Since a few months or perhaps almost a couple of years, I've been trying to learn how to manually update a few software from an ordinary Solaris 11.3 GA. That's because even if I had a contract support I wouldn't have all the latest software versions I would like to use on my Solaris system.

Of course, that's not the Solaris main goals by nowadays as currently advertised. Nevertheless, against the vision one should get another system as a front-end to Solaris, I still prefer to also use Solaris as my front-end system because I also very much appreciate all the potential that Solaris has to offer in this area as well. But for that purpose I have to manually update several software to more current versions in order to keep pace with the modern world and natural evolution. It's true that Solaris 11.4 is promising a somewhat better catch-up with this, but still it doesn't seem to be among its main goals, of course! Solaris is huge and specially fitted to big-iron SPARC systems and I totally agree with this while I'm greatly delighted I still could enjoy it on my low-end toy x86_64 system.

SAFE HARBOUR / DISCLAMER

Please, note that I mean no Licenses, Rights or Copyright infringement at all in what I post here. It's just that I appreciate so much Solaris that I'd dream of (as if it were possible) having the best of many worlds merged on a single system. For instance, I would dream of something called GNU/Solaris somewhat as presented on Linux & GNU and somewhat in the sense of GNU's What's in a Name.

The Java Archive

One may get to the official The Java Archive (as I call it) download page which link is found to the bottom of the official Java SE Downloads page. As dully warned by Oracle the legacy artifacts are unsupported and security flawed:
 

"The Java Archive offers access to some of our historical Java releases. WARNING: These older versions of the JRE and JDK are provided to help developers debug issues in older systems. They are not updated with the latest security patches and are not recommended for use in production."

Building Node.js 8.10.0

This post is more of a continuous effort in order to learn how to build by myself some more up-to-date artifacts, this time Node.js 8.10.0 (LTS), to my preferred software platform: Solaris 11. In particular, I'm still considering Solaris 11.3 GA, which by the time of this writing is about 3 years old with GCC 4.8.2. Solaris 11.4 GA is expected to arrive late this year (2018).

To partially quote Node.js:
 

Node.js® is a JavaScript runtime built on Chrome's V8 JavaScript engine. Node.js uses an event-driven, non-blocking I/O model that makes it lightweight and efficient. ... .

Node.js is a typical case for a Solaris back-end system, where its cloud-ready infrastructure would be typically very well-suited to the task: SMF, ZFS, zones, higher threading count and advanced networking capabilities.

The version used in this post, Node.js 8.10.0 (LTS), is a March 2018 security update. I've chosen the LTS 8.x series because it's the latest branch that better aligns to the Solaris way-of-things. It's expected to live until Dec 2019 and not beyond because it's aligning to OpenSSL 1.0.2 life-cycle to which it depends.

This post's (at the time of this writing) up-to-date custom build is relevant not only because of the security side, but because the software isn't available in the official repositories and, even if it could be one day, it would be uncertain if the update pace of the support repository would be acceptable. Furthermore, by performing a custom build instead of just downloading pre-built binaries one ensures getting more exactly what's needed and optimized for a particular machine as per specific compiler (CPU) options and source-code modules selection.

Right from the start I was able to build a 64-bits Node.JS 8.10.0 (LTS) in Solaris 11.4 Beta (already available) with GCC 5.5.0, but at first I wasn't able to repeat the task under Solaris 11.3 GA and I was somewhat settled with it as the "build instructions" stated a more updated (4.9.4 - Aug 3, 2016) GCC was required and I only had an almost 3-years old 4.8.2 (from Oct 16, 2013) in Solaris 11.3. Although a newer GCC is generally better, by inspecting the initial build failures in Solaris 11.3 GA I've noticed that there weren't that many changes that could render the task impossible. This motivated me to take some time to further investigate a way of trying to accomplish the task.

Building Ruby 2.5.0

This post is more of a continuous effort in order to learn how to build by myself some more up-to-date artifacts, this time Ruby version 2.5.0, to my preferred software platform: Solaris 11.

In particular, I'm still considering Solaris 11.3 GA, which by the time of this writing is about 3 years old with GCC 4.8.2. Solaris 11.4 GA is expected to arrive late this year (2018), but curiously I wasn't yet able to succeed in building Ruby 2.5.0 in Solaris 11.4 Beta (already available) with GCC 5.5.0.

This very latest stable Ruby version 2.5.0 isn't yet available on the package repositories of any Solaris version and that's why I'm doing all this. For instance, on Solaris 11.3 GA:

$ pkg search -H -o pkg.shortfmri *:set:pkg.fmri:runtime/ruby* \
  |sort -u
pkg:/runtime/ruby-18@1.8.7.374-0.175.3.0.0.24.0
pkg:/runtime/ruby-19@1.9.3.551-0.175.3.0.0.30.0
pkg:/runtime/ruby-19/ruby-tk@1.9.3.551-0.175.3.0.0.30.0
pkg:/runtime/ruby-21@2.1.6-0.175.3.0.0.30.0
pkg:/runtime/ruby-21/ruby-tk@2.1.6-0.175.3.0.0.30.0
pkg:/runtime/ruby@1.9-0.175.3.0.0.30.0

In my previous learning path towards building myself a few open-source software to Solaris 11.3 I've ended up with a facility script presented on another post, GNU build preparation, which was the result from a few early experiments (which I shall review at some point in the near future, I hope):

• Building Qt 4.8.7
• Building SQLite 
• Building Tcl / Tk 
• Building Harbour 

I assume the developer-gnu IPS package has been installed and that the following ZFS delegations are already in place:

$ zfs allow rpool/software/build
---- Permissions on rpool/software/build ------------------
Permission sets:
    @descendent clone,compression,destroy,promote,quota,
                readonly,rename,reservation,share,sharenfs
    @generic create,diff,hold,mount,receive,
             release,rollback,send,snapshot,userprop
Descendent permissions:
    user user1 @descendent
Local+Descendent permissions:
    user user1 @generic

$ zfs allow rpool/software/prototype
---- Permissions on rpool/software/prototype --------------
Permission sets:
    @descendent clone,compression,destroy,promote,quota,
                readonly,rename,reservation,share,sharenfs
    @generic create,diff,hold,mount,receive,
             release,rollback,send,snapshot,userprop
Descendent permissions:
    user user1 @descendent
Local+Descendent permissions:
    user user1 @generic

After downloading and verifying the checksum of a compressed source-code tarball and using the gnu-build-prepration script from the GNU build preparation post I end up with the following build structure mounted at /software (the details of running the GNU build preparation script are presented in the last example of that that post):
  
rpool/software/build/ruby
rpool/software/build/ruby/ruby-2.5.0
rpool/software/build/ruby/ruby-2.5.0@source 
rpool/software/build/ruby/ruby-2.5.0-gnu32 
rpool/software/build/ruby/ruby-2.5.0-gnu32@start
rpool/software/build/ruby/ruby-2.5.0-gnu64 
rpool/software/build/ruby/ruby-2.5.0-gnu64@start
 
rpool/software/prototype/ruby 
rpool/software/prototype/ruby/ruby-2.5.0
rpool/software/prototype/ruby/ruby-2.5.0/gnu32
rpool/software/prototype/ruby/ruby-2.5.0/gnu64

NOTE ❊

Since the original writing of this post I have revised some important assumptions and recommendations which would lead to an extensive rewriting. Instead of doing it all over again, I'm kindly asking that you pay attention to the following changes:

1. I've adjusted all the ZFS tree datasets' names
   (the development tree is now rooted at /stage)
    
2. I've started using DESTDIR and a different --prefix
   (DESTDIR=/stage/prototype/ruby/2.5.0/64)
   (--prefix=/opt/sfw/ruby/2.5.0)
    
3. I'm taking a final ZFS snapshot as the deployment closing.
   (zfs snapshot -r .../opt/sfw/ruby/2.5.0@release)

Hence, in what follows just keep an eye on adjusting accordingly.

The next step is to edit the configuration script (setenv) that is delivered in the previous step in order to adjust / tune up some environment variables which are influential to the build process. Then, before starting the build, extend the environment accordingly. For instance, to prepare a 64-bits build on my particular machine I do as follows:

$ cd ruby/ruby-2.5.0-64

$ source ../setenv 64

--prefix=/software/prototype/.../gnu64
--build=x86_64-pc-solaris2.11

CONFIG_SHELL=

CC=/usr/bin/gcc CFLAGS=-m64 -march=core2 -std=gnu99

CXX=/usr/bin/g++ CXXFLAGS=-m64 -march=core2 -std=gnu++11

LD=/usr/bin/ld LDFLAGS=-m64 -march=core2

PATH=/usr/gnu/bin:/usr/bin:/usr/sbin

PKG_CONFIG_PATH=

Solaris 11 Express awesome update

Solaris 11 Express was a transitional version from Solaris 10 to Solaris 11, post OpenSolaris 2009.06. Since it became publicly available everybody could see by that time how promising Solaris 11 would be, in many ways, part because of OpenSolaris community. Nowadays, one can better recognize what has been achieved up to Solaris 11.3, which by the time of this writing, has been around about 3 years. There's also an early announcement that Solaris 11.4 will be available by mid-2018, and according to planned EoLN (end-of-life notices) chances are that many desktop features will be further trimmed out, although GCC should be finally upgraded, hopefully to version greater than 7, let's see.

NOTE

Although by now Solaris 11 Express is officially obsolete and OpenSolaris has been left behind by Oracle, there are still community efforts to reestablish open-source variants, such as OpenIndiana and SmartOS, both running over a lagging-behind kernel called Illumos based on the original open-source version of the SunOS 5.11. But unfortunately, that kernel isn't as nearly modernized and optimized as Oracle's current closed-source product.

But in spite of all that, Solaris 11 Express major benefit was to early incorporate many Solaris advancements over Solaris 10 and still run under a legacy 32-bits platform! Yes! This is key, because despite the official business strategies and propaganda focusing on 64-bits mid and high-end big-iron, the truth is that there's a lot of legacy hardware can still be put to good service to the crowds on the 3rd world which are thriving to evolve and that do not count on a lot of money and other powerful and current resources.

The initial GA release of Solaris 11 Express didn't perform well, perhaps due to a lot of debugging hooks (code assertions) and conservative strategies, after all it was a key transitional milestone of Solaris. But fact is that those who had payed for a support contract could benefit from regular updates, called SRU (service release updates), which fixed many issues and greatly improved the system performance, including booting speed. By the last general SRU, SRU-13, things were noticeably better.

For instance, Solaris 11 Express SRU-13 rivals the speed of Solaris 11.3 GA and certainly runs faster than OpenIndiana 2017.04. In my opinion, a relative comparison among "recent" Solaris distros could be depicted by the following table:

But wait! Things could become even better because Engineered Systems for high-performance grid-computing started to see the light of the day and some of them were to be driven by Solaris 11 Express! This transitional version of Solaris then became so acclaimed and accredited that it deserved a additional and special SRU update targeted to Exalogic, the SRU-14. The SRU-14 could not be applied to ordinary systems because it had a special dependency associated to Exalogic Engineered System. Of course, there should be good reasons for such constrain. But fact is or at least seems to be that in general it runs amazingly well on ordinary systems too!
To enjoy all the power of SRU-14 on ordinary systems, some homework is necessary in order lift the impeding constrain embedded on the update.

NOTE

I'll assume that a support repository has already been made available by means described on procedures I've visited on the past, such as, the IPS repository update post. 

For instance, consider the following local support repository:

# zfs list -o mountpoint -H -r /depot
/depot
/depot/solaris
/depot/solaris/11e
/depot/solaris/11e/release
/depot/solaris/11e/sru-13
/depot/solaris/11e/sru-14

At first, an usual update attempt from SRU-13 to SRU-14 fail:

# pkg update --be-name solaris-11e-sru-14
Creating Plan ...
pkg update: No solution was found to satisfy constraints
Plan Creation: Package solver has not found a solution 
               to update to latest available versions.
               This may indicate an overly constrained 
               set of packages are installed.

latest incorporations:

  pkg://solaris/consolidation/gnome/gnome-incorporation@...151.0.1.14...
  pkg://solaris/consolidation/sfw/sfw-incorporation@...151.0.1.14...
  pkg://solaris/consolidation/osnet/osnet-incorporation@...151.0.1.14...
  pkg://solaris/entire@...151.0.1.14...

The following indicates why the system cannot update to the latest version:

    Reject:  pkg://solaris/entire@...151.0.1.14...
    Reason:  A version for 'require-any' dependency on
             pkg:/system/platform/exalogic/firstrun cannot be found

From the diagnostic messages above it's possible to realize that the SRU-14 was crafted to be applied as part of an automated installation of Solaris 11 Express target to the Exalogic Engineered System. The only constrain was a missing IPS package delivering a one-time-run SMF service performing initial configurations to Exalogic:

pkg:/system/platform/exalogic/firstrun.

NOTE

It's noticeable that in more recent releases, the description of the technique for tailoring a one-time-run IPS packages has evolved while it was completely lacking on Solaris 11 Express. Nevertheless, the simpler and straightforward instructions found on Solaris 11/11 Information Library is enough to perfectly work under Solaris 11 Express. Despite the evolution in documentation it still lacks a lot of clarity by sticking to do it that way instead of building knowledge, unfortunately. For this post I'll stay as much as possible with the clearer and simpler procedures which seem to still work equally well in terms of backward compatibility.

The major steps in creating the missing package are:

1. Creating a SMF service manifest for a dummy service.
2. Deploying the special IPS package unlocking SRU-14.

The above steps does nothing more than completing the set of requirements that unlock the SRU-14 installation. I'm not sure if I could create an "empty" package, that is, maybe the dummy SMF service is unneeded after all. Anyway, the difficulties rely just on the intrinsics of these steps themselves, not on the big picture. Let's visit each of the above steps in more detail:

1. Creating a SMF service manifest for a dummy service.

In recent versions of Solaris, this has been somewhat simplified by svcbundle(1M), but I won't rely on it at this moment. I prefer to know all the details and stay in control as much as possible.

# mkdir /tmp/sru-14-unlock
# cd !!$

# cat sru-14-unlock.xml

...
<?xml version="1.0" ?>
<!DOCTYPE service_bundle SYSTEM "/usr/share/lib/xml/dtd/service_bundle.dtd.1">

<service_bundle name="sru-14-unlock" type="manifest">

  <service name="sru-14-unlock" type="service" version="1">

    <create_default_instance enabled="false"/>
    <single_instance/>

    <dependency name="multi_user" type="service" grouping="require_all" restart_on="none">
      <service_fmri value="svc:/milestone/multi-user:default"/>
    </dependency>

    <exec_method name="start"   type="method" exec=":true" timeout_seconds="60"/>
    <exec_method name="stop"    type="method" exec=":true" timeout_seconds="60"/>
    <exec_method name="refresh" type="method" exec=":true" timeout_seconds="60"/>

    <!-- must be defined at this exact place -->
    <property_group name="startd" type="framework">
      <propval name="duration" type="astring" value="transient"/>
    </property_group>

  </service>

</service_bundle>

# svccfg validate !!$
...

2. Deploying the special IPS package unlocking SRU-14.

This amounts to the creation and installation of the missing package.
What matters most is the package name, highlighted below.

# pwd
/tmp/sru-14-unlock

# mkdir -p ./prototype/lib/svc/manifest/site
# cp sru-14-unlock/sru-14-unlock.xml !!$
...

# cat sru-14-unlock.p5m
set \
  name=pkg.fmri \
  value=system/platform/exalogic/firstrun@1.0,5.11

set \
  name=pkg.summary \
  value="SRU-14 unlock."

set \
  name=pkg.description \
  value="Dummy package to unlock SRU-14 installation."
 
set \
  name=org.opensolaris.smf.fmri \
  value=svc:/sru-14-unlock

set \
  name=org.opensolaris.consolidation \
  value=userland

set \
  name=info.classification \
  value="org.opensolaris.category.2008:System/Packaging"
 
file \
  path=lib/svc/manifest/site/sru-14-unlock.xml \
  mode=0444 owner=root group=sys

NOTE

The extension .p5m most probably means package v5 manifest.

# pkglint !!$
Lint engine setup...
Starting lint run...
 
This package must be placed into a package repository, from which it can be subsequently installed. If the installation is to be part of an automated install (AI), then the repository must created on a location accessible to AI clients during the first boot. On this post I'm not using AI, so I'll just create the repository under /tmp which suffices to a one-time interactive install.

# pwd
/tmp/sru-14-unlock

# pkgrepo create ./repo
# pkgrepo add-publisher -s ./repo solaris

# pkgsend publish -d ./prototype -s ./repo sru-14-unlock.p5m
pkg://solaris/system/platform/exalogic/firstrun@1.0,5.11:...Z
PUBLISHED

# pkg list -af -g ./repo
NAME (PUBLISHER)                     VERSION    IFO
system/platform/exalogic/firstrun    1.0        ---

# pkg info -g ./repo firstrun
       Name: system/platform/exalogic/firstrun
    Summary: SRU-14 unlock.
Description: This dummy package...
      State: Not installed
  Publisher: solaris
    Version: 1.0
... Release: 5.11
     Branch: None
   ... Date: ...
       Size: 928.00 B
       FMRI: pkg://solaris/system/platform/exalogic/firstrun...

# pkg install -g ./repo -nv firstrun
           Packages to install:        1
     Estimated space available:   ... GB
Estimated space to be consumed: 14.31 MB
       Create boot environment:       No
Create backup boot environment:       No
          Rebuild boot archive:       No

Changed packages:
solaris
  system/platform/exalogic/firstrun
    None -> 1.0,5.11:...

# pkg install -g ./repo -v firstrun 
...
DOWNLOAD               PKGS       FILES    XFER (MB)
Completed               1/1         1/1      0.0/0.0

PHASE                                        ACTIONS
Install Phase                                    7/7

PHASE                                          ITEMS
Package State Update Phase                       1/1
Image State Update Phase                         2/2

PHASE                                          ITEMS
Reading Existing Index                           8/8
Indexing Packages                                1/1

# pkg info firstrun
       Name: system/platform/exalogic/firstrun
    Summary: SRU-14 unlock.
Description: Dummy package to unlock SRU-14 installation.
   Category: System/Packaging
      State: Installed
  Publisher: solaris
    Version: 1.0
...

# svcadm restart manifest-import

On the console one sees:
Loading smf(5) servicd descriptions: 1/1

# svcs -a |grep sru-14
disabled       19:36:52 svc:/sru-14-unlock:default

And voilà!

For a default text-installation of Solaris 11 Express with SRU-13 one gets:

# pkg update -nv --be-name solaris-11e-sru-14
            Packages to update:        15 
     Estimated space available:    ... GB
Estimated space to be consumed: 366.68 MB
       Create boot environment:       Yes
     Activate boot environment:       Yes
Create backup boot environment:        No 
          Rebuild boot archive:       Yes

Changed packages:

solaris
  SUNWcs
    0.5.11,5.11-0.151.0.1.13:... -> 0.5.11,5.11-0.151.0.1.14:...
  consolidation/gnome/gnome-incorporation
    0.5.11,5.11-0.151.0.1.13:... -> 0.5.11,5.11-0.151.0.1.14:...
  consolidation/osnet/osnet-incorporation
    0.5.11,5.11-0.151.0.1.13:... -> 0.5.11,5.11-0.151.0.1.14:...
  consolidation/sfw/sfw-incorporation
    0.5.11,5.11-0.151.0.1.13:... -> 0.5.11,5.11-0.151.0.1.14:...
  database/sqlite-3
    3.6.23,5.11-0.151.0.1.4:...  ->  3.7.5,5.11-0.151.0.1.14:...
  entire
    0.5.11,5.11-0.151.0.1.13:... -> 0.5.11,5.11-0.151.0.1.14:...
  image/library/libpng
    0.5.11,5.11-0.151.0.1:...    -> 0.5.11,5.11-0.151.0.1.14:...
  library/desktop/gtk2
    0.5.11,5.11-0.151.0.1:...    -> 0.5.11,5.11-0.151.0.1.14:...
  library/libtasn1
    0.5.11,5.11-0.151.0.1:...    -> 0.5.11,5.11-0.151.0.1.14:...
  runtime/python-26
    2.6.4,5.11-0.151.0.1:...     ->  2.6.4,5.11-0.151.0.1.14:...
  system/file-system/zfs
    0.5.11,5.11-0.151.0.1.11:... -> 0.5.11,5.11-0.151.0.1.14:...
  system/kernel
    0.5.11,5.11-0.151.0.1.13:... -> 0.5.11,5.11-0.151.0.1.14:...
  system/kernel/platform
    0.5.11,5.11-0.151.0.1.12:... -> 0.5.11,5.11-0.151.0.1.14:...
  system/library
    0.5.11,5.11-0.151.0.1.13:... -> 0.5.11,5.11-0.151.0.1.14:...
  system/network/nis
    0.5.11,5.11-0.151.0.1.8:...  -> 0.5.11,5.11-0.151.0.1.14:...

# pkg update --be-name solaris-11e-sru-14
            Packages to update:  15
       Create boot environment: Yes
Create backup boot environment:  No

DOWNLOAD               PKGS       FILES    XFER (MB)
Completed             15/15   1743/1743    41.8/41.8

PHASE                                        ACTIONS
Removal Phase                                  58/58
Install Phase                                  52/52
Update Phase                               4258/4258

PHASE                                          ITEMS
Package State Update Phase                     30/30
Package Cache Update Phase                     15/15
Image State Update Phase                         2/2

PHASE                                          ITEMS
Reading Existing Index                           8/8
Indexing Packages                              15/15

A clone of ... exists and has been updated and activated.
On the next boot the Boot Environment solaris-11e-sru-14 will be
mounted on '/'.  Reboot when ready to switch to this updated BE.

----------------------------------------------------------
NOTE: Please review release notes posted at:

http://www.oracle.com/pls/topic/lookup?ctx=E23824&id=SERNS
----------------------------------------------------------

# init 6

And this concludes this post.
 

Disk naming

This topic may be technically verbose due to its nature.
In the past this knowledge was much more common and relevant.
Nowadays new technologies such as ZFS have greatly simplified matters.

Anyway, when dealing with local hardware (interfaces, buses, controllers, disks) and even remote storage (SAN) it's inevitably necessary to possess at least a basic understanding of the overall scheme adopted by Solaris when abstracting these kind of peripheral devices and remote storage resources.

I've always found this topic somewhat difficult to understand because no-one had enough patience or sufficiently broad understanding at the time to present me a reasonable big picture of things. That's why I'm now attempting to fulfill this gap by writing down my current knowledge on this matter and hopefully better paving the way for others interest on this subject too.

To summarize, all or most of these details get reflected on the disk naming schemes adopted by the system which one perceives while dealing with disks devices for ZFS storage pools. The most elementary example comes from the format command. For instance:

# echo |format
Searching for disks...done

AVAILABLE DISK SELECTIONS:
       0. c1t0d0 ...
          /pci@0,0/pci8086,2829@d/disk@0,0
       1. c1t1d0 ...
          /pci@0,0/pci8086,2829@d/disk@1,0
       2. c1t2d0 ...
          /pci@0,0/pci8086,2829@d/disk@2,0
       3. c1t3d0 ...
          /pci@0,0/pci8086,2829@d/disk@3,0
       4. c1t4d0 ...
          /pci@0,0/pci8086,2829@d/disk@4,0
       5. c4t3d0 ...
          /pci@0,0/pci1000,8000@14/sd@3,0
       6. c5t7d0 ...
          /pci@0,0/pci1000,8000@16/sd@7,0

 At the very first time this information may seem quite complex!

• What's the meaning of c1t0d0 ?
• Worse, what the heck is /pci@0,0/pci8086,2829@d/disk@0,0 ?

Well, to start:
 

• The first is a logical (disk) name.
  A system-wide convenient disk name simplification.
  Actually, a symbolic link at /dev/rdsk mapping into /devices.
     
• The second is a physical (device) name.
  A file path abstraction associated to a hardware component.
  Such paths are rooted at the /devices special file-system.

LOGICAL NAMES

Let's first dig into the simpler logical names.
Logical names obey the more simple format:

c?[t?]d?[s?|p?]

Where each portion identifies the following:

c - controller #  (logical)
    Some identification assigned by the system.
    Not necessarily related to any physical ordering.

t - bus target # (physical)
    Optional: present only on bus-oriented controllers.
    Not present for direct controllers (which don't have a bus).
    Important to SCSI, iSCSI, SAS and SATA.

d - LUN # (physical)

    Always 0, except when many LUNs exist on a single bus.
    For SCSI, SAS and SATA, always 0.

s - slice #
    Usually from 0 thru 6. Optional. Mutually exclusive with p.
    Slice 2 (s2) "conventionally" means the whole disk.
    Legacy; important for old SMI (VTOC) disks.

p - fdisk partition # (i86pc only)
    Ranging from 0 thru 4. Optional. Mutually exclusive with s.
    Partition 0 (p0) means the whole disk.
    Legacy; used with PCFS on some USB devices.
 

PHYSICAL NAMES

A physical name is much more difficult to understand, not necessarity with respect to its format, but to its relationship in system internals.

One can start by looking up the logical name:

# l /dev/rdsk/c1t0d0
l...  ... /dev/rdsk/c1t0d0 
       -> /devices/pci@0,0/pci8086,2829@d/disk@0,0:wd,raw

Looking at the physical name, in reverse order, one sees:

1. The raw suffix.
   Any references to this entry means using the the device as a character device, in other words, as in a byte-by-byte mode.This raw view of the device is depicted by the letter r in rdsk device path name component.
     
2. The wd suffix.
   I deduce this is an abbreviation to whole-disk.
   This deduction comes from the following straightforward observation:
    
   # prtconf -v |ggrep -B2 /dev/rdsk/c1t0d0 \
                |grep -v spectype \
                |tr -s ' '
    dev_path=/pci@0,0/pci8086,2829@d/disk@0,0:a,raw
    dev_link=/dev/rdsk/c1t0d0s0
   --
    ...
   --
    dev_path=/pci@0,0/pci8086,2829@d/disk@0,0:g,raw
    dev_link=/dev/rdsk/c1t0d0s6
   --
    dev_path=/pci@0,0/pci8086,2829@d/disk@0,0:q,raw
    dev_link=/dev/rdsk/c1t0d0p0
   --
    ...
   --
    dev_path=/pci@0,0/pci8086,2829@d/disk@0,0:u,raw
    dev_link=/dev/rdsk/c1t0d0p4
   --
    dev_path=/pci@0,0/pci8086,2829@d/disk@0,0:wd,raw
    dev_link=/dev/rdsk/c1t0d0 
      
3. A series of device path components denoting hardware nodes.
   The general form is: hardware-node@address.
   In the example:
    
     disk@0,0
     pci8086,2829@d
     pci@0,0 
   
   The disk component is the most important one.
   Its general form is: disk@bus-target-#,LUN-#.
    
   Furthermore, the whole path is associated to a device-driver:
    
   # grep /pci@0,0/pci8086,2829@d/disk@0,0 /etc/path_to_inst
   "/pci@0,0/pci8086,2829@d/disk@0,0" 0 "sd"
   
   The device-driver name could also have been found by inspecting the entry's major-number (driver), 214, present on the special devfs file-system type of /devices: 
    
   # l /devices/pci\@0\,0/pci8086\,2829\@d/disk\@0\,0:wd,raw
   c...  214,  7  ...  /.../pci@0,0/pci8086,2829@d/disk...
    
   # grep 214 /etc/name_to_major |cut -d' ' -f1
   sd
   
   Just for completeness, in the example above, note that 7, to the right of 214, is the minor-number, denoting the driver instance unique id for that specific device.
   
   The controller's assigned # is undocumented, hence it's nearly impossible to fully understand something about that. With 4 SATA, 1 SCSI and 1 SAS disk, one gets:
   
   # prtconf -v |ggrep -B5 /dev/cfg \
                |ggrep -E -v 'spectype|\(' \
                |tr -s ' '
    dev_path=/pci@0,0/pci8086,2829@d:devctl
    dev_path=/pci@0,0/pci8086,2829@d:0
    dev_link=/dev/cfg/sata0/0
    dev_path=/pci@0,0/pci8086,2829@d:1
    dev_link=/dev/cfg/sata0/1
    dev_path=/pci@0,0/pci8086,2829@d:2
    dev_link=/dev/cfg/sata0/2
    dev_path=/pci@0,0/pci8086,2829@d:3
    dev_link=/dev/cfg/sata0/3
    dev_path=/pci@0,0/pci8086,2829@d:4
    dev_link=/dev/cfg/sata0/4
   --
    dev_path=/pci@0,0/pci1000,8000@14:devctl
    dev_path=/pci@0,0/pci1000,8000@14:scsi
    dev_link=/dev/cfg/c4
   --
    dev_path=/pci@0,0/pci1000,8000@16:devctl
    dev_path=/pci@0,0/pci1000,8000@16:scsi
    dev_link=/dev/cfg/c5

That's a reasonable kick-off, isn't it?
   

ZFS disk preparation

ZFS disk preparation is kind of legacy topic as more and more newer systems start to support EFI (GPT) labeling for disks on the ZFS root pool. So this post is about the legacy SMI (VTOC) labeled disks, not EFI (GPT) labeled disks. In general, the old scheme may still appear in the years to come, so better take some some notes on it, just in case.

There are lots and lots of perfect information on official documentation, good books, articles and posts, but still not as simple and straightforward as one would probably desired. Frequently, one has to dig into lots of information until finding the exact steps that fit the bill. Hence, I'll take some time in trying to add some little more contribution for fixing these shortcomings.

NOTE

I need to recall that the term slice and partition always refers to the same thing under a SPARC platform, but not so under an i86pc platform when dealing with the SMI (VTOC) scheme. On the later platform, a slice is implicitly understood as a sub-partition and a partition is usually spoken as fdisk partition. History tells this was so to help multi-boot coexistence with other systems, which is fully deprecated. As good practice (and sanity check) keep all slices on the same (primary) partition and rest in peace. With this good practice, most of the burden goes away and one can once more interchangeably use the terms partition and slice.

The goal is to set up an appropriate partition (slice) map in order to assure that a single mountable partition covers the whole disk, its maximum usable area. This is important in order to get the most out of ZFS by letting it enable disks' local caches and get rid of other alien coexistence. Therefore, this addresses the bold recommended to dedicate whole disks to ZFS.

For instance, let's say that the disk c8t1d0 is to be prepared in order to establish a mirrored root pool for a certain system (by the way, disk naming schemes are not part of this post). Let's also assume, at this moment and for the sake of simplicity, that the disk is already recognized by the system (usually said to be configured or available), hence listed by the format utility. Under these assumptions, one way (there are variations) to prepare it for the ZFS root pool is as follows:

# format
Searching for disks...done
 
AVAILABLE DISK SELECTIONS:
     0. c8t0d0 <...>
        /pci@0,0/pci8086,2829@1f,2/disk@0,0
     1. c8t1d0 <...>
        /pci@0,0/pci8086,2829@1f,2/disk@1,0
     ...
Specify disk (enter its number): 1
 
Under an i86pc platform, if the chosen disk had been previously used, better re-format it and you begin by deleting any current partitions on it and then creating a single SOLARIS2 partition. This can be achieved by choosing the fdisk subcommand, followed by options 3 (as needed and until no alien partition is left), 1 and 6. Then one should get something similar to:
 
     Total disk size is ... cylinders
     Cylinder size is ... (512 byte) blocks

                                   Cylinders
Partition Status Type         Start  End Length  %
========= ====== ============ =====  === ====== ===
1         Active Solaris2         1  ...  ....  100
  
SELECT ONE OF THE FOLLOWING:
   1. Create a partition
   2. Specify the active partition
   3. Delete a partition
   4. Change between Solaris and Solaris2 Partition IDs
   5. Edit/View extended partitions
   6. Exit (update disk configuration and exit)
   7. Cancel (exit without updating disk configuration)
Enter Selection: 6

A simpler case happens if originally no alien partitions were present such as when the disk is brand new or have been cleaned up previously. When initially choosing such a disk the format command will display:

...
selecting c8t1d0
[disk formatted]
No Solaris fdisk partition found.
...

Under this condition the fdisk subcommand will again inform the same and ask what should be done, which answer should by y to create a single i86pc partition (here, not a slice) of type SOLARIS2 for the whole disk capacity:

No fdisk table exists.
The default partition for the disk is:

  a 100% "SOLARIS System" partition

Type "y" to accept the default partition, 
otherwise type "n" to edit the partition table.
y
 
Although not strictly required at this point, after any of the above cases, it may be of good practice committing the changes so far (the fdisk partition creation also creates a default slice map within the fdisk partition), by doing as follows:
 
format> label
Ready to label disk, continue? yes

At this point, both for SPARC and i86pc platform, it's necessary to define the partition (slice) 0 (traditionally called the root partition if back on the old age of UFS file-systems) as covering the maximum available disk capacity. Fortunately, it's possible to cover both platform cases with the same sequence of required format subcommands as follows (but the example below is from an i86pc platform):
   
format> partition 
...

format> modify
Select partitioning base:
        0. Current partition table (original)
        1. All Free Hog
Choose base (enter number) [0]? 1

Part     Tag Flag Cylinders      Size         Blocks
0       root wm   0             0      (0/0/0)          0
1       swap wu   0             0      (0/0/0)          0
2     backup wu   0 - 1020   1021.00MB (1021/0/0) 2091008
3 unassigned wm   0             0      (0/0/0)          0
4 unassigned wm   0             0      (0/0/0)          0
5 unassigned wm   0             0      (0/0/0)          0
6        usr wm   0             0      (0/0/0)          0
7 unassigned wm   0             0      (0/0/0)          0
8       boot wu   0 -    0      1.00MB (1/0/0)       2048
9 alternates wm   0             0      (0/0/0)          0

Do you wish to continue creating a new partition
table based on above table[yes]?
Free Hog partition[6]? 0
Enter size of partition '1' [0b, 0c, 0.00mb, 0.00gb]:
Enter size of partition '3' [0b, 0c, 0.00mb, 0.00gb]:
Enter size of partition '4' [0b, 0c, 0.00mb, 0.00gb]:
Enter size of partition '5' [0b, 0c, 0.00mb, 0.00gb]:
Enter size of partition '6' [0b, 0c, 0.00mb, 0.00gb]:
Enter size of partition '7' [0b, 0c, 0.00mb, 0.00gb]:

Part     Tag Flag Cylinders     Size            Blocks
0       root wm   1 - 1020  1020.00MB (1020/0/0) 2088960
1       swap wu   0            0      (0/0/0)          0
2     backup wu   0 - 1020  1021.00MB (1021/0/0) 2091008
3 unassigned wm   0            0      (0/0/0)          0
4 unassigned wm   0            0      (0/0/0)          0
5 unassigned wm   0            0      (0/0/0)          0
6        usr wm   0            0      (0/0/0)          0
7 unassigned wm   0            0      (0/0/0)          0
8       boot wu   0 -    0     1.00MB (1/0/0)       2048
9 alternates wm   0            0      (0/0/0)          0

Okay to make this the current partition table[yes]?
Enter table name (remember quotes): "c8t1d0"

Ready to label disk, continue? y

partition> print
Current partition table (c8t1d0):
Total disk cylinders available: 1020 + 2 (reserved cylinders)

Part     Tag Flag Cylinders     Size         Blocks
0 unassigned wm   1 - 1019  1019.00MB (1019/0/0) 2086912
1 unassigned wm   0            0      (0/0/0)          0
2     backup wu   0 - 1019  1020.00MB (1020/0/0) 2088960
3 unassigned wm   0            0      (0/0/0)          0
4 unassigned wm   0            0      (0/0/0)          0
5 unassigned wm   0            0      (0/0/0)          0
6 unassigned wm   0            0      (0/0/0)          0
7 unassigned wm   0            0      (0/0/0)          0
8       boot wu   0 -    0     1.00MB (1/0/0)       2048
9 unassigned wm   0            0      (0/0/0)          0

partition> label
Ready to label disk, continue? y
 
partition> quit
format> quit

NOTE

Just for curiosity, the Flags column above can have 4 possible values:

• wm : writable-mountable
• wu : writable-unmountable
• rm : readable-mountable
• ru : readable-unmountble

One can also see that under an i86pc, s8 is assigned 1 cylinder, cylinder 0, for holding some boot information, which size, in this particular disk geometry, takes 2048 blocks totaling 1 MB. Therefore, each block is 512 bytes long, which suggests that a block is the same thing of a sector because, in general, it's invariably that long on current disks.

Invoke the format command once more to double-check the results. I have faced a situation that upon doing so, the following message appeared:

Note: detected additional allowable expansion storage space
that can be added to current SMI label's computed capacity.
Select to adjust the label capacity.

In following the above instructions another messaged appeared:
(which I noted that s0 hadn't been updated accordingly)

Expansion of label cannot be undone; continue (y/n) ? y
The expanded capacity was added to the disk label and "s2".
Disk label was written to disk. 

So to close the above loop, I repeated all the previously shown steps since the modify sub-command and all seemed fine since then. This time I "gained" just 1 MB, but who knows?

NOTE

It's noticeable that partition 2 (slice s2) overlaps with partition 0 (slice s0). This is not an issue, but forces the use of -f flag with the zfs attach command. It should be said that this slice must be kept in order to make possible an eventual (re)installation of the boot-loader (at least under the i86pc platform).

Now that things are presumably more clear, it can be recognized as simple after all, which, in fact, it is. But one issue remains: it's not automated, thus inefficient or impractical if many systems are to be prepared in a row, such as on a not so small virtualization or cloud infrastructure. Fortunately the procedure can be streamlined, as long as all the disks have equal geometry (which makes sense for deployments in large chunks). What needs to be done is:

1. Do the manual process once and then save the result as a template:
   # prtvtoc /dev/rdsk/c8t1d0s2 >/tmp/vtoc-template
    
2. Repeatedly apply the template, preferably via some ordinary scripting strategy:
   # fmthard -s /tmp/vtoc-template /dev/rdsk/c?[t?]d?s2 
      

 And that's it for SMI (VTOC) labeled disks preparation for ZFS.
  

ZFS basic mirroring

Mirroring is a traditional strategy in providing fault-tolerance which became more popular for secondary storage systems, typically for the hard-disks. ZFS improves the strategy by introducing checksums to prevent that eventually corrupted data (due to bit-rot or some other component malfunction) at one side of the mirror gets replicated to the other healthy side of the mirror. This ZFS enhancement has been unique since in general it seems not viable to implement it solely at the physical layer (controller) as it may have dependencies at the logical layer (file-systems). ZFS achieves its goal by abstracting the physical layer into storage pools over which logical datasets (file-systems ans raw volumes) are managed.

Establishing mirrors within storage pools is a relatively simple task, specially in more recent versions of Solaris such as the Solaris 11.x. But in late Solaris 10 U1x as well as in Solaris 11 Express some initial disk preparation was required. In addition, for root pools under these older systems, it was necessary to manually install (via installboot(1M) or installgrub(1M)) the boot-loader on new disks just integrated into a mirror. On more recent versions of Solaris the bool-loader management for mirrored root pools were automated yet eventually manually manageable via the install-bootloader sub-command of bootadm(1M).

Another usual difference contrasting older systems (Solaris 10 U1x and Solaris 11 Express) from newer Solaris 11.x is as how underlying disks comprising storage pools are seem with respect to disk-labeling: SMI (VTOC) for older systems and disks and EFI (GPT) for newer ones. The most important implications about these two types of label is that SMI labels impose a limit of 2 TB of usable storage even on larger disks and are the only supported label for root pools under older systems. Typically, referring to whole-disks (which are preferred for ZFS over legacy slices/partitions), when a SMI label is used, disks names take the form c?[t?]d?s0 , otherwise they lack the trailing s0.

Here is some exemplification on how to hassle-free successfully establish a basic mirror on storage pools initially consisting on a single disk:


1) When SMI-labeled disks are required for a pool:

This is typical for older systems in general, for some SPARC systems or for not-so-old systems that don't yet support EFI devices on the root pool.

I assume that the disks were already appropriately prepared.

# zpool status
  pool: rpool
 state: ONLINE
 scan: ...
config:

        NAME          STATE     READ WRITE CKSUM
        rpool         ONLINE       0     0     0
          c8t0d0s0    ONLINE       0     0     0

errors: No known data errors

# zpool attach -f rpool c8t0d0s0 c8t1d0s0
Make sure to wait until resilver is done before rebooting.

# zpool status
  pool: rpool
 state: ONLINE
status: One or more devices is currently being resilvered.  
        The pool will continue to function, 
        possibly in a degraded state.
action: Wait for the resilver to complete.
 scan: resilver in progress since ...
    1.50G scanned out of 3.78G at ...M/s, 0h2m to go
    1.50G resilvered, 39.70% done
config:

        NAME          STATE     READ WRITE CKSUM
        rpool         ONLINE       0     0     0
          mirror-0    ONLINE       0     0     0
            c8t0d0s0  ONLINE       0     0     0
            c8t1d0s0  ONLINE       0     0     0  (resilvering)

errors: No known data errors

As this is a root pool, when the resilver is complete, one can optionally make sure the boot-loader is properly installed on the newly attached disk as well. But according to the official documentation, this extra step is only mandatory when a zpool replace command is issued on the root pool. For an i86pc system, if one decide so, the command would be similar to:

# installgrub \
  /boot/grub/stage1 /boot/groub/stage2 \
  /dev/rdsk/c8t1d0s0

or the newer and far superior:

# bootadm install-bootloader


2) Systems supporting EFI-labeled disks for any kind of pool:

This is good news as no tedious disk preparation is required beforehand at all and moreover is totally useless as during an attachment the disk will be automatically formatted and labeled as necessary and accordingly.

Therefore, the attachment procedure is as simple as:

# zpool attach rpool c1t0d0 c1t1d0

NOTE

It's possible to have N disks in a mirror which means the mirror will withstand as many as N-1 members failing at a given time. This may seem highly exaggerated at first but it may make sense on some scenarios.

But let me exclude the case of a 3-way mirror for a root pool with over 2 TB disks as an insane case: a root pool should really never require that much space justifying a 3rd member preventing a double-fault while resilvering from a single-fault.

For instance, a N-way mirror (N>3) for a non-root critical pool may make sense and be an straightforward solution if one intends to keep critical data replicated at N-2 remote locations. The mirrored devices (not disks) forming this pool could be iSCSI LUNs from separate remote storage facilities (preferably also backed by ZFS) as long as each LUN isn't comprised of many individual disks and as long as the pool also keeps local log and cache devices indispensable for better equalizing disparate remote storage performances and link latencies.

NOTE

Mirrors can be created or added right from the start with a single command, such as:

# zpool create hq \
  mirror c0t0d0 c1t0d0 \
  mirror c0t1d0 c2t0d0

(each mirror above will resist a single disk and controller failure)
(it's similar to RAID-10 but RAID-Z(1) could rival if I/O block is over 128KB)

# zpool add hq \
  mirror c1t1d0 c2t1d0

(the hq pool above is now stripping over 3 2-way mirrors)
(a better solution could be a RAID-Z2 scheme depending on block size)
 
Each mirror on the example above is known as a vdev.
Not surprisingly, ZFS stripes I/O along the top-level vdevs.
By the way, root pools support just 1 mirror vdev.

To remove a device from a mirror:
# zpool detach rpool c1t1d0

To replace a device in a mirror:
# zpool replace rpool c1t1d0 c1t2d0

And that seems the pretty much basics.