VirtualBox VM additions

I took a long while until the VirtualBox Guest Additions for Solaris become available. So long that I used to not count on it ever more. But then all of a sudden for some reason I can't remember why I mounted the VirtualBox 5.1.18 Guest Additions on a Solaris 11.3 and happily noticed that, yes, they were there! It's an old (SVr4) Solaris package file (.pkg extension). So I immediately set to install it but, as usual, there was no available documentation, at least I couldn't easily find it. At first I tried an unattended SVr4 package install but it didn't work so I was forced to do it interactively.

For recap I started by mounting the VirtualBox Guest Additions as follows:

And then the usual optical media icon kicked in on my desktop:

But double-clicking on it won't work as expected, unfortunately. They probably didn't have the time for this final perfection, but I wonder if that was a typical case of lazyness; whatever, I went to the very comfortable CLI and served myself:

# cd /media/VBOXADDITIONS_5.1.18_114002

# ll *.pkg
-r-xr-xr-x   1 root root  17M ... VBoxSolarisAdditions.pkg
 
# pkgadd -d VBoxSolarisAdditions.pkg all

Processing package instance <SUNWvboxguest> from </media/VBOXADDITIONS_5.1.18_114002/VBoxSolarisAdditions.pkg>

Oracle VM VirtualBox Guest Additions(i386) 5.1.18,REV=r114002.2017.03.15.16.33
Oracle Corporation
Using </> as the package base directory.
## Processing package information.
## Processing system information.
## Verifying package dependencies.
## Verifying disk space requirements.
## Checking for conflicts with packages already installed.
## Checking for setuid/setgid programs.

... contains scripts which will be executed with super-user
permission during the process of installing this package.

... continue with the installation of <SUNWvboxguest> [y,n,?] y

Installing ... Guest Additions as <SUNWvboxguest>

## Installing part 1 of 1.
/etc/fs/vboxfs/mount <symbolic link>
/opt/VirtualBoxAdditions/1099.vboxclient
/opt/VirtualBoxAdditions/LICENSE
/opt/VirtualBoxAdditions/VBox.sh
/opt/VirtualBoxAdditions/VBoxControl
/opt/VirtualBoxAdditions/amd64/VBoxClient.Z
/opt/VirtualBoxAdditions/amd64/VBoxControl.Z
/opt/VirtualBoxAdditions/amd64/VBoxService.Z
/opt/VirtualBoxAdditions/amd64/pam_vbox.so
/opt/VirtualBoxAdditions/amd64/vboxfs
/opt/VirtualBoxAdditions/amd64/vboxfs_s10
/opt/VirtualBoxAdditions/amd64/vboxfsmount
/opt/VirtualBoxAdditions/amd64/vboxmslnk
/opt/VirtualBoxAdditions/amd64/vboxvideo_drv_110.so.Z
/opt/VirtualBoxAdditions/amd64/vboxvideo_drv_111.so.Z
/opt/VirtualBoxAdditions/amd64/vboxvideo_drv_112.so.Z
/opt/VirtualBoxAdditions/amd64/vboxvideo_drv_113.so.Z
/opt/VirtualBoxAdditions/amd64/vboxvideo_drv_114.so.Z
/opt/VirtualBoxAdditions/amd64/vboxvideo_drv_117.so.Z
/opt/VirtualBoxAdditions/amd64/vboxvideo_drv_118.so.Z
/opt/VirtualBoxAdditions/amd64/vboxvideo_drv_13.so.Z
/opt/VirtualBoxAdditions/amd64/vboxvideo_drv_14.so.Z
/opt/VirtualBoxAdditions/amd64/vboxvideo_drv_15.so.Z
/opt/VirtualBoxAdditions/amd64/vboxvideo_drv_16.so.Z
/opt/VirtualBoxAdditions/amd64/vboxvideo_drv_17.so.Z
/opt/VirtualBoxAdditions/amd64/vboxvideo_drv_18.so.Z
/opt/VirtualBoxAdditions/amd64/vboxvideo_drv_19.so.Z
/opt/VirtualBoxAdditions/amd64/vboxvideo_drv_70.so.Z
/opt/VirtualBoxAdditions/amd64/vboxvideo_drv_71.so.Z
/opt/VirtualBoxAdditions/i386/VBoxClient.Z
/opt/VirtualBoxAdditions/i386/VBoxControl.Z
/opt/VirtualBoxAdditions/i386/VBoxService.Z
/opt/VirtualBoxAdditions/i386/pam_vbox.so
/opt/VirtualBoxAdditions/i386/vboxfs
/opt/VirtualBoxAdditions/i386/vboxfs_s10
/opt/VirtualBoxAdditions/i386/vboxfsmount
/opt/VirtualBoxAdditions/i386/vboxmslnk
/opt/VirtualBoxAdditions/i386/vboxvideo_drv_110.so.Z
/opt/VirtualBoxAdditions/i386/vboxvideo_drv_111.so.Z
/opt/VirtualBoxAdditions/i386/vboxvideo_drv_112.so.Z
/opt/VirtualBoxAdditions/i386/vboxvideo_drv_113.so.Z
/opt/VirtualBoxAdditions/i386/vboxvideo_drv_114.so.Z
/opt/VirtualBoxAdditions/i386/vboxvideo_drv_117.so.Z
/opt/VirtualBoxAdditions/i386/vboxvideo_drv_118.so.Z
/opt/VirtualBoxAdditions/i386/vboxvideo_drv_13.so.Z
/opt/VirtualBoxAdditions/i386/vboxvideo_drv_14.so.Z
/opt/VirtualBoxAdditions/i386/vboxvideo_drv_15.so.Z
/opt/VirtualBoxAdditions/i386/vboxvideo_drv_16.so.Z
/opt/VirtualBoxAdditions/i386/vboxvideo_drv_17.so.Z
/opt/VirtualBoxAdditions/i386/vboxvideo_drv_18.so.Z
/opt/VirtualBoxAdditions/i386/vboxvideo_drv_19.so.Z
/opt/VirtualBoxAdditions/i386/vboxvideo_drv_70.so.Z
/opt/VirtualBoxAdditions/i386/vboxvideo_drv_71.so.Z
/opt/VirtualBoxAdditions/solaris_xorg.conf
/opt/VirtualBoxAdditions/solaris_xorg_modeless.conf
/opt/VirtualBoxAdditions/vbox_vendor_select
/opt/VirtualBoxAdditions/vboxclient.desktop
/opt/VirtualBoxAdditions/vboxguest.sh
/opt/VirtualBoxAdditions/x11config15sol.pl
/opt/VirtualBoxAdditions/x11restore.pl
/usr/bin/VBoxClient <symbolic link>
/usr/bin/VBoxClient-all <symbolic link>
/usr/bin/VBoxControl <symbolic link>
/usr/bin/VBoxService <symbolic link>
/usr/kernel/drv/amd64/vboxguest
/usr/kernel/drv/amd64/vboxms
/usr/kernel/drv/vboxguest
/usr/kernel/drv/vboxguest.conf
/usr/kernel/drv/vboxms
/usr/kernel/drv/vboxms.conf
/usr/lib/VBoxOGL.so
/usr/lib/VBoxOGLarrayspu.so
/usr/lib/VBoxOGLcrutil.so
/usr/lib/VBoxOGLerrorspu.so
/usr/lib/VBoxOGLfeedbackspu.so
/usr/lib/VBoxOGLpackspu.so
/usr/lib/VBoxOGLpassthroughspu.so
/usr/lib/amd64/VBoxOGL.so
/usr/lib/amd64/VBoxOGLarrayspu.so
/usr/lib/amd64/VBoxOGLcrutil.so
/usr/lib/amd64/VBoxOGLerrorspu.so
/usr/lib/amd64/VBoxOGLfeedbackspu.so
/usr/lib/amd64/VBoxOGLpackspu.so
/usr/lib/amd64/VBoxOGLpassthroughspu.so
/usr/sbin/vboxmslnk <sUnattended SVr4 pkg installymbolic link>
[ verifying class <none> ]
/opt/VirtualBoxAdditions/VBoxClient <linked pathname>
/opt/VirtualBoxAdditions/VBoxISAExec <linked pathname>
/opt/VirtualBoxAdditions/VBoxService <linked pathname>
/opt/VirtualBoxAdditions/vboxmslnk <linked pathname>
[ verifying class <manifest> ]
## Executing postinstall script.
Uncompressing files...
Configuring VirtualBox guest kernel module...
VirtualBox guest kernel module loaded.
VirtualBox pointer integration module loaded.
Creating links...
Installing video driver for X.Org 1.14.5...
Configuring client...
Installing 64-bit shared folders module...
Installing 32-bit shared folders module...
Configuring services (this might take a while)...
Enabling services...
Updating boot archive...
Done.
Please re-login to activate the X11 guest additions.
If ... just un-installed the previous ... a REBOOT is required.

Installation of <SUNWvboxguest> was successful.

# eject
 
$ pkginfo -l SUNWvboxguest
   PKGINST:  SUNWvboxguest
      NAME:  Oracle VM VirtualBox Guest Additions
  CATEGORY:  application
      ARCH:  i386
   VERSION:  5.1.18,REV=r114002.2017.03.15.16.33
   BASEDIR:  /
    VENDOR:  Oracle Corporation
      DESC:  ... Guest Additions for Solaris guests
    PSTAMP:  vboxguest20170315163319_r114002
  INSTDATE:  May 08 2017 17:03
   HOTLINE:  Please contact your local service provider
     EMAIL:  info@virtualbox.org
    STATUS:  completely installed
     FILES:       80 installed pathnames
                   4 linked files
                   5 directories
                  21 executables
               37263 blocks used (approx)
   

VirtualBox VM autostart

The VirtualBox VM autostart feature first appeared in version 4.2.0. It's intended purpose is not only to automatically start selected virtual machines upon the host system startup, but to automatically and gracefully stop them upon the host system shutdown. But, of course, unless perhaps the VM is interactive, the stop part of the intent is fictional, specially on headless Solaris server systems. Thus for the stop part the feature acts like a placebo, at least for Solaris hosts.

I've been almost comfortable with headless startups for some time so I've never feel inclined to use the autostart, but now on version 5.1.18 I'm using VirtualBox frequently enough to justify taking some time to set it up and relax about manually starting (but not stopping!) VMs, now a tedious task.

There is a bare minimum documentation about this feature, as it's the case with many other features. In general, there are some mistakes and many omissions in the community provided documentation. Eventually it shall catch up but it would be wise not counting so much on it, anyway...

In a Solaris host environment the solution mechanism is elegant (yet unfortunately there some bugs requiring workarounds) as the autostart feature has been integrated/encapsulated into a SMF service:

The physical memory

The physical memory is a crucial and precious resource and is commonly one of the major system bottlenecks as well as one of the system components that bumps a system price to the skies.

Knowing and, better yet, determining at runtime the amount of physical memory that is physically installed on a host and that is actually available to the system is important to many deployment and administration strategies.

Without recurring to programming at the system APIs level, it is possible to easily determine such figures as shown below.

$ prtconf | grep Mem
Memory size: 8192 Megabytes

# echo ::memstat | mdb -k | grep Total
Total            2096958            7.9G

$ kstat -p -n system_pages | egrep 'avail|physmem|locked|total'
unix:0:system_pages:availrmem    930155
unix:0:system_pages:pageslocked 1162706
unix:0:system_pages:pagestotal  2092861
unix:0:system_pages:physmem     2092861

Note that pagestotal = availrmem + pageslocked and that it seems that interestingly pagestotal = physmem, all in multiples of page sizes.

$ pagesize
4096

Then we can now compare things and better grasp the reality:

$ echo "(2092861 * `pagesize`) / 1024 ^ 2" | bc
8175

$ echo "(2096958 * `pagesize`) / 1024 ^ 2" | bc
8191
 
To me the 16 MB (4096 pages) difference between 8191 and 8175 seems to be fixed (non-pageable) and is still a mystery, a matter to open investigation, perhaps some part of the kernel known only by the internal staff.

That is, according to system's best result it actually sees 8191 MB, 1 MB less than what's physically installed on the host and that's not so hard to wonder why (perhaps set aside for the on-board video or so). Using closer to perfect figures ought to provide more exact results for planning and assessments.
 

Kernel zones support

The advent of kernel zones in Solaris 11.2 is another great improvement to Solaris. But it may not be supported on aging hardware as I may have just found out. I happen to use a box more than 5 years old, from later 2009, which seems not support all the required virtualization technology for kernel zones. But I'm pending confirmation if my issue is just because I have VirtualBox installed on my x86-64 and this is posing some sort of conflict with kernel zones availability in terms of lack of (already allocated to VirtualBox) virtualization resources.

So if you're planning to set aside some "cool hardware" for your Solaris 11.3 kernel zones, I suggest you learn from this experience of mine beforehand in order to make sure your "cool hardware" and system setup meet all the requirements.

You may start by checking the man page solaris-kz(5):

...

The  solaris-kz  brand  uses certain hardware features which may not be available in older systems, or in virtualized environments.  To  detect whether  a  system  supports  the  solaris-kz brand, install the brand-solaris-kz package and then run the virtinfo command.

# virtinfo -c supported list kernel-zone

If kernel-zone is not shown in the supported list, you can  see  syslog for  more information. Messages pertaining to kernel zones will contain the string kernel-zone.
 
...

In my case, in general, I've got:

$ virtinfo -c supported list kernel-zone
kernel-zone: no such supported virtual environment found

$ virtinfo
NAME            CLASS    
non-global-zone supported

And under VirtualBox 5.1.18 r114002, I've got:

$ virtinfo
NAME            CLASS    
virtualbox      current  
non-global-zone supported

Well, it's true that in the logs you'll have to look for kernel-zone.
But you'll have to do so in /var/adm/messages instead.
So I set out to further investigate what was missing.

For my physical box I've got:

$ grep kernel-zone /var/adm/messages | cut -d: -f5,6 | sort -u
... environment not supported: VMX already in use
... unsupported Intel model 15

And under VirtualBox (on that same physical box), I've got:

$ grep kernel-zone /var/adm/messages | cut -d: -f5,6 | sort -u
... environment not supported: CPU doesn't have VMX

According to a wikipedia article on x86 virtualization, VMX happen to be the designation for the CPU flag related to VT-x support. What caught my attention was the single message VMX already in use. It appeared just once and it's true I have enabled virtualization support on my physical box's BIOS which makes me wonder if the situation would change in favor of kernel-zones meeting all its requirements if I completely uninstall VirtualBox. I haven't tried it yet nor I'm willing to do it right now as I do heavy use of VirtualBox. But depending on the scenario, the trade-off would certainly pay off.

By the way, I'd like to mention that I did try something less drastic than uninstalling VirtualBox. I tried disabling (setting to off) the VirtualBox's property hwvirtexclusive but that didn't make any difference in solving the problem (at least as to version 5.1.18). Later I found a forum entry about this that claims to have worked, but this was for earlier versions.
    

NIS & logins - troubleshooting #1

Despite all the quality and stability of Solaris 11 sometimes things go wrong.
Before pinning up anything it's important to assess all the variables.

For instance, upon forcibly rebooting a x86 with a few Solaris 11 zones, one of them acting as a NFS server, within a NIS services infrastructure something odd happened; the following values simply vanished:

• The config/* of name-service/switch SMF service.
• The config/domainname of nis/domain SMF service.
• The sharectl nfsmapid_domain.

  
The resolution was simply reentering the correct values, of course.
But until narrow down to what was wrong the symptoms were diverse:

• The NIS users were not being able to log in.
• The NFS server was not resolving any uid and gid.
• The NIS clients were listing nobody for user and group.

Nevertheless it's most important to determine the cause.
In my specific case I suspect of the well known:

Using ZFS Storage Pools in VirtualBox

Yes, I'm using VirtualBox for composing many posts.
Yes, due to a video adapter failure the host system had to be halted for repair.
And, yes, I haven't enabled the cache flushing on the VMs.
  

Shared memory run sample 1.0

Let's follow a shared memory code sample 1.0 on a debugger and watch.
This should clear up any remaining doubts and misunderstandings.
Yet for an additional example, check shared memory code sample 2.0.

The particular environment configuration where the code is run isn't relevant.
I believe it suffices to say that it presents:

• Solaris 11.1 SRU 7.5 running on HP ProLiant BL460c G7.
   
• 24 GiB of physical memory and 18 GiB available virtual memory.
  This is due to some physical memory being reserved to VirtualBox.
  VirtualBox has no other relation to this post, it's there by accident.
  Sure there are VirtualBox guests running and consuming memory.
     
• Disk-based swap is defined at 4 GiB and is not being used at first.
  Certainly, the available virtual memory includes this figure.
     
• The binary will be run under group.staff project resource controls.
  The resource control default was at around 5.99 GiB, ¼ of physical memory.
  Hence, it was adjusted to 20 GiB in order to cause no interferences here.
  

So here is the essential starting point settings and figures:

# tail /etc/system

...

* 24GB - 1.5GB -> 22GB - 5% -> 20GB due to VirtualBox

set zfs:zfs_arc_max=0x400000000

# swap -lh; swap -sh

swapfile                   dev    swaplo   blocks     free

/dev/zvol/dsk/rpool/swap 214,2        4K     4.0G     4.0G

total: 436M allocated + 211M reserved = 648M used, 18G available

# projmod 
  -K "project.max-shm-memory=(privileged,20G,deny)" 
  group.staff

# projects -l group.staff

group.staff

 projid : 10

 comment: ""

 users  : (none)

 groups : (none)

 attribs: project.max-shm-memory=(privileged,21474836480,deny)

---

The 1^st sample run will require 17 GiB of ISM.
As seen this would require enough available lockable physical memory.
This particular run will fail, but note how the swap figures are affected.
Virtual memory isn't directly involved, but it's availability is recomputed.
This is because virtual memory is partially backed by free physical memory.
As physical memory is reserved, less of it is free for backing virtual memory.
This was also discussed in the swap analysis post.

The essential excerpts are as follows:

# swap -lh; swap -sh

swapfile             dev    swaplo   blocks     free

/dev/swap             -         4K     4.0G     4.0G
total: 448M allocated + 212M reserved = 660M used, 18G available

// size == 17 GiB

int id = ::shmget( 1, size, IPC_CREAT | IPC_EXCL | 0660 );

# swap -lh; swap -sh

swapfile             dev    swaplo   blocks     free

/dev/swap             -         4K     4.0G     4.0G
total: 448M allocated + 17G reserved = 18G used, 772M available

// Failure!

void * p = ::shmat( id, 0, SHM_SHARE_MMU );

# swap -lh; swap -sh

swapfile             dev    swaplo   blocks     free

/dev/swap             -         4K     4.0G     4.0G
total: 448M allocated + 17G reserved = 18G used, 816M available

::shmctl( id, IPC_RMID, 0 );

# swap -lh; swap -sh

swapfile             dev    swaplo   blocks     free

/dev/swap             -         4K     4.0G     4.0G
total: 448M allocated + 212M reserved = 660M used, 18G available

---

The 2^nd sample run will also require 17 GiB, this time of DISM kind.
As seen this would entirely rely on virtual memory and directly affect swap.
This particular run will succeed as no physical memory is initially required.

The essential excerpts are as follows:

# swap -lh; swap -sh

swapfile             dev    swaplo   blocks     free

/dev/swap             -         4K     4.0G     4.0G

total: 448M allocated + 212M reserved = 660M used, 18G available

 

// size == 17 GiB

int id = ::shmget( 1, size, IPC_CREAT | IPC_EXCL | 0660 );

# swap -lh; swap -sh

swapfile             dev    swaplo   blocks     free

/dev/swap             -         4K     4.0G     4.0G

total: 448M allocated + 17G reserved = 18G used, 800M available

// Success!

void * p = ::shmat( id, 0, SHM_PAGEABLE );

# swap -lh; swap -sh

swapfile             dev    swaplo   blocks     free

/dev/swap             -         4K     4.0G     4.0G

total: 448M allocated + 17G reserved = 18G used, 792M available

# ipcs -mA

IPC status ...
T ID KEY ... NATTCH       SEGSZ  CPID ... ISMATTCH     PROJECT
Shared Memory:
m 24 0x1 ...      1 18253611008 10385 ...        1 group.staff

# pmap -x 10385

10385:    /.../shm_v01/dist/Debug/...

         Address    Kbytes       RSS ... Mapped File

0000000000400000         8         8     shm_v01

0000000000411000         4         4     shm_v01

0000000000412000       160        68       [ heap ]

FFFF80FB40000000  17825792         -       [ dism shmid=0x18 ]

...

// Somewhat lengthy run...

::memset( p, '*', size );

# swap -lh; swap -sh

swapfile             dev    swaplo   blocks     free

/dev/swap             -         4K     4.0G     4.0G

total: 448M allocated + 17G reserved = 18G used, 792M available

# swap -lh; swap -sh

swapfile             dev    swaplo   blocks     free

/dev/swap             -         4K     4.0G     4.0G

total: 8.6G allocated + 9.0G reserved = 18G used, 792M available

# swap -lh; swap -sh

swapfile             dev    swaplo   blocks     free

/dev/swap             -         4K     4.0G     4.0G

total: 15G allocated + 2.7G reserved = 18G used, 792M available

# swap -lh; swap -sh

swapfile             dev    swaplo   blocks     free

/dev/swap             -         4K     4.0G     3.7G

total: 17G allocated + 214M reserved = 18G used, 792M available

# pmap -x 10385

10385:    /.../shm_v01/dist/Debug/...

         Address    Kbytes       RSS ... Mapped File

0000000000400000         8         8     shm_v01

0000000000411000         4         4     shm_v01

0000000000412000       160        68       [ heap ]

FFFF80FB40000000  17825792  17510200       [ dism shmid=0x18 ]

...

switch ( ::shmdt( p ) )

# pmap -x 10385

10385:    /.../shm_v01/dist/Debug/...

         Address    Kbytes       RSS ... Mapped File

0000000000400000         8         8     shm_v01

0000000000411000         4         4     shm_v01

0000000000412000       160        68       [ heap ]

...

# swap -lh; swap -sh

swapfile             dev    swaplo   blocks     free

/dev/swap             -         4K     4.0G     3.7G

total: 17G allocated + 214M reserved = 18G used, 792M available

switch ( ::shmctl( id, IPC_RMID, 0 ) )

# swap -lh; swap -sh

swapfile             dev    swaplo   blocks     free

/dev/swap             -         4K     4.0G     4.0G

total: 460M allocated + 201M reserved = 664M used, 18G available

From the above output we can list a few interesting conclusions, which I believe certainly will help improve knowledge on shared memory:
    

• ISM relies exclusively on the ability to lock pages of physical memory. It doesn't really participate on virtual memory (virtual swap) although its usage affects the total amount of available virtual memory since any unused physical memory is also used to form virtual memory (virtual swap). As such, disk-based swap space (swap -lh) is never impacted by ISM. At least in theory, ISM can offer better performance as it isn't subject to memory paging.
     
• DISM shares the optimizations of ISM but relies on a completely distinct layer, the virtual memory (virtual swap), not on free physical memory. The main advantages of DISM over ISM is its ability to be larger than the lockable available physical memory and be paged out from memory if it's not being used. So it does can make active use of disk-based swap space (swap -lh). If paging becomes frequent, then performance will certainly suffer.  
     
• By the last output of pmap -x 10385 before ::shmdt(), from the 17825792 Kbytes reserved (= SEGSZ of 18253611008 listed by ipcs -mA), 17510200 KB (or 17930444800 bytes) could fit in physical memory and the rest was accommodated on disk-based swap (18253611008 - 17930444800 = 323166208 ≅ 0.3 GB ). I suspect that the largest lockable available physical memory by that time was that figure of 17510200. In general, I'm not sure if there's an objective method for finding this out.
      
• Although DISM, by default, doesn't lock any memory as ISM, applications can judiciously perform that. Unfortunately, on previous output samples, I have omitted the Locked column of pmap -x output as they were null. When ISM succeeds, the three columns Kbytes, RSS and Locked list the same figure. For DISM, the last two generally vary or float, with Kbytes ≥ RSS ≥ Locked, and which is precisely the advantage of DISM.
    

For very specialized applications such as Oracle Database, I'd say that if the server hardware is dedicated to an instance of it, then DISM would be advantageous with respect to the ability to dynamically adjusting its memory structures up to the defined DISM limit, avoiding application restarts.
 
In terms of resource control, beyond max-shm-memory, there is the max-shm-ids which refers to how many pointers or segments of shared memory can be obtained. What's important to know is that the larger the memory page size, the lesser of them are needed. Thus, SPARC architectures, possessing larger memory pages, tend to consume less of them. In addition, there is the max-locked-memory which affects both ISM and DISM. Remember that DISM may also lock physical memory pages. Of course max-rss, max-swap and max-address-space may also affect limits. I expect to explore some or all of this resource controls later on another post.
  

VirtualBox VM startup

A fully configured VM is expected to be ready to launching.
I like virtualized Solaris guests from a Solaris 11 host with no desktop (GUI) installed.

For RDP access so guest console is accessible:

$ VBoxHeadless --startvm vm1 &

For no console access (generally not a good idea, unless security constrains mandate):
 

$ VBoxHeadless --startvm vm1 --vrde off &

Alternatively, on more recent versions of VirtualBox you may be more attracted to the VirtualBox VM autostart feature, which saves us typing the above commands for the VMs used every time the host system is booted.
  

VirtualBox setup information

This post is just to wrap up the basic toolset for operating VirtualBox.
Listing available VMs and its detailed configuration:
 
$ VBoxManage list vms
$ VBoxManage showvminfo vm1 --details | less

Other useful options can be obtained from:
 
$ VBoxManage list 
  

VirtualBox VM RDP encryption

Nowadays encrypting everything seems a requirement.
New SPARC and Intel systems posseses hardware support for off-loading encryption.

VirtualBox can reference a self-signed CA, certificates and private keys for RDP.

We'll be using the PEM (Privacy Enhanced Mail) format for certificates and keys.
It's primarily used by UNIX and consists on a Base64 representation of the binary DER.

PEM's content is delimited by headers and footers, such as:

-----BEGIN CERTIFICATE-----   -----BEGIN ENCRYPTED PRIVATE KEY-----

-----END CERTIFICATE-----     -----END ENCRYPTED PRIVATE KEY-----

Create a directory to store the X.509 certificates and RSA keys generated by OpenSSL:

  

$ mkdir -m 0700 ~/.VirtualBox/encryption
$ cd ~/.VirtualBox/encryption

Create the CA by creating its self-signed certificate:
  
$ openssl req 
  -new -x509
  -days 365
  -extensions v3_ca 
  -keyout ca_priv_key.pem
  -out ca_cert.pem

Generate the RDP server's private key:
  

$ openssl genrsa 

  -out srv_priv_key.pem

Generate the RDP server's certificate request based on its private key:
  

$ openssl req 

  -new
  -key srv_priv_key.pem

  -out srv_cert_req.pem

Generate the CA signed RDP server certificate from its certificate request:
  

$ openssl x509

  -in srv_cert_req.pem
  -out srv_cert.pem

  -CA ca_cert.pem
  -CAkey ca_priv_key.pem

  -req
  -days 365
  -setserial 01 

Finally, due replacing path for, in my case, /home/.../.VirtualBox/encryption:

$ VBoxManage modifyvm vm1 
  --vrdeproperty="Security/Method=TLS"
  --vrdeproperty "Security/CACertificate=path/ca_cert.pem"
  --vrdeproperty "Security/ServerCertificate=path/srv_cert.pem"
  --vrdeproperty "Security/ServerPrivateKey=path/srv_priv_key.pem"

 
An UNIX RDP client such as rdesktop can connect as follows:
  
$ rdesktop -u user -p - rdp_srv:rdp_srv_port
 

Interestingly, VBoxManage showvminfo vm1 --details insists on displaying encryption as RDP4.  If I didn't make any mistake, this is rather odd as  Security/Method has been forced to TLS and rdesktop -4 fails.

OpenSSL References:

• http://www.openssl.org/docs/apps/req.html
• http://www.openssl.org/docs/apps/genrsa.html
• http://www.openssl.org/docs/apps/x509.html
• http://www.openssl.org/docs/apps/x509v3_config.html
• http://www.openssl.org/docs/apps/ca.html

  

VirtualBox VM RDP access

For remote access to a VM's user interface, RDP is an interesting additional option.
Of course, other usual client network access methods apply as well.
 
$ read RDP_USER
$ RDP_PASS=$(stty -echo; read INPUT; stty echo; echo $INPUT)
$ RDP_HASH=$(VBoxManage internalcommands
  passwordhash "$RDP_PASS" | cut -d' ' -f3)

$ VBoxManage setproperty vrdeauthlibrary "VBoxAuthSimple"

$ VBoxManage modifyvm vm1
  --vrdeauthtype external
  --vrdeport 33891
  --vrde on

$ VBoxManage setextradata vm1
  "VBoxAuthSimple/users/$RDP_USER" "$RDP_HASH"
     

VirtualBox VM basic storage

At first, storage controllers must be setup on the VM:
  
$ VBoxManage storagectl vm1
  --name SATA
  --add sata
  --controller IntelAHCI
  
Then, virtual hard-disks are created and attached to a storage controller on the VM:
 
$ cd $HOME
 
$ VBoxManage createhd 
  --filename .VirtualBox/VM/vm1/hd1
  --size 20480 
 
$ VBoxManage storageattach vm1 
  --storagectl SATA
  --port 0 
  --device 0 
  --type hdd 
  --medium .VirtualBox/VM/vm1/hd1.vdi

$ VBoxManage showhdinfo .VirtualBox/VM/vm1/hd1.vdi
  

VirtualBox VM basic networking

Multiple network setups are possible, depending on particular needs.
As an example, bridging is an interesting option for unrestricted connectivity.
It's important to let Solaris 11 itself provide the network virtualization.
Be aware that, on a guest, a vnic created on top of the assigned vnic won't work.
Be sure to reference an unused vnic in --bridgeadaptern option. 

$ dladm show-vnic 
LINK      OVER    SPEED  MACADDRESS        MACADDRTYPE  VID
vnic0        ?     1000  8:0:27:af:4c:5b   fixed          0
vnic1        ?     1000  2:8:20:e6:a6:24   random         0
 
$ VBoxManage list bridgedifs | less 
 
Don't assign to the VM any vnic already in use.
In this example, vnic0 can be assigned.

$ dlstat
 
$ ipadm show-if
IFNAME     CLASS    STATE    ACTIVE OVER
lo0        loopback ok       yes    --
net0       ip       ok       yes    --

$ VBoxManage modifyvm vm1  
  --nic1 bridged   
  --bridgeadapter1 vnic0
 
If under Solaris 10, to avoid problems, then you'd better have additional available physical interfaces to dedicate to VirtualBox VMs themselves. Furthermore, you may have to consider some NAT scheme, preferably disabling VirtualBox DHCP server.

As mentioned, don't create additional vnics on the VM. If the VM will have non-global zones, don't use the anet resource which implicitly create a vnic, but a simple net resource. On both cases, the end result would be a vnic on top of another vnic which doesn't work at all.
             

VirtualBox VM basic setup

Setting up a new VM is relatively easy.
Beforehand, it must be known the ID of the guest OS type.
Except for storage configuration, most resources are initially preset.
Certain values intialy preset are usually insufficient, for example, memory.

List possible guest OS type IDs, if not already known:
  
$ VBoxManage list ostypes | grep ID | sed 's/ID: *\(.*\)/\1/'

Define the virtual machine, preferably specifying a "custom" base folder.
By default, the "custom" base folder will be created inside ~/.VirtualBox.
  
$ VBoxManage createvm 
  --name "vm1" 
  --ostype "Solaris11_64" 
  --basefolder "VM" 
  --register 
 
Adjust memory, cpu, and boot order:
  
$ VBoxManage modifyvm vm1
  --memory 1536 --cpus 2
  --boot1 net
  --boot2 dvd
  --boot3 disk
  --boot4 none
 
Adjust BIOS options for faster reboots:
  
$ VBoxManage modifyvm vm1
  --bioslogodisplaytime 0
  --bioslogofadein off
  --bioslogofadeout off
    
NOTE

This note, as of 2013/2014, appears to be relevant.
It seem to be related to NIS & logins - troubleshooting #1.
It's about the Using ZFS Storage Pools in VirtualBox.
Bottom line do the following extra step:

$ VBoxManage setextradata vm1
"VBoxInternal/Devices/ahci/0/LUN0/Config/IgnoreFlush" 0 

     
NOTE

Later on, after performing the basic installation of the client-OS, if available, install the corresponding Guest Additions.

    

VirtualBox on non-global zones

Of course it's best to run VirtualBox from a non-global zone.
I have successfully tested this on VirtualBox version 4.2.18.
But, unfortunately, since 4.3 series, at least up to 4.3.4, it's broken.
Create and configure non-global zones adding the following:

# zonecfg -z vboxzone              
     "add device;                  
      set match=/dev/vboxdrv;      
      end"

# zonecfg -z vboxzone              
     "add device;                  
      set match=/dev/vboxusbmon;   
      end"

Next boot the non-global zone and install the VirtualBox package stream (.pkg) and its extension pack (.vbox-extpack) as performed in VirtualBox on the global zone.

Alternatively, for greater simplicity, instead of repeating the installation in the non-global zone, it may be well enough just to lofs map /opt/VirtualBox from the global zone and adjust PATH to include it.

# zonecfg -z vboxzone              
     "add fs;                  
      set type=lofs;   
      set dir=/opt/VirtualBox;   
      set special=/opt/VirtualBox;   
      end;   
      end"

NOTE
 

• Naturally, /dev/vboxusbmon is not applicable to Solaris 10.
• If on Solaris 11, consider making the zone immutable. 

   

VirtualBox on the global zone

It's relatively simple to have Solaris as a VirtualBox host.
Below is what I've found to be the cleaner steps for installation.
   
If upgrading, remove previous versions.
Each existing VM configuration won't be lost.
 
# pkgrm SUNWvbox
   
Download and extract it to an empty folder:
  
# tar xzf VirtualBox-<version>.tar.gz
 
The subsequent installation action (pkgadd) may create the directory /opt/VirtualBox. If you pre-created it as the mountpoint of a ZFS dataset dedicated to VirtualBox files, then make sure you adjust the ownership attributes (chgrp) to root:bin, otherwise the installation will fail.