Learnings on Solaris™: May 2013

Monday, May 27, 2013

Shared memory run sample 2.0

This post builds on the premises of shared memory run sample 1.0.
The following is the output of shared memory code sample 2.0.

IMPORTANT

For running this new binary the proc_lock_memory privilege is required.
If running as root (not recommended) this might not be a problem.

# usermod -K defaultpriv="basic,proc_lock_memory" <user>

# getent user_attr prime

<user>::::defaultpriv=basic,proc_lock_memory;...

I'll directly present the results of shared memory code sample 2.0.
The analysis and comments are found at the end of this post.

# swap -lh; swap -sh

swapfile             dev    swaplo   blocks     free
/dev/swap             -         4K     4.0G     3.9G
total: 436M allocated + 201M reserved = 636M used, 18G available

// size == 1 GiB 

int id = ::shmget( 1, size, IPC_CREAT | IPC_EXCL | 0660 );

# swap -lh; swap -sh

swapfile dev swaplo blocks free

/dev/swap - 4K 4.0G 3.9G
total: 436M allocated + 1.2G reserved = 1.6G used, 17G available

// Success!

void * p = ::shmat( id, 0, SHM_PAGEABLE );

# swap -lh; swap -sh

swapfile dev swaplo blocks free

/dev/swap - 4K 4.0G 3.9G
total: 436M allocated + 1.2G reserved = 1.6G used, 17G available

# ipcs -mA

IPC status ...
T ID KEY ... NATTCH SEGSZ CPID ... ISMATTCH PROJECT
Shared Memory:
m 36 0x1 ... 1 1073741824 16670 ... 1 group.staff

# pmap -x 16670

16670:    /.../shm_v03/dist/Debug/...

 Address  Kbytes     RSS  Locked Mode   Mapped File

08050000       8       8       - r-x--  shm_v03

08061000       4       4       - rwx--  shm_v03

08062000     128      44       - rwx--    [ heap ]

80000000 1048576       -       - rwxs-    [ dism shmid=0x24 ]

...

// Touch the shared memory pages...

::memset( p, '*', size );

# swap -lh; swap -sh

swapfile dev swaplo blocks free

/dev/swap - 4K 4.0G 3.9G
total: 1.4G allocated + 204M reserved = 1.6G used, 17G available

# pmap -x 16670

16670:    /.../shm_v03/dist/Debug/...

 Address  Kbytes     RSS  Locked Mode   Mapped File

08050000       8       8       - r-x--  shm_v03

08061000       4       4       - rwx--  shm_v03

08062000     128      44       - rwx--    [ heap ]

80000000 1048576 1048576       - rwxs-    [ dism shmid=0x24 ]

...

// Lock the shared memory pages...

if ( ::mlock( p, size ) == 0 )

# swap -lh; swap -sh

swapfile dev swaplo blocks free

/dev/swap - 4K 4.0G 3.9G
total: 1.4G allocated + 204M reserved = 1.6G used, 16G available

# pmap -x 16670

16670:    /.../shm_v03/dist/Debug/...

 Address  Kbytes     RSS  Locked Mode   Mapped File

08050000       8       8       - r-x--  shm_v03

08061000       4       4       - rwx--  shm_v03

08062000     128      44       - rwx--    [ heap ]

80000000 1048576 1048576 1048576 rwxs-    [ dism shmid=0x24 ]

...

// Unock the shared memory pages...

switch ( ::munlock( p, size ) )

# swap -lh; swap -sh

swapfile dev swaplo blocks free

/dev/swap - 4K 4.0G 3.9G
total: 1.4G allocated + 205M reserved = 1.6G used, 17G available

# pmap -x 16670

16670:    /.../shm_v03/dist/Debug/...

 Address  Kbytes     RSS  Locked Mode   Mapped File

08050000       8       8       - r-x--  shm_v03

08061000       4       4       - rwx--  shm_v03

08062000     128      44       - rwx--    [ heap ]

80000000 1048576 1048576       - rwxs-    [ dism shmid=0x24 ]

...

switch ( ::shmdt( p ) )

# pmap -x 16670

16670:    /.../shm_v03/dist/Debug/...

 Address  Kbytes     RSS  Locked Mode   Mapped File

08050000       8       8       - r-x--  shm_v03

08061000       4       4       - rwx--  shm_v03

08062000     128      44       - rwx--    [ heap ]

...

# swap -lh; swap -sh

swapfile dev swaplo blocks free

/dev/swap - 4K 4.0G 3.9G
total: 1.4G allocated + 205M reserved = 1.6G used, 17G available

switch ( ::shmctl( id, IPC_RMID, 0 ) )

# swap -lh; swap -sh

swapfile dev swaplo blocks free

/dev/swap - 4K 4.0G 3.9G
total: 452M allocated + 197M reserved = 648M used, 18G available

Handling just 1 GiB is more than enough for presenting my argument.
Incidentally I forgot to compile for 64-bit as seen on pmap addresses.
This won't be a problem as I said will handle just 1 GiB for now.

The differences in this sample starts after attaching to the DISM.
swap -sh reserved and allocated changes when DISM pages are:

touched (written to) as in this code sample.
locked even before been written to.

But pay careful attention to what's uncovered by this sample:

::shmget() causes the amount of available virtual memory to reduce in face of a corresponding memory reservation, as shown by swap -sh. So far, the referred to deduction is due to mere accounting of virtual memory: originally there were 17 GiB (as set up by the sysadmin) and after the application reserves 1 GiB, the remaining capacity is simply the difference, 16 GiB. I have to emphasize what's already told on MOS documents: "... no actual memory pages are allocated ...".
::shmat() tells the kernel the strategy to apply to the shared memory reservation, in this case it's DISM. In practice the kernel just bookmark things regarding this other processes interested on mapping this same shared memory segment.
As reserved virtual memory pages are actually allocated by writing to or locking portions of the shared memory segment, an accounting is reflected by swap -sh but with absolute no change of used or available virtual memory figures. After all, after a reservation, no additional reservations are happening. What's going on is that the kernel is just honoring (allocating) what it has previously promised (reserved).
Then something crucial happens: the shared memory segment is locked with ::mlock(). This means that the affected memory pages (all of them in this sample) must be taken out of the virtual memory machinery and stuck (non-pageable) into an available region physical memory. As virtual memory is backed by both physical memory and disk (see swap analysis), this causes and immediate deduction of the available virtual memory as less physical memory remains available for backing the virtual memory. Furthermore, the locked pages in physical memory can't be paged out to disk, so the amount of available pageable memory (the total amount of virtual memory) must be reduced.

Hence, the crucial point when dealing with DISM is to provide enough:

Disk-based space for the whole DISM requirements.

That's so, unless MAP_NORESERVE is used as per pmap(1).
A no disk space reservation is indicated by pmap -x mode flag S.
But what's the point? Not reserving means to risk not honoring.
The application must deal with unpredictable lack of resources.
Available physical memory for the whole locking requirements.

The conclusions are:

DISM consumes disk-based virtual memory space.
It consumes twice more resources, assuming maximum locking.
What are the trade-offs of shrinking memory for changing boards?
ISM doesn't consume any disk space at all.
Best choice if plenty of physical memory is available.

Shared memory code sample 2.0

If the previous few lines of code sample didn't help, then let's try a little more.
Of course I build on all previous premises so I won't repeat that part.
This time, playing with just 1 GiB or any other quantity is enough.
We now use ISM and manual locking to wrap up the exposition.
Some real output from the following code is here.

/*
* File:   main.cpp
* Author: AZ
*
* Created on May 27, 2013, 8:46 AM
*/

#include <iostream>
#include <cerrno>

#include <sys/ipc.h>
#include <sys/shm.h>
#include <sys/mman.h>

int main()
{
    //
    // Play with shared memory!
    //

    std::size_t const size = 1UL * 1024UL * 1024UL * 1024UL;

    std::cout << "Getting shared memory: ";

    int id = ::shmget( 1, size, IPC_CREAT | IPC_EXCL | 0660 );

    if ( id == -1 )
    {
        std::cout << "Failure!" << std::endl;
    }
    else
    {
        std::cout << "Success!" << std::endl;

        std::cout << "Attaching to shared memory: ";

        //
        // For a "standard" shared memory segment,
        // don't pass any flags, that is, pass 0.
        //
        // SHM_RND [Optional]
        // Asks to round down the address to a page boundary.
        // Not necessary for ISM or DISM.
        //
        // SHM_SHARE_MMU
        // Asks for a ISM segment locked in physical memory.
        //
        // SHM_PAGEABLE
        // Asks for a DISM segment pageable (virtual memory).
        //

        void * p = ::shmat( id, 0, SHM_PAGEABLE );

        if ( reinterpret_cast < intptr_t > ( p ) == -1 )
        {
            std::cout << "Failure!" << std::endl;
        }
        else
        {
            std::cout << "Success!" << std::endl;

            std::cout << "Using shared memory..." << std::endl;

            //
            // Touch the whole shared segment.
            // Fill it with '*'.
            //
            // This will force the amount reserved
            // to be actually allocated (at least with DISM).
            //

            ::memset( p, '*', size );

            std::cout << "Locking shared memory: ";

            if ( ::mlock( p, size ) == 0 )
            {
                std::cout << "Success!" << std::endl;

                std::cout << "Unlocking shared memory: ";

                switch ( ::munlock( p, size ) )
                {
                    case 0:
                        std::cout << "Success!" << std::endl;
                        break;

                    case -1:
                        std::cout << "Failure!" << std::endl;
                        break;

                    default:
                        std::cout << "?" << std::endl;
                        break;
                }
            }
            else
            {
                std::cout << "Failure! >>> ";

                switch ( errno )
                {
                    case EINVAL:
                        std::cout << "EINVAL" << std::endl;
                        break;

                    case ENOMEM:
                        std::cout << "ENOMEM" << std::endl;
                        break;

                    case ENOSYS:
                        std::cout << "ENOSYS" << std::endl;
                        break;

                    case EPERM:
                        std::cout << "EPERM" << std::endl;
                        break;

                    case EAGAIN:
                        std::cout << "EAGAIN" << std::endl;
                        break;

                    default:
                        std::cout << "?" << std::endl;
                        break;
                }
            }

            std::cout << "Detaching shared memory: ";

            switch ( ::shmdt( p ) )
            {
                case 0:
                    std::cout << "Success!" << std::endl;
                    break;

                case -1:
                    std::cout << "Failure!" << std::endl;
                    break;

                default:
                    std::cout << "?" << std::endl;
                    break;
            }
        }

        //
        // End of fun!
        //

        std::cout << "Removing shared memory: ";

        switch ( ::shmctl( id, IPC_RMID, 0 ) )
        {
            case 0:
                std::cout << "Success!" << std::endl;
                break;

            case -1:
                std::cout << "Failure!" << std::endl;
                break;

            default:
                std::cout << "?" << std::endl;
                break;
        }
    }

    return 0;
}

Also look at a small variation of the above code.

Sunday, May 26, 2013

GRUB menu

There's a few things we can do to customize the GRUB menu.
In general I tend to keep the defaults as much as possible.
One little thing I always like to touch is the entries' texts.

By default GRUB menu entries get the same name as a corresponding BE.
When I update a system I use specify the new version as the BE name such as:

# pkg update --be-name 11.1-7.5.0
# beadm create -e 11.1-7.5.0 11.1-7.5.0-bk

As a side effect, this generates a corresponding GRUB menu entry such as:

# bootadm list-menu
...
0 Oracle Solaris 11.1
1 11.1-7.5.0
2 11.1-7.5.0-bk

This is convenient and expected, but it isn't very cute.
For example, I'd like to prepend the "Oracle Solaris" string.

Of course we can manually edit the appropriate file as in previous releases.
In fact, that used to be the only way to customize anything about GRUB.
Fortunately (or not) we now count on bootadm sucommands:

# bootadm
...
bootadm set-menu ... key=value
bootadm list-menu ... -i
bootadm add-entry ...
bootadm remove-entry ... -i
bootadm change-entry ... -i {key=value}+ [set-default]
bootadm generate-menu [-f] ...

In particular, for the purpose of this post if suffices to me just the following:

# bootadm change-entry -i 1 title="Oracle Solaris 11.1-7.5.0"
# bootadm change-entry -i 2 title="Oracle Solaris 11.1-7.5.0-bk"
# bootadm list-menu
...
0 Oracle Solaris 11.1
1 Oracle Solaris 11.1-7.5.0
2 Oracle Solaris 11.1-7.5.0-bk

That's really cool, safe and more comfortable than manual editing.
Check bootadm(1M) for more detail.

For example, it's now easy to create boot entries for:

single user mode/state (kernel argument -s).
backups on another ZFS pool

Friday, May 24, 2013

Shared memory run sample 1.0

Let's follow a shared memory code sample 1.0 on a debugger and watch.
This should clear up any remaining doubts and misunderstandings.
Yet for an additional example, check shared memory code sample 2.0.

The particular environment configuration where the code is run isn't relevant.
I believe it suffices to say that it presents:

Solaris 11.1 SRU 7.5 running on HP ProLiant BL460c G7.
24 GiB of physical memory and 18 GiB available virtual memory.
This is due to some physical memory being reserved to VirtualBox.
VirtualBox has no other relation to this post, it's there by accident.
Sure there are VirtualBox guests running and consuming memory.
Disk-based swap is defined at 4 GiB and is not being used at first.
Certainly, the available virtual memory includes this figure.
The binary will be run under group.staff project resource controls.
The resource control default was at around 5.99 GiB, ¼ of physical memory.
Hence, it was adjusted to 20 GiB in order to cause no interferences here.

So here is the essential starting point settings and figures:

# tail /etc/system

...

* 24GB - 1.5GB -> 22GB - 5% -> 20GB due to VirtualBox

set zfs:zfs_arc_max=0x400000000

# swap -lh; swap -sh

swapfile                   dev    swaplo   blocks     free

/dev/zvol/dsk/rpool/swap 214,2        4K     4.0G     4.0G

total: 436M allocated + 211M reserved = 648M used, 18G available

# projmod 

  -K "project.max-shm-memory=(privileged,20G,deny)" 

  group.staff 

# projects -l group.staff

group.staff

 projid : 10

 comment: ""

 users  : (none)

 groups : (none)

 attribs: project.max-shm-memory=(privileged,21474836480,deny)

The 1^st sample run will require 17 GiB of ISM.
As seen this would require enough available lockable physical memory.
This particular run will fail, but note how the swap figures are affected.
Virtual memory isn't directly involved, but it's availability is recomputed.
This is because virtual memory is partially backed by free physical memory.
As physical memory is reserved, less of it is free for backing virtual memory.
This was also discussed in the swap analysis post.

The essential excerpts are as follows:

# swap -lh; swap -sh

swapfile dev swaplo blocks free

/dev/swap - 4K 4.0G 4.0G
total: 448M allocated + 212M reserved = 660M used, 18G available

// size == 17 GiB 

int id = ::shmget( 1, size, IPC_CREAT | IPC_EXCL | 0660 );

# swap -lh; swap -sh

swapfile dev swaplo blocks free

/dev/swap - 4K 4.0G 4.0G
total: 448M allocated + 17G reserved = 18G used, 772M available

// Failure!

void * p = ::shmat( id, 0, SHM_SHARE_MMU );

# swap -lh; swap -sh

swapfile dev swaplo blocks free

/dev/swap - 4K 4.0G 4.0G
total: 448M allocated + 17G reserved = 18G used, 816M available

::shmctl( id, IPC_RMID, 0 );

# swap -lh; swap -sh

swapfile dev swaplo blocks free

/dev/swap - 4K 4.0G 4.0G
total: 448M allocated + 212M reserved = 660M used, 18G available

The 2^nd sample run will also require 17 GiB, this time of DISM kind.
As seen this would entirely rely on virtual memory and directly affect swap.
This particular run will succeed as no physical memory is initially required.

The essential excerpts are as follows:

# swap -lh; swap -sh

swapfile dev swaplo blocks free

/dev/swap - 4K 4.0G 4.0G

total: 448M allocated + 212M reserved = 660M used, 18G available

// size == 17 GiB

int id = ::shmget( 1, size, IPC_CREAT | IPC_EXCL | 0660 );

# swap -lh; swap -sh

swapfile dev swaplo blocks free

/dev/swap - 4K 4.0G 4.0G

total: 448M allocated + 17G reserved = 18G used, 800M available

// Success!

void * p = ::shmat( id, 0, SHM_PAGEABLE ); 

# swap -lh; swap -sh

swapfile dev swaplo blocks free

/dev/swap - 4K 4.0G 4.0G

total: 448M allocated + 17G reserved = 18G used, 792M available

# ipcs -mA

IPC status ...
T ID KEY ... NATTCH SEGSZ CPID ... ISMATTCH PROJECT
Shared Memory:
m 24 0x1 ... 1 18253611008 10385 ... 1 group.staff

# pmap -x 10385

10385: /.../shm_v01/dist/Debug/...

Address Kbytes RSS ... Mapped File

0000000000400000 8 8 shm_v01

0000000000411000 4 4 shm_v01

0000000000412000 160 68 [ heap ]

FFFF80FB40000000 17825792 - [ dism shmid=0x18 ]

...

// Somewhat lengthy run...

::memset( p, '*', size );

# swap -lh; swap -sh

swapfile dev swaplo blocks free

/dev/swap - 4K 4.0G 4.0G

total: 448M allocated + 17G reserved = 18G used, 792M available

# swap -lh; swap -sh

swapfile dev swaplo blocks free

/dev/swap - 4K 4.0G 4.0G

total: 8.6G allocated + 9.0G reserved = 18G used, 792M available

# swap -lh; swap -sh

swapfile dev swaplo blocks free

/dev/swap - 4K 4.0G 4.0G

total: 15G allocated + 2.7G reserved = 18G used, 792M available

# swap -lh; swap -sh

swapfile dev swaplo blocks free

/dev/swap - 4K 4.0G 3.7G

total: 17G allocated + 214M reserved = 18G used, 792M available

# pmap -x 10385

10385: /.../shm_v01/dist/Debug/...

Address Kbytes RSS ... Mapped File

0000000000400000 8 8 shm_v01

0000000000411000 4 4 shm_v01

0000000000412000 160 68 [ heap ]

FFFF80FB40000000 17825792 17510200 [ dism shmid=0x18 ]

...

switch ( ::shmdt( p ) )

# pmap -x 10385

10385: /.../shm_v01/dist/Debug/...

Address Kbytes RSS ... Mapped File

0000000000400000 8 8 shm_v01

0000000000411000 4 4 shm_v01

0000000000412000 160 68 [ heap ]

...

# swap -lh; swap -sh

swapfile dev swaplo blocks free

/dev/swap - 4K 4.0G 3.7G

total: 17G allocated + 214M reserved = 18G used, 792M available

switch ( ::shmctl( id, IPC_RMID, 0 ) )

# swap -lh; swap -sh

swapfile dev swaplo blocks free

/dev/swap - 4K 4.0G 4.0G

total: 460M allocated + 201M reserved = 664M used, 18G available

From the above output we can list a few interesting conclusions, which I believe certainly will help improve knowledge on shared memory:

ISM relies exclusively on the ability to lock pages of physical memory. It doesn't really participate on virtual memory (virtual swap) although its usage affects the total amount of available virtual memory since any unused physical memory is also used to form virtual memory (virtual swap). As such, disk-based swap space (swap -lh) is never impacted by ISM. At least in theory, ISM can offer better performance as it isn't subject to memory paging.
DISM shares the optimizations of ISM but relies on a completely distinct layer, the virtual memory (virtual swap), not on free physical memory. The main advantages of DISM over ISM is its ability to be larger than the lockable available physical memory and be paged out from memory if it's not being used. So it does can make active use of disk-based swap space (swap -lh). If paging becomes frequent, then performance will certainly suffer.
By the last output of pmap -x 10385 before ::shmdt(), from the 17825792 Kbytes reserved (= SEGSZ of 18253611008 listed by ipcs -mA), 17510200 KB (or 17930444800 bytes) could fit in physical memory and the rest was accommodated on disk-based swap (18253611008 - 17930444800 = 323166208 ≅ 0.3 GB ). I suspect that the largest lockable available physical memory by that time was that figure of 17510200. In general, I'm not sure if there's an objective method for finding this out.
Although DISM, by default, doesn't lock any memory as ISM, applications can judiciously perform that. Unfortunately, on previous output samples, I have omitted the Locked column of pmap -x output as they were null. When ISM succeeds, the three columns Kbytes, RSS and Locked list the same figure. For DISM, the last two generally vary or float, with Kbytes ≥ RSS ≥ Locked, and which is precisely the advantage of DISM.

For very specialized applications such as Oracle Database, I'd say that if the server hardware is dedicated to an instance of it, then DISM would be advantageous with respect to the ability to dynamically adjusting its memory structures up to the defined DISM limit, avoiding application restarts.

In terms of resource control, beyond max-shm-memory, there is the max-shm-ids which refers to how many pointers or segments of shared memory can be obtained. What's important to know is that the larger the memory page size, the lesser of them are needed. Thus, SPARC architectures, possessing larger memory pages, tend to consume less of them. In addition, there is the max-locked-memory which affects both ISM and DISM. Remember that DISM may also lock physical memory pages. Of course max-rss, max-swap and max-address-space may also affect limits. I expect to explore some or all of this resource controls later on another post.

Shared memory code sample 1.0

A few lines of code may help understanding shared memory, ISM and DISM.
Compile the following code to 64-bits to get beyond the 4 GiB barrier.
In Solaris Studio 12.3 this simply requires compilation option -m64.

/*
* File:   main.cpp
* Author: AZ
*
* Created on May 21, 2013, 3:06 PM
*/

#include <iostream>

#include <sys/ipc.h>
#include <sys/shm.h>

/*
* This is just a raw demonstration of the shared memory API.
* To avoid leaks use "resource acquisition is initialization".
*/

int main()
{
   //
   // Play with 17 GiB of shared memory!
   //

   std::size_t const size = 17UL * 1024UL * 1024UL * 1024UL;

   std::cout << "Getting shared memory. ";

   int id = ::shmget( 1, size, IPC_CREAT | IPC_EXCL | 0660 );

   if ( id == -1 )
   {
      std::cout << "Failure!" << std::endl;
   }
   else
   {
      std::cout << "Success!" << std::endl;

      std::cout << "Attaching to shared memory. ";

      //
      // For a "standard" shared memory segment,
      // don't pass any flags, that is, pass 0.
      //
      // SHM_RND [Optional]
      // Asks to round down the address to a page boundary.
      // Not necessary for ISM or DISM.
      //
      // SHM_SHARE_MMU
      // Asks for a ISM segment locked in physical memory.
      //
      // SHM_PAGEABLE
      // Asks for a DISM segment pageable (virtual memory).
      //

      void * p = ::shmat( id, 0, SHM_SHARE_MMU );

      if ( reinterpret_cast < intptr_t > ( p ) == -1 )
      {
         std::cout << "Failure!" << std::endl;
      }
      else
      {
         std::cout << "Success!" << std::endl;

         std::cout << "Using shared memory." << std::endl;

         //
         // Touch the whole shared segment.
         // Fill it with '*'.
   //
   // This will force the amount reserved
   // to be actually allocated (at least with DISM).
         //

         ::memset( p, '*', size );

         std::cout << "Detaching shared memory. ";

         switch ( ::shmdt( p ) )
         {
            case 0:
               std::cout << "Success!" << std::endl;
               break;

            case -1:
               std::cout << "Failure!" << std::endl;
               break;

            default:
               std::cout << "?" << std::endl;
               break;
         }
      }

      //
// End of fun!
      //

      std::cout << "Removing shared memory. ";

      switch ( ::shmctl( id, IPC_RMID, 0 ) )
      {
         case 0:
            std::cout << "Success!" << std::endl;
            break;

         case -1:
            std::cout << "Failure!" << std::endl;
            break;

         default:
            std::cout << "?" << std::endl;
            break;
      }
   }

   return 0;
}

To take advantage of the above code excerpt for learning, a debugger is required.
Step over the shared memory API calls and watch the CLI ipcs and swap output.
More precisely, the commands are ipcs -mA and swap -sh and swap -lh.

By the way, why do I chose to play with 17 GiB?
That's because my particular test system had 18 GiB to offer.
For more detail showing some real output check shared memory run sample 1.0.
In time I have also added a DISM version with manual locking.