ISM run sample 3.0

This is a sample run for ISM of the shared memory code sample 3.0:

$ getent user_attr prime    ...::::defaultpriv=basic,proc_lock_memory;...

$ swap -lh; swap -sh
swapfile             dev    swaplo   blocks     free
/dev/swap             -         4K     4.0G     3.9G
total: 496M allocated + 170M reserved = 668M used, 18G available

$ ipcs -mA
IPC status ...
T ... KEY ... NATTCH      SEGSZ CPID ... ISMATTCH     PROJECT
Shared Memory:

$ zonestat 1 1
Collecting data for first interval...
Interval: 1, Duration: 0:00:01
SUMMARY   Cpus/Online: 24/24   PhysMem: 23.9G  VirtMem: 27.9G
          ---CPU----  --PhysMem-- --VirtMem-- --PhysNet--
     ZONE  USED %PART  USED %USED  USED %USED PBYTE %PUSE
  [total]  0.26 1.09% 7719M 31.4% 10.0G 35.8%  540K 0.00%
 [system]  0.26 1.09% 7632M 31.0%  9.9G 35.4%     -     -
      ...  0.00 0.00% 86.9M 0.35%  117M 0.40%  1280 0.00%

Getting shared memory.
Press <ENTER> to continue...
Success!

$ swap -lh; swap -sh
swapfile             dev    swaplo   blocks     free
/dev/swap             -         4K     4.0G     3.9G
total: 496M allocated + 1.2G reserved = 1.7G used, 17G available

$ ipcs -mA
IPC status ...
T ... KEY ... NATTCH      SEGSZ CPID ... ISMATTCH     PROJECT
Shared Memory:
m ... 0x1 ...      0 1073741824 3253 ...        0 group.staff

$ zonestat 1 1
Collecting data for first interval...
Interval: 1, Duration: 0:00:01
SUMMARY  Cpus/Online: 24/24   PhysMem: 23.9G  VirtMem: 27.9G
          ---CPU----  --PhysMem-- --VirtMem-- --PhysNet--
     ZONE  USED %PART  USED %USED  USED %USED PBYTE %PUSE
  [total]  0.25 1.05% 7720M 31.4% 11.0G 39.4%   317 0.00%
 [system]  0.25 1.05% 7633M 31.0%  9.9G 35.4%     -     -
      ...  0.00 0.00% 86.9M 0.35% 1144M 3.99%   317 0.00%

$ prstat -c -p 3253 1 1
Please wait...
   PID USERNAME  SIZE   RSS ... 
  3253 ...      3832K 1720K 
...

Attaching to shared memory.
Press <ENTER> to continue...
Success!

$ swap -lh; swap -sh
swapfile             dev    swaplo   blocks     free
/dev/swap             -         4K     4.0G     3.9G
total: 1.5G allocated + 182M reserved = 1.7G used, 17G available

$ ipcs -mA
IPC status ...
T ... KEY ... NATTCH      SEGSZ CPID ... ISMATTCH     PROJECT
Shared Memory:
m ... 0x1 ...      1 1073741824 3253 ...        1 group.staff

$ zonestat 1 1
Collecting data for first interval...
Interval: 1, Duration: 0:00:01
SUMMARY  Cpus/Online: 24/24   PhysMem: 23.9G  VirtMem: 27.9G
          ---CPU----  --PhysMem-- --VirtMem-- --PhysNet--
     ZONE  USED %PART  USED %USED  USED %USED PBYTE %PUSE
  [total]  0.25 1.07% 8741M 35.5% 11.0G 39.4%  1347 0.00%
 [system]  0.25 1.06% 7631M 31.0%  9.9G 35.4%     -     -
      ...  0.00 0.00% 1109M 4.51% 1148M 4.00%  1347 0.00%

$ prstat -c -p 3253 1 1
Please wait...
   PID USERNAME  SIZE   RSS ...     
  3253 ...      1028M 1026M ...
...

$ pmap -x 3253 | head
3253:    ./shm_v04
 Address  Kbytes     RSS ...  Locked Mode  Mapped File
08050000       8       8 ...       - r-x-- shm_v04
08061000       4       4 ...       - rwx-- shm_v04
08062000     128      44 ...       - rwx--   [ heap ]
80000000 1048576 1048576 ... 1048576 rwxsR   [ ism shmid=... ]
...

Locking shared memory.
Press <ENTER> to continue...
Success!

    not relevant for ISM

Using shared memory.
Press <ENTER> to continue...
Success!

    not relevant for ISM

Unlocking shared memory.
Press <ENTER> to continue...
Success!

    not relevant for ISM

Detaching shared memory.
Press <ENTER> to continue...
Success!

$ swap -lh; swap -sh
swapfile             dev    swaplo   blocks     free
/dev/swap             -         4K     4.0G     3.9G
total: 1.5G allocated + 182M reserved = 1.7G used, 17G available

$ ipcs -mA
IPC status ...
T ... KEY ... NATTCH      SEGSZ CPID ... ISMATTCH     PROJECT
Shared Memory:
m ... 0x1 ...      0 1073741824 3253 ...        0 group.staff

$ zonestat 1 1
Collecting data for first interval...
Interval: 1, Duration: 0:00:01
SUMMARY  Cpus/Online: 24/24   PhysMem: 23.9G  VirtMem: 27.9G
          ---CPU----  --PhysMem-- --VirtMem-- --PhysNet--
     ZONE  USED %PART  USED %USED  USED %USED PBYTE %PUSE
  [total]  0.24 1.03% 8742M 35.5% 11.0G 39.4%   228 0.00%
 [system]  0.24 1.03% 8656M 35.2%  9.9G 35.4%     -     -
      ...  0.00 0.00% 85.9M 0.34% 1143M 3.99%   228 0.00%

$ prstat -c -p 3253 1 1
Please wait...
   PID USERNAME  SIZE   RSS ...     
  3253 ...      3832K 1720K ...
...

$ pmap -x 3253 | head
3253:    ./shm_v04
 Address  Kbytes     RSS ... Locked Mode   Mapped File
08050000       8       8 ...      - r-x--  shm_v04
08061000       4       4 ...      - rwx--  shm_v04
08062000     128      44 ...      - rwx--    [ heap ]
FE380000      24      12 ...      - rwx--    [ anon ]
...

Removing shared memory.
Press <ENTER> to continue...
Success!

$ swap -lh; swap -sh
swapfile             dev    swaplo   blocks     free
/dev/swap             -         4K     4.0G     3.9G
total: 496M allocated + 174M reserved = 668M used, 18G available

$ ipcs -mA
IPC status ...
T ... KEY ... NATTCH      SEGSZ CPID ... ISMATTCH     PROJECT
Shared Memory:

$ zonestat 1 1
Collecting data for first interval...
Interval: 1, Duration: 0:00:01
SUMMARY  Cpus/Online: 24/24   PhysMem: 23.9G  VirtMem: 27.9G
          ---CPU----  --PhysMem-- --VirtMem-- --PhysNet--
     ZONE  USED %PART  USED %USED  USED %USED PBYTE %PUSE
  [total]  0.30 1.25% 7718M 31.4% 10.0G 35.8%  9852 0.00%
 [system]  0.30 1.25% 7632M 31.0%  9.9G 35.4%     -     -
      ...  0.00 0.00% 85.9M 0.34%  120M 0.41%   228 0.00%

$ prstat -c -p 3253 1 1
Please wait...
   PID USERNAME  SIZE   RSS ...     
  3253 ...      3832K 1720K ...
...

Exiting program.
Press <ENTER> to continue...

When Oracle Database fellows argues that "::shmget() requires swap reservation" they induce sysadmins to confusion or error.

As seen above (on the 2nd "output group") what happens is simply virtual memory reservation within the kernel. Note there's no other swap maneuver or switch between disk and physical memory and so on. The process that does the call doesn't even exhibit different figures.

With ISM, physical memory actually gets allocated only when the reserved shared segment is mapped in the calling process via ::shmat(). This is clearly seen in the 3rd "output group" above.

We also see that ::shmdt() doesn't really release any allocated memory as seen in the 4th "output group". The shared memory segment is unmapped from the process but continues to exist within the kernel.

It's only when ::shmctl() is called that memory is actually freed (fifth "output group").
  

Shared memory code sample 3.0

Evolution never stops, even on sample code :-)
As such, I'd like to insert minor changes on the previous coding.
Check out sample outputs for both ISM and DISM experiments.

The sample learning code below is using the SHM_SHARE_MMU flag to ::shmat() which is for ISM, but exactly the same code can be used replacing the flag for SHM_PAGEABLE (DISM) or 0 (System V Shared Memory).

Don't forget to compile it for 64-bits (-m64) if you want segments larger than 4 GB. This version has a crude pause() routine to avoid relying on a debugger. 
 
001: /*
002:  * File:   main.cpp
003:  * Author: AZ
004:  *
005:  * Created on June 4, 2013, 7:31 AM
006:  *
007:  */
008:
009: #include <iostream>
010:
011: #include <sys/ipc.h>
012: #include <sys/shm.h>
013:
014: #include <sys/mman.h>
015: #include <errno.h>
016:
017: void pause( char const * message )
018: {
019:     std::cout << message << std::endl 
                   << "Press <ENTER> to continue...";

020:     std::cin.get( );
021: }
022:
023: void success( )
024: {
025:     std::cout << "Success!" << std::endl 
                   << std::endl;
026: }
027:
028: void failure( )
029: {
030:     std::cout << "Failure!" << std::endl 
                   << std::endl;
031: }
032:
033: int main( )
034: {
035:     pause( "Getting shared memory." );
036:
037:     std::size_t const size = 
         1UL * 1024UL * 1024UL * 1024UL;
038:
039:     int id = 
         ::shmget( 1, size, IPC_CREAT | IPC_EXCL | 
         SHM_R | SHM_W | ( SHM_R >> 3 ) | ( SHM_W >> 3 ) );
040:
041:     if ( id == -1 )
042:     {
043:         failure( );
044:     }
045:     else
046:     {
047:         success( );
048:
049:         pause( "Attaching to shared memory." );
050:
051:         void * p = ::shmat( id, 0, SHM_SHARE_MMU );
052:
053:         if ( reinterpret_cast < intptr_t > ( p ) == -1 )
054:         {
055:             failure( );
056:         }
057:         else
058:         {
059:             success( );
060:
061:             pause( "Locking shared memory." );
062:
063:             if ( ::mlock( p, size ) == 0 )
064:             {
065:                 success( );
066:
067:                 pause( "Using shared memory." );
068:
069:                 ::memset( p, '*', size );
070:
071:                 success( );
072:
073:                 pause( "Unlocking shared memory." );
074:
075:                 switch ( ::munlock( p, size ) )
076:                 {
077:                     case 0:
078:                         success( );
079:                         break;
080:
081:                     case -1:
082:                         failure( );
083:                         break;
084:                 }
085:
086:             }
087:             else
088:             {
089:                 ::perror( 0 );
090:
091:                 failure( );
092:             }
093:
094:             pause( "Detaching shared memory." );
095:
096:             switch ( ::shmdt( p ) )
097:             {
098:                 case 0:
099:                     success( );
100:                     break;
101:
102:                 case -1:
103:                     failure( );
104:                     break;
105:             }
106:         }
107:
108:         pause( "Removing shared memory." );
109:
110:         switch ( ::shmctl( id, IPC_RMID, 0 ) )
111:         {
112:             case 0:
113:                 success( );
114:                 break;
115:
116:             case -1:
117:                 failure( );
118:                 break;
119:         }
120:     }
121:
122:     pause( "Exiting program." );
123:   
124:     return 0;
125: }
  

Shared memory run sample 2.0

This post builds on the premises of shared memory run sample 1.0.
The following is the output of shared memory code sample 2.0.
 

IMPORTANT

For running this new binary the proc_lock_memory privilege is required.
If running as root (not recommended) this might not be a problem.

# usermod -K defaultpriv="basic,proc_lock_memory" <user>

# getent user_attr prime
<user>::::defaultpriv=basic,proc_lock_memory;...

I'll directly present the results of shared memory code sample 2.0.
The analysis and comments are found at the end of this post.
 

# swap -lh; swap -sh

swapfile             dev    swaplo   blocks     free

/dev/swap             -         4K     4.0G     3.9G
total: 436M allocated + 201M reserved = 636M used, 18G available

// size == 1 GiB

int id = ::shmget( 1, size, IPC_CREAT | IPC_EXCL | 0660 );

# swap -lh; swap -sh

swapfile             dev    swaplo   blocks     free

/dev/swap             -         4K     4.0G     3.9G
total: 436M allocated + 1.2G reserved = 1.6G used, 17G available
 
// Success!

void * p = ::shmat( id, 0, SHM_PAGEABLE );

 
# swap -lh; swap -sh

swapfile             dev    swaplo   blocks     free

/dev/swap             -         4K     4.0G     3.9G
total: 436M allocated + 1.2G reserved = 1.6G used, 17G available
 
# ipcs -mA 

IPC status ...
T ID KEY ... NATTCH       SEGSZ  CPID ... ISMATTCH     PROJECT
Shared Memory:
m 36 0x1 ...      1  1073741824 16670 ...        1 group.staff

 

# pmap -x 16670

16670:    /.../shm_v03/dist/Debug/...
 Address  Kbytes     RSS  Locked Mode   Mapped File
08050000       8       8       - r-x--  shm_v03
08061000       4       4       - rwx--  shm_v03
08062000     128      44       - rwx--    [ heap ]
80000000 1048576       -       - rwxs-    [ dism shmid=0x24 ]

...
 

// Touch the shared memory pages...

::memset( p, '*', size );

# swap -lh; swap -sh

swapfile             dev    swaplo   blocks     free

/dev/swap             -         4K     4.0G     3.9G
total: 1.4G allocated + 204M reserved = 1.6G used, 17G available

# pmap -x 16670

16670:    /.../shm_v03/dist/Debug/...
 Address  Kbytes     RSS  Locked Mode   Mapped File
08050000       8       8       - r-x--  shm_v03
08061000       4       4       - rwx--  shm_v03
08062000     128      44       - rwx--    [ heap ]
80000000 1048576 1048576       - rwxs-    [ dism shmid=0x24 ]

...

// Lock the shared memory pages...

if ( ::mlock( p, size ) == 0 )

# swap -lh; swap -sh

swapfile             dev    swaplo   blocks     free

/dev/swap             -         4K     4.0G     3.9G
total: 1.4G allocated + 204M reserved = 1.6G used, 16G available

# pmap -x 16670

16670:    /.../shm_v03/dist/Debug/...
 Address  Kbytes     RSS  Locked Mode   Mapped File
08050000       8       8       - r-x--  shm_v03
08061000       4       4       - rwx--  shm_v03
08062000     128      44       - rwx--    [ heap ]
80000000 1048576 1048576 1048576 rwxs-    [ dism shmid=0x24 ]

...

// Unock the shared memory pages...

switch ( ::munlock( p, size ) )

# swap -lh; swap -sh

swapfile             dev    swaplo   blocks     free

/dev/swap             -         4K     4.0G     3.9G
total: 1.4G allocated + 205M reserved = 1.6G used, 17G available

# pmap -x 16670

16670:    /.../shm_v03/dist/Debug/...
 Address  Kbytes     RSS  Locked Mode   Mapped File
08050000       8       8       - r-x--  shm_v03
08061000       4       4       - rwx--  shm_v03
08062000     128      44       - rwx--    [ heap ]
80000000 1048576 1048576       - rwxs-    [ dism shmid=0x24 ]

...

switch ( ::shmdt( p ) )

# pmap -x 16670

16670:    /.../shm_v03/dist/Debug/...
 Address  Kbytes     RSS  Locked Mode   Mapped File
08050000       8       8       - r-x--  shm_v03
08061000       4       4       - rwx--  shm_v03
08062000     128      44       - rwx--    [ heap ]

...
 
# swap -lh; swap -sh

swapfile             dev    swaplo   blocks     free

/dev/swap             -         4K     4.0G     3.9G
total: 1.4G allocated + 205M reserved = 1.6G used, 17G available
 

switch ( ::shmctl( id, IPC_RMID, 0 ) )

 
# swap -lh; swap -sh

swapfile             dev    swaplo   blocks     free

/dev/swap             -         4K     4.0G     3.9G
total: 452M allocated + 197M reserved = 648M used, 18G available

Handling just 1 GiB is more than enough for presenting my argument.
Incidentally I forgot to compile for 64-bit as seen on pmap addresses.
This won't be a problem as I said will handle just 1 GiB for now.
 
The differences in this sample starts after attaching to the DISM.
swap -sh reserved and allocated changes when DISM pages are:

• touched (written to) as in this code sample.
• locked even before been written to.

  
But pay careful attention to what's uncovered by this sample:
   

1. ::shmget() causes the amount of available virtual memory to reduce in face of a corresponding memory reservation, as shown by swap -sh. So far, the referred to deduction is due to mere accounting of virtual memory: originally there were 17 GiB (as set up by the sysadmin) and after the application reserves 1 GiB, the remaining capacity is simply the difference, 16 GiB. I have to emphasize what's already told on MOS documents: "... no actual memory pages are allocated ...".
     
2. ::shmat() tells the kernel the strategy to apply to the shared memory reservation, in this case it's DISM. In practice the kernel just bookmark things regarding this other processes interested on mapping this same shared memory segment.
      
3. As reserved virtual memory pages are actually allocated by writing to or locking portions of the shared memory segment, an accounting is reflected by swap -sh but with absolute no change of used or available virtual memory figures. After all, after a reservation, no additional reservations are happening. What's going on is that the kernel is just honoring (allocating) what it has previously promised (reserved).
        
4. Then something crucial happens: the shared memory segment is locked with ::mlock(). This means that the affected memory pages (all of them in this sample) must be taken out of the virtual memory machinery and stuck (non-pageable) into an available region physical memory.  As virtual memory is backed by both physical memory and disk (see swap analysis), this causes and immediate deduction of the available virtual memory as less physical memory remains available for backing the virtual memory. Furthermore, the locked pages in physical memory can't be paged out to disk, so the amount of available pageable memory (the total amount of virtual memory) must be reduced.

  
Hence, the crucial point when dealing with DISM is to provide enough:
 

• Disk-based space for the whole DISM requirements.
    
  That's so, unless MAP_NORESERVE is used as per pmap(1).
  A no disk space reservation is indicated by pmap -x mode flag S.
  But what's the point? Not reserving means to risk not honoring.
  The application must deal with unpredictable lack of resources.
         
• Available physical memory for the whole locking requirements.

  
The conclusions are:
  

• DISM consumes disk-based virtual memory space.
  It consumes twice more resources, assuming maximum locking.
  What are the trade-offs of shrinking memory for changing boards?
    
      
• ISM doesn't consume any disk space at all.
  Best choice if plenty of physical memory is available.
  

 

Shared memory code sample 2.0

If the previous few lines of code sample didn't help, then let's try a little more.
Of course I build on all previous premises so I won't repeat that part.
This time, playing with just 1 GiB or any other quantity is enough.
We now use ISM and manual locking to wrap up the exposition.
Some real output from the following code is here.

/*
 * File:   main.cpp
 * Author: AZ
 *
 * Created on May 27, 2013, 8:46 AM
 */

#include <iostream>
#include <cerrno>

#include <sys/ipc.h>
#include <sys/shm.h>
#include <sys/mman.h>
 
int main()
{
    //
    // Play with shared memory!
    //

    std::size_t const size = 1UL * 1024UL * 1024UL * 1024UL;

    std::cout << "Getting shared memory: ";

    int id = ::shmget( 1, size, IPC_CREAT | IPC_EXCL | 0660 );

    if ( id == -1 )
    {
        std::cout << "Failure!" << std::endl;
    }
    else
    {
        std::cout << "Success!" << std::endl;

        std::cout << "Attaching to shared memory: ";

        //
        // For a "standard" shared memory segment,
        // don't pass any flags, that is, pass 0.
        //
        // SHM_RND [Optional]
        // Asks to round down the address to a page boundary.
        // Not necessary for ISM or DISM.
        //
        // SHM_SHARE_MMU
        // Asks for a ISM segment locked in physical memory.
        //
        // SHM_PAGEABLE
        // Asks for a DISM segment pageable (virtual memory).
        //

        void * p = ::shmat( id, 0, SHM_PAGEABLE );

        if ( reinterpret_cast < intptr_t > ( p ) == -1 )
        {
            std::cout << "Failure!" << std::endl;
        }
        else
        {
            std::cout << "Success!" << std::endl;

            std::cout << "Using shared memory..." << std::endl;

            //
            // Touch the whole shared segment.
            // Fill it with '*'.
            //
            // This will force the amount reserved
            // to be actually allocated (at least with DISM).
            //

            ::memset( p, '*', size );

            std::cout << "Locking shared memory: ";

            if ( ::mlock( p, size ) == 0 )
            {
                std::cout << "Success!" << std::endl;

                std::cout << "Unlocking shared memory: ";

                switch ( ::munlock( p, size ) )
                {
                    case 0:
                        std::cout << "Success!" << std::endl;
                        break;

                    case -1:
                        std::cout << "Failure!" << std::endl;
                        break;
 
                    default:
                        std::cout << "?" << std::endl;
                        break;
                }
            }
            else
            {
                std::cout << "Failure! >>> ";

                switch ( errno )
                {
                    case EINVAL:
                        std::cout << "EINVAL" << std::endl;
                        break;

                    case ENOMEM:
                        std::cout << "ENOMEM" << std::endl;
                        break;

                    case ENOSYS:
                        std::cout << "ENOSYS" << std::endl;
                        break;

                    case EPERM:
                        std::cout << "EPERM" << std::endl;
                        break;

                    case EAGAIN:
                        std::cout << "EAGAIN" << std::endl;
                        break;
  
                    default:
                        std::cout << "?" << std::endl;
                        break;
                }
            }

            std::cout << "Detaching shared memory: ";

            switch ( ::shmdt( p ) )
            {
                case 0:
                    std::cout << "Success!" << std::endl;
                    break;

                case -1:
                    std::cout << "Failure!" << std::endl;
                    break;
  
                default:
                    std::cout << "?" << std::endl;
                    break;
            }
        }

        //
        // End of fun!
        //

        std::cout << "Removing shared memory: ";

        switch ( ::shmctl( id, IPC_RMID, 0 ) )
        {
            case 0:
                std::cout << "Success!" << std::endl;
                break;

            case -1:
                std::cout << "Failure!" << std::endl;
                break;
   
            default:
                std::cout << "?" << std::endl;
                break;
        }
    }

    return 0;
}
   
Also look at a small variation of the above code.