The following is the output of shared memory code sample 2.0.
IMPORTANT
For running this new binary the proc_lock_memory privilege is required.If running as root (not recommended) this might not be a problem.
# usermod -K defaultpriv="basic,proc_lock_memory" <user>
# getent user_attr prime
<user>::::defaultpriv=basic,proc_lock_memory;...
<user>::::defaultpriv=basic,proc_lock_memory;...
I'll directly present the results of shared memory code sample 2.0.
The analysis and comments are found at the end of this post.
# swap -lh; swap -sh
total: 436M allocated + 201M reserved = 636M used, 18G available
swapfile dev swaplo blocks free
/dev/swap - 4K 4.0G 3.9Gtotal: 436M allocated + 201M reserved = 636M used, 18G available
// size == 1 GiB
int id = ::shmget( 1, size, IPC_CREAT | IPC_EXCL | 0660 );
# swap -lh; swap -sh
swapfile dev swaplo blocks free
/dev/swap - 4K 4.0G 3.9Gtotal: 436M allocated + 1.2G reserved = 1.6G used, 17G available
// Success!
void * p = ::shmat( id, 0, SHM_PAGEABLE );
# swap -lh; swap -sh
swapfile dev swaplo blocks free
/dev/swap - 4K 4.0G 3.9Gtotal: 436M allocated + 1.2G reserved = 1.6G used, 17G available
# ipcs -mA
IPC status ...
T ID KEY ... NATTCH SEGSZ CPID ... ISMATTCH PROJECT
Shared Memory:
m 36 0x1 ... 1 1073741824 16670 ... 1 group.staff
T ID KEY ... NATTCH SEGSZ CPID ... ISMATTCH PROJECT
Shared Memory:
m 36 0x1 ... 1 1073741824 16670 ... 1 group.staff
# pmap -x 16670
16670: /.../shm_v03/dist/Debug/...
Address Kbytes RSS Locked Mode Mapped File
08050000 8 8 - r-x-- shm_v03
08061000 4 4 - rwx-- shm_v03
08062000 128 44 - rwx-- [ heap ]
80000000 1048576 - - rwxs- [ dism shmid=0x24 ]
...Address Kbytes RSS Locked Mode Mapped File
08050000 8 8 - r-x-- shm_v03
08061000 4 4 - rwx-- shm_v03
08062000 128 44 - rwx-- [ heap ]
80000000 1048576 - - rwxs- [ dism shmid=0x24 ]
// Touch the shared memory pages...
::memset( p, '*', size );
# swap -lh; swap -sh
swapfile dev swaplo blocks free
/dev/swap - 4K 4.0G 3.9Gtotal: 1.4G allocated + 204M reserved = 1.6G used, 17G available
# pmap -x 16670
16670: /.../shm_v03/dist/Debug/...
Address Kbytes RSS Locked Mode Mapped File
08050000 8 8 - r-x-- shm_v03
08061000 4 4 - rwx-- shm_v03
08062000 128 44 - rwx-- [ heap ]
80000000 1048576 1048576 - rwxs- [ dism shmid=0x24 ]
...Address Kbytes RSS Locked Mode Mapped File
08050000 8 8 - r-x-- shm_v03
08061000 4 4 - rwx-- shm_v03
08062000 128 44 - rwx-- [ heap ]
80000000 1048576 1048576 - rwxs- [ dism shmid=0x24 ]
// Lock the shared memory pages...
if ( ::mlock( p, size ) == 0 )
# swap -lh; swap -sh
swapfile dev swaplo blocks free
/dev/swap - 4K 4.0G 3.9Gtotal: 1.4G allocated + 204M reserved = 1.6G used, 16G available
# pmap -x 16670
16670: /.../shm_v03/dist/Debug/...
Address Kbytes RSS Locked Mode Mapped File
08050000 8 8 - r-x-- shm_v03
08061000 4 4 - rwx-- shm_v03
08062000 128 44 - rwx-- [ heap ]
80000000 1048576 1048576 1048576 rwxs- [ dism shmid=0x24 ]
...Address Kbytes RSS Locked Mode Mapped File
08050000 8 8 - r-x-- shm_v03
08061000 4 4 - rwx-- shm_v03
08062000 128 44 - rwx-- [ heap ]
80000000 1048576 1048576 1048576 rwxs- [ dism shmid=0x24 ]
// Unock the shared memory pages...
switch ( ::munlock( p, size ) )
# swap -lh; swap -sh
swapfile dev swaplo blocks free
/dev/swap - 4K 4.0G 3.9Gtotal: 1.4G allocated + 205M reserved = 1.6G used, 17G available
# pmap -x 16670
16670: /.../shm_v03/dist/Debug/...
Address Kbytes RSS Locked Mode Mapped File
08050000 8 8 - r-x-- shm_v03
08061000 4 4 - rwx-- shm_v03
08062000 128 44 - rwx-- [ heap ]
80000000 1048576 1048576 - rwxs- [ dism shmid=0x24 ]
...Address Kbytes RSS Locked Mode Mapped File
08050000 8 8 - r-x-- shm_v03
08061000 4 4 - rwx-- shm_v03
08062000 128 44 - rwx-- [ heap ]
80000000 1048576 1048576 - rwxs- [ dism shmid=0x24 ]
switch ( ::shmdt( p ) )
# pmap -x 16670
16670: /.../shm_v03/dist/Debug/...
Address Kbytes RSS Locked Mode Mapped File
08050000 8 8 - r-x-- shm_v03
08061000 4 4 - rwx-- shm_v03
08062000 128 44 - rwx-- [ heap ]
...Address Kbytes RSS Locked Mode Mapped File
08050000 8 8 - r-x-- shm_v03
08061000 4 4 - rwx-- shm_v03
08062000 128 44 - rwx-- [ heap ]
# swap -lh; swap -sh
swapfile dev swaplo blocks free
/dev/swap - 4K 4.0G 3.9Gtotal: 1.4G allocated + 205M reserved = 1.6G used, 17G available
switch ( ::shmctl( id, IPC_RMID, 0 ) )
# swap -lh; swap -sh
swapfile dev swaplo blocks free
/dev/swap - 4K 4.0G 3.9Gtotal: 452M allocated + 197M reserved = 648M used, 18G available
Handling just 1 GiB is more than enough for presenting my argument.
Incidentally I forgot to compile for 64-bit as seen on pmap addresses.
This won't be a problem as I said will handle just 1 GiB for now.
The differences in this sample starts after attaching to the DISM.
swap -sh reserved and allocated changes when DISM pages are:
- touched (written to) as in this code sample.
- locked even before been written to.
But pay careful attention to what's uncovered by this sample:
- ::shmget() causes the amount of available virtual memory to reduce in face of a corresponding memory reservation, as shown by swap -sh. So far, the referred to deduction is due to mere accounting of virtual memory: originally there were 17 GiB (as set up by the sysadmin) and after the application reserves 1 GiB, the remaining capacity is simply the difference, 16 GiB. I have to emphasize what's already told on MOS documents: "... no actual memory pages are allocated ...".
- ::shmat() tells the kernel the strategy to apply to the shared memory reservation, in this case it's DISM. In practice the kernel just bookmark things regarding this other processes interested on mapping this same shared memory segment.
- As reserved virtual memory pages are actually allocated by writing to or locking portions of the shared memory segment, an accounting is reflected by swap -sh but with absolute no change of used or available virtual memory figures. After all, after a reservation, no additional reservations are happening. What's going on is that the kernel is just honoring (allocating) what it has previously promised (reserved).
- Then something crucial happens: the shared memory segment is locked with ::mlock(). This means that the affected memory pages (all of them in this sample) must be taken out of the virtual memory machinery and stuck (non-pageable) into an available region physical memory. As virtual memory is backed by both physical memory and disk (see swap analysis), this causes and immediate deduction of the available virtual memory as less physical memory remains available for backing the virtual memory. Furthermore, the locked pages in physical memory can't be paged out to disk, so the amount of available pageable memory (the total amount of virtual memory) must be reduced.
Hence, the crucial point when dealing with DISM is to provide enough:
- Disk-based space for the whole DISM requirements.
That's so, unless MAP_NORESERVE is used as per pmap(1).
A no disk space reservation is indicated by pmap -x mode flag S.
But what's the point? Not reserving means to risk not honoring.
The application must deal with unpredictable lack of resources.
- Available physical memory for the whole locking requirements.
The conclusions are:
- DISM consumes disk-based virtual memory space.
It consumes twice more resources, assuming maximum locking.
What are the trade-offs of shrinking memory for changing boards?
- ISM doesn't consume any disk space at all.
Best choice if plenty of physical memory is available.
