Monday, October 28, 2013

NIS & Automounter

The NFS Automounter technology leverages NFS and is essential to most Unix networking environment. It seems that most, if not all Linux implementations are more or less broken, but fortunately with the Solaris Autofs land we have nothing but blue sky.

Even better is the integration of NIS services with the Automounter, adding flexibility and central administration with the elimination of client-by-client local files administration. That's exceptional for medium-to-large networks.

The integration is extremely simple and straightforward, just requiring to enable the corresponding (automount) Name Service Switch (NSS) database and crafting appropriate built-in (auto.master and auto.home) NIS maps alongside additional auto_* custom NIS maps for extended flexibility.

It's critical to avoid as much as possible any hard-coded dependencies in the /etc/auto_master of each client in order to avoid expensive post-installation management in case locations must be substituted. It's enough to consider the importance of this on a network of hundreds or thousands of computers. As a consequence, it's important to carefully choose stable names for the topmost mounting points for the data at rest, such as /data, /public, /download, /business.corp, and so on.

Once all the requirements have been fulfilled, typical usages are:

  1. The in-line (a.k.a. the no absolute paths prefixes) syntax.
     
    The referenced NIS map contents are considered in relation to: i) a mountpoint prefix (to the left) of the map's name in the current Automounter configuration file or else ii) a nested map (multiple indirection) into another referenced NIS map
     
  2. The inclusion (+map_name) syntax.
     
    The map_name NIS map contents are inserted at the current Automounter configuration file (a.k.a. local map) referenced line position. This has the main disadvantage of not integrating with variable substitution as previously described. So except for special cases I won't consider this option due to its inherent limitation.
     
    Again, a couple of examples should help clarify matters.
    I'll start by the simplest and progress to the more elaborated examples.

    Example 1: 

    Requirement: Standards based centrally management of on-demand mounting of assorted NFS shares by several hosts under /data.

    Solution: Configure the automounter of each host to use a specific NIS map describing what's to be mounted as needed. Assuming a host called app-server-1, this is as simple as modifying its /etc/auto_master as follows:

    app-server-1:~# cat /etc/auto_master 
    ...
    /data        auto_data_${HOST}       -nobrowse


    Note:
    I can make use of variable substitution as long as it happens on the value (not the key) field of NIS maps themselves. In addition, it can't be used in included NIS maps. This last limitation is precisely what prevents me from taking advantage of the auto.master NIS map (which is referenced in the standard /etc/auto_master as simply +auto_master at the position where I inserted the ellipses).
    app-server-1:~# ypcat -k auto_data_app-server-1
    area-4 file-server-1:/export/data/tank-4
    area-3 file-server-1:/export/data/tank-3
    area-2 file-server-1:/export/data/tank-2
    area-1 file-server-1:/export/data/tank-1


    In this example, areas area-1 through area-4 are to be on-demand mapped to app-server-1 under /data. Later, if any modifications are needed, it's just a matter of updating the corresponding (auto_data_app-server-1) NIS map. Given the normal automounter timeouts (for indirect NIS maps — direct ones require automounter restart), nothing else is necessary to get the updates reflected to app-server-1.

    I'd like to show the corresponding NIS server side changes in order to support the new custom map. But at the same time, I'd like to show the include directive, which is a more manageable approach then simply ever adding stuff to /var/yp/Makefile to the point it can turn cumbersome or virtually unmanageable. As such, in /var/yp/Makefile just add the 2 following changes for each new custom NIS map:

    # cat /var/yp/Makefile
    ... 

    all: passwd ageing group netid \
            project netgroup aliases publickey \
            hosts ipnodes ethers networks netmasks \

            rpc services protocols \
            auto.master auto.home \
            auth.attr exec.attr prof.attr user.attr \

            auto_data_app-server-1

    ...

    include target.auto_data_app-server-1

    ...

    The file now being included is as follows:

    # cat /var/yp/target.auto_data_app-server-1
    auto_data_app-server-1.time: $(DIR)/auto_data_app-server-1
        -@if [ -f $(DIR)/auto_
    data_app-server-1 ]; then\
            # Join any continuation lines.;\
            (\
            while read L;\
            do;\
                echo "$$L";\
            done\
            < $(DIR)/auto_data_
    app-server-1\
            $(CHKPIPE)) |\
            #;\
            # Normalize the input to makedbm.;\
            # Stripe-out comments,;\
            # then delete blank lines.;\
            (sed -e "s/[`echo '\t'` ]*#.*$$//" -e "/^ *$$/d"\
            $(CHKPIPE)) |\
            #;\
            # Build the updated map.;\
            $(MAKEDBM) - $(YPDBDIR)/$(DOM)/auto_
    data_app-server-1;\
            #;\
            # Finishing house-keeping.;\
            touch auto_
    data_app-server-1.time;\
            echo "updated auto_
    data_app-server-1";\
            #;\
            # Push the updated map to slaves?;\
            if [ ! $(NOPUSH) ]; then\
                $(YPPUSH) auto_
    data_app-server-1;\
                echo "pushed auto_
    data_app-server-1";\
            fi\
        else\
            echo "couldn't find $(DIR)/auto_
    data_app-server-1";\
        fi

    auto_
    data_app-server-1: auto_data_app-server-1.time


    But repeating all this stuff for every time is boring, inefficient and error-prone. A more intelligent approach is required. I can say I know the basics of the make utility but until know I haven't faced the need to go beyond. Well that's one of those moments. After a couple of days thinking over the problem I've been finally inspired to the following solution:

    Instead of the aforementioned auto_data_app-server-1 include file, use the following alternative include file, call it target.template-1 if you will, which adds more 2 maps — not shown on the original all make target — just to illustrate the better scalability and manageability of the new approach:

    BUILD_TEMPLATE_1 = -@if [ -f $(DIR)/$(CUSTOM_MAP) ];\
    then\
        (\
            while read L;\
            do\
                echo "$$L";\
            done\
            < $(DIR)/$(CUSTOM_MAP)\
            $(CHKPIPE)\
        )\
        |\
        (\
            sed -e "s/[`echo '\t'` ]*\#.*$$//" -e "/^ *$$/d"\
            $(CHKPIPE)\
        )\
        |\
        $(MAKEDBM) - $(YPDBDIR)/$(DOM)/$(CUSTOM_MAP);\
        :;\
        touch $(CUSTOM_MAP).time;\
        echo "updated $(CUSTOM_MAP)";\
        :;\
        if [ ! $(NOPUSH) ];\
        then\
            $(YPPUSH) $(CUSTOM_MAP);\
            echo "pushed $(CUSTOM_MAP)";\
        fi\
    else\
        echo "couldn't find $(DIR)\$(CUSTOM_MAP)";\
    fi

    #---------------------------------------------------------

    auto_data_app-server-1.time := CUSTOM_MAP = $(@:%.time=%)
    auto_data_app-server-1.time : $(DIR)/auto_data_app-server-1
            $(BUILD_TEMPLATE_1)
     
       
    auto_data_app-server-1: auto_data_app-server-1.time

    #---------------------------------------------------------
    auto_data_group-1.time := CUSTOM_MAP = $(@:%.time=%)
    auto_data_group-1.time : $(DIR)/auto_data_group-1
            $(BUILD_TEMPLATE_1)


    auto_data_group-1: auto_data_group-1.time

    #---------------------------------------------------------
    auto_data_group-2.time := CUSTOM_MAP = $(@:%.time=%)
    auto_data_group-2.time : $(DIR)/auto_data_group-2
            $(BUILD_TEMPLATE_1)
     


    auto_data_group-2: auto_data_group-2.time

    As seen, I make use of the following new knowledge:
    • macros ( = );
    • conditional macros ( := );
    • pattern replacement macro reference ( : %=% ).
      
    I also had to adjust what's passed to BUILD_TEMPLATE_1 taking into consideration that now I putting it all into a make macro.

    Example 2:

    Consider a slightly more complex variation of example 1, where multiple indirection is used to factor out commonalities. This greatly improves manageability and flexibility. Note that the reference in /etc/auto_master doesn't change, but the contents of the auto_data_${HOST} map, change as follows:

    app-server-1:~# ypcat -k auto_data_app-server-1
    group-2 -fstype=autofs,nobrowse        auto_data_&
    group-1 -fstype=autofs,nobrowse        auto_data_&


    And the 2 NIS maps being referenced with the aid of key substitution are as follows:

    app-server-1:~# ypcat -k auto_data_group-1
    area-2 -fstype=autofs,nobrowse file-server-1:/export/data/tank2
    area-1 -fstype=autofs,nobrowse file-server-1:/export/data/tank1


    app-server-1:~# ypcat -k auto_data_group-2
    area-4 -fstype=autofs,nobrowse file-server-1:/export/data/tank4
    area-3 -fstype=autofs,nobrowse file-server-1:/export/data/tank3


    In this example, areas area-1 and area-2 were grouped into group-1 and areas area-3 and area-4 were grouped into group-2. In order to prevent spurious directories under /data (note that /etc/auto_master is the same from example 1), I had to explicitly declare group-1 and group-2 in auto_data_app-server-1 where I used key substitution to somewhat enhance manageability.

    Example 3:

    This example demonstrates the extremely flexible automounter's executable maps. Executable maps are local files that are run whenever indirect mounting requests take place. Perhaps its greatest advantage is precisely the freedom to dynamically correlate several pieces of information in building the mounting string. For instance, in can query and correlate multiple NIS maps. Whatever it performs it must be efficient. If the executable map happens to be a shell script, an obvious requirement is setting the execution bit. In addition it may be advisable to set the file mode to 0550. Furthermore, the expected behavior is to accept a lookup key as its $1 parameter and, case successful, to return the contents of a respective automounter map entry to standard output, otherwise nothing.

    The local file that is run can be of any type as long it exhibits the expected aforementioned behavior. So, for instance, let me present a source code boilerplate for a binary implementation based on previous NIS programming examples:

    #include <cstdlib>
    #include
    <iostream>

    #include <rpcsvc/ypclnt.h>

    #include "pointer.hxx"

    inline void std_c_free( void * p ) throw()
    {
        ::free( p );
    }

    int main( int argc, char * argv[] )
    {

        // No input, treat as invalid key.
        // No output to the standard output.
        if ( argc != 2 )
            ::exit( YPERR_KEY );


        // The lookup logic could be rather involved.
        // It could:
        //     - Query systems
        //     - Query databases
        //     - Trigger special actions
        //
        // For better startup performance,
        // a companion custom multi-threaded SMF service
        // to which most tasks were to be delegated could help.
        //
        // Here, as an example, it's just a simple NIS lookup.

        // Consider hard-coded values.
        // Trade-offs: software engineering x performance.

        int code;
        char * domain;

        if ( ( code = ::yp_get_default_domain( & domain ) ) == 0 )
        {  

            char map[] = "auto_query_001";
            char * key = argv[ 1 ];

            char * value;
            int length;

            if
            (
                (
                    code = ::yp_match
                    (
                        domain, 

                        map,
                        key, 

                        ::strlen( key ),
                        & value, 

                        & length
                    )
                )
                == 0
            )
            {

                // Lookup success.
                // Send (including the \n) to the standard output.
                pointer< char, std_c_free > p_value( value );
                std::cout << p_value;
            }
            else

                // Lookup error.
                // No output to the standard output.
                ::exit( code );
        }
        else

            // Lookup error.
            // No output to the standard output.
            ::exit( code );

        return EXIT_SUCCESS;
    }


    The simplest compilation line for the above code could be:

    # CC -m64 -lsocket -lnsl \
         -o auto_x_query_001 auto_x_query_001.cxx

    In this particular example, the equivalent shell script could be as simple as:

    # ll /etc/auto_x_query_001
    -r-xr-x---  1 root bin ... /etc/auto_x_query_001

    # cat /etc/auto_x_query_001
    #!/bin/sh -
    /usr/bin/ypmatch "$1" auto_query_001
     
    In general, the main advantage in adopting an indirect executable map is the possibility of adding a dynamic touch or override, thereby changing the otherwise deterministic value statically associated to the lookup key.

    For instance, considering the following sample on overriding:

    # ypcat -k auto_query_001
    area-1 file-server-1:/export/data/tank-1
    area-2 file-server-1:/export/data/tank-2

    To enforce the nosuid mount option I could have:

    # cat /etc/auto_x_query_001
    #!/bin/sh -
    /usr/bin/ypmatch "$1" auto_query_001 | 2> /dev/null \
    /usr/bin/sed -e 's/.*/-nosuid &/'
      
    The typical output of the previous script is:

    # /etc/auto_x_query_001 area-1
    -nosuid file-server-1:/export/data/tank-1

    By referring to auto_x_query_001 instead of auto_query_001 in /etc/auto_master, all the respective mountings will get the nosuid mount flag enforced.

    Example 4:

    This sort of multi-example attempts to show the extra flexibility provided by combining variable substitution (built-in and custom) and hierarchical mounts.

    For instance, let's define a custom variable for depicting the RD (for Research & Development) class a certain workstation.

    Note: Save yourself from trouble by not using shell characters.

    workstation-1:~# sharectl set -p environment=CLASS=RD autofs
    workstation-1:~# sharectl get -p environment autofs
    environment=CLASS=RD

    Let's say we want to give each client equipment a specific view of /business.corp depending on to which class it belongs (as above described, by means of a custom variable called CLASS). Their /etc/auto_master could include something similar to:

    # cat /etc/auto_master
    ...
    /business.corp   auto_business.corp   -nobrowse
    ...

    Let's say that the auto_business.corp and the auto_projects_RD custom NIS maps are somewhat as follows:

    # ypcat -k auto_business.corp
    ...
    projects    -fstype=autofs,nobrowse   auto_projects_${CLASS}
    ...
    standards   file-server-1:/export/standards
    templates   file-server-2:/export/templates
    ...

    # ypcat -k auto_projects_RD
    project_C   file-server-4:/export/projects/rd/&
    project_B   file-server-5:/export/projects/rd/&
    project_A   file-server-6:/export/projects/rd/&

     
    The following logical structure will result:

    # tree -d /business.corp
    /business.corp
    ├── projects
    │   ├── project_A
    │   ├── project_B
    │   └── project_C
    ├── standards
    └── templates


    Solaris has the following built-in variables:
    (examples are shown for Solaris 11.1 on a typical Intel microprocessor)
    ARCH      arch                 i86pc
    KARCH     arch -k / uname -m   i86pc
    CPU       uname -p            i386
    HOST     
    uname -n            ...
    OSNAME    
    uname -s             SunOS
    OSREL     
    uname -r            5.11
    OSVERS    
    uname -v             11.1
    PLATFORM  
    uname -i            i86pc
    NATISA     isainfo -n          amd64