A necessary consequence of multiples paths is that performance can be enhanced as well as streaming data can flow through multiple paths in parallel. But, by the connection-oriented nature of TCP/IP this performance enhancement frequently narrows down to outbound traffic, that is, traffic flowing out of the IPMP system to remote clients.
IPMP has been available since older Solaris releases and I would say it has become progressively better and simpler to configure since its inception. According to another post of mine called Legacy & Future I'll be focusing on Solaris 11 as my discussions cut-point. Things started to get significantly simpler and better since Solaris 11 Express and really top-notch onward with Solaris 11.x.
I could only talk about Solaris 11.x but I'll address Solaris 11 Express because it's still a nice back-end system capable of running on x86 (32-bits) platform. As everybody knows, beyond mid-range and high-end big-iron SPARC systems, Solaris 11.x only runs on x86-64 (64-bits) platforms. Oracle has completely dropped support to Solaris 11 Express as it was marketed as a short-term transition from Solaris 10 to Solaris 11. The last update was SRU-13 or SRU-14 (focused on some Engineered-Systems). But the truth is that Solaris 11 Express is an awesome system to the near-zero or very small IT budgets business models based on legacy x86 hardware, and it still rivals much more recent Linux and BSD alternatives because it embeds very advanced key technologies such as ZFS and BEs (boot-environments), beyond, of course, other high-end technologies such as IPMP. So if you still have this piece software consider using it, specially because it's quite possible to independently update some of its crucial components and applications based on open software.
Back to IPMP the central idea is to group a given number of network interfaces and associate it to a pool of new (data) addresses by which the group will be publicly accessible. The group is materialized as a new network interface in the system which operation and availability is provided by the collaboration of the underlying group members. In general the number of members network interfaces should greater than the number of data addresses and some member network interfaces can each be set as a hot stand-by to the group. When stand-by network interfaces are present the IPMP group is said to be of an active-standby type, otherwise it's an active-active type. Unless you really have lots of network interfaces to spare an active-standby IPMP type would waste a precious network resource, hence otherwise prefer an active-active IPMP group.
NOTE
Sometimes there's some confusion, argumentation and comparison to another technology know as Link-Aggregation but things are quite different beats although both contribute to resilience and performance. One advantage of IPMP is that it operates on layer-3 thus possessing no layer-2 special driver and hardware requirements as Link-Aggregation dos. Both are not mutually exclusive and can even be combined, but perhaps each one is better suited to an specific scenario or requirement. For instance, a back-to-back connection between two servers is better implemented via Link-Aggregation while out-bound traffic load spread may be better deployed via IPMP.Let's go straight to a minimal practical example, first on Solaris 11 Express and then on Solaris 11.3. Don't be fooled by the simplicity because the solution is still quite a lot powerful and significant to a many applications infrastructure models which is not easily attained, if at all, by more modern competitors systems. By the way, I will assume that some techniques and technologies (NCP, routes and name resolution) described for manual wired connections will be implicitly used as needed.
EXAMPLE:
Setting up an active-active IPMP group from interfaces net2 and net3 which link names have been respectively renamed from an e1000g2 and an e1000g3 originally available on the system.
# dladm show-phys
LINK MEDIA STATE SPEED DUPLEX DEVICE
...
net2 Ethernet unknown 0 half e1000g2
net3 Ethernet unknown 0 half e1000g3
...
The newly generated network interface representing the new IPMP group will stop working only if both net2 and net3 fail simultaneously, but as long as both underlying interfaces remains operational up to 2 Gbps of overall outbound bandwidth will be available for multiple TCP connections, letting clear that still no more that 1 Gbps inbound bandwidth per single TCP connection.
NOTE
It may still not be crystal clear, but having N underlying interfaces of 1 Gbps on a given IPMP group will generally provide an overall outbound bandwidth of N Gbps for that IPMP group. The inbound bandwidth is a different story; if M < N data addresses are configured for an IPMP group of 1 Gbps underlying interfaces, then the overall inbound performance will still be limited to 1 Gbps per TCP session inbound traffic even though it may be possible to simultaneously have M such sessions.
On Solaris 11 Express:
Under Solaris 11 Express the update of the IPMP management interface is still transitioning and its crucial parts still must be managed via the old ifconfig command. Do not attempt to manage the underlying interfaces net2 and net3 via the new ipadm command for anything related to IPMP.
Configure the IPMP group and its data-address.
The group will subsequently receive the underlying member interfaces:
# ifconfig ipmp0 ipmp 192.168.1.230/24 upConfigure the underlying interfaces:
# ifconfig net2 plumb group ipmp0 up
# ifconfig net3 plumb group ipmp0 up
NOTE
Verify the configuration:In the case of an active-standby configuration it would be necessary to choose one of the underlying interfaces as a stand-by interface by simply inserting the standby keyword just before the up keyword.
# ifconfig -a |ggrep -A2 'ipmp0:'Make the configuration persistent across reboots:
ipmp0: flags=8001000842<UP,BROADCAST,RUNNING,MULTICAST,IPv4,IPMP> ...
inet 192.168.1.230 netmask ffffff00 broadcast 192.168.1.255
groupname ipmp0
# ifconfig -a |ggrep -A3 -E 'net(2|3):'
net2: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> ...
inet 0.0.0.0 netmask ff000000
groupname ipmp0
ether 8:0:27:fe:f6:44
net3: flags=1000843<UP,BROADCAST,RUNNING,MULTICAST,IPv4> ...
inet 0.0.0.0 netmask ff000000
groupname ipmp0
ether 8:0:27:c3:94:2
# ipadm show-if |ggrep -E 'ipmp0|net2|net3'
ipmp0 ok bm--I-----4- ---
net2 ok bm--------4- ---
net3 ok bm--------4- ---
# ipadm show-addr 'ipmp0/'
ADDROBJ TYPE STATE ADDR
ipmp0/? static ok 192.168.1.230/24
# ipmpstat -g
GROUP GROUPNAME STATE FDT INTERFACES
ipmp0 ipmp0 ok -- net3 net2
# ipmpstat -i
INTERFACE ACTIVE GROUP FLAGS LINK PROBE STATE
net3 yes ipmp0 ------- up disabled ok
net2 yes ipmp0 --mb--- up disabled ok
(the order of parameters below is important in obtaining the exact results)
# cat /etc/hostname.ipmp0To eventually disable and clean up the IPMP group:
ipmp 192.168.1.230/24 up
# cat /etc/hostname.net2
group ipmp0 up
# cat /etc/hostname.net3
group ipmp0 up
# rm /etc/hostname.net3
# rm /etc/hostname.net2
# rm /etc/hostname.ipmp0
# ifconfig ipmp0 down
# ifconfig net2 down
# ifconfig net3 down
# ifconfig net2 group ""
# ifconfig net3 group ""
# ifconfig net2 unplumb
# ifconfig net3 unplumb
# ifconfig ipmp0 unplumb
On Solaris 11.3:
Under Solaris 11.3 things are somewhat easier. The IPMP management has been fully integrated into the ipadm command and the case for persistency across reboots is on by default requring no additional actions.
Configure the underlying interfaces:
# ipadm create-ip net2Configure the IPMP group:
# ipadm create-ip net3
# ipadm create-ipmp -i net2,net3 ipmp0Set the data address for the IPMP group:
# ipadm create-addr -T static -a 192.168.1.230/24 ipmp0To eventually disable and clean up the IPMP group:
ipmp0/v4
NOTE
Unfortunately, perhaps due to some subtle bug in the GA release of Solaris 11.3, it seems safer to only set the IPMP group data-address after the underlying interfaces have been added to the IPMP group.
(the order is important, again, due to some subtle bug)
# ipadm delete-addr ipmp0/v4In the rare case where a standby underlying interface is still desired, for instance net4, it suffices to perform the following commands:
# ipadm remove-ipmp -i net2,net3 ipmp0
# ipadm delete-ipmp ipmp0
# ipadm delete-ip net2
# ipadm delete-ip net3
# ipadm create-ip net4
# ipadm set-ifprop -p standby=on -m ip net4
# ipadm add-ipmp -i net4 ipmp0
# ipadm show-if
IFNAME CLASS STATE ACTIVE OVER
lo0 loopback ok yes --
ipmp0 ipmp ok yes net2 net3 net4
net2 ip ok yes --
net3 ip ok yes --
net4 ip ok no --
# ipmpstat -g
GROUP GROUPNAME STATE FDT INTERFACES
ipmp0 ipmp0 ok -- net3 net2 (net4)
That's all very powerful and not that difficult to set up.
For sure one more cool technology available in Solaris!
