Network and Infiniband

It is highly recommended to increase MTU for eth network, as usage on it will be ssh and data, and that increasing MTU will increase data performances without reducing ssh performances. 9000 would be a good value for ethernet, and 65520 for infiniband IPoIB (prefer using directly connected_mode).

To do so, add the following parameter in incfg-enp0sx files:

MTU=9000

LACP can be used on pets servers to increase bandwidth (with multi-streams only) and reliability (in case one NIC fail, the other will continue to operate). LACP is often used on master nodes and io nodes. You need first to configure your switch to gather links, then on OS side, do the following (to gather eth0 and eth1 into one LACP):

Edit (in /etc/sysconfig/network-scripts/) ifcfg-eth0:

DEVICE=eth0
TYPE=Ethernet
MTU=9000
ONBOOT=yes
BOOTPROTO=none
USERCTL=no
SLAVE=yes
MASTER=bond0

Then edit ifcfg-eth1:

DEVICE=eth1
TYPE=Ethernet
MTU=9000
ONBOOT=yes
BOOTPROTO=none
USERCTL=no
SLAVE=yes
MASTER=bond0

And create ifcfg-bond0 with the following content:

DEVICE=bond0
MTU=9000
ONBOOT=yes
BOOTPROTO=static
USERCTL=no
IPV6INIT=no
IPV6_AUTOCONF=no
IPADDR=10.0.0.1
NETMASK=255.255.0.0
BONDING_OPTS="miimon=100 mode=4 lacp_rate=1 xmit_hash_policy=layer3+4"

Bonding needs a kernel module to be loaded.

/etc/modprobe.d/bonding.conf

alias netdev-bond0 bonding

And after reboot your bond should be up.

More : http://backdrift.org/lacp-configure-network-bonding-linux

VLAN can be used to divide your network, for management or security reasons.
VLAN can also be used on a bonding interface (bond0 most of the time).

Here is an example how to configure it, after having done configuration on switch side.

Native VLAN is 10.0.0.0, in ifcfg-bond0 :

DEVICE=bond0
MTU=9000
ONBOOT=yes
BOOTPROTO=static
USERCTL=no
IPV6INIT=no
IPV6_AUTOCONF=no
IPADDR=10.0.0.1
NETMASK=255.255.0.0
BONDING_OPTS="miimon=100 mode=4 lacp_rate=1 xmit_hash_policy=layer3+4"

VLAN 10 is 10.10.0.0, in ifcfg-bond0.10 :

DEVICE=bond0.10
MTU=9000
ONBOOT=yes
BOOTPROTO=none
IPADDR=10.10.0.1
NETMASK=255.255.0.0
USERCTL=no
IPV6INIT=no
IPV6_AUTOCONF=no
VLAN=yes

VLAN need a kernel module to be loaded.

/etc/modprobe.d/8021q.conf

8021q

And after reboot, your VLANs should be up.

To install infiniband, use manufacturer provided modules, or use the RHEL/CENTOS provided (RDMA). See https://access.redhat.com/documentation/en-US/Red_Hat_Enterprise_Linux/7/html/Networking_Guide/ch-Configure_InfiniBand_and_RDMA_Networks.html

In short, you need to install packages, and reboot system. Diag and utils tools are optional but very useful. We are going here to install official RHEL infiniband.

yum groupinstall "Infiniband Support"
yum install infiniband-diags ibutils
systemctl enable rdma
reboot

After reboot, check with ibstatus if interface is “ACTIVE”. After that, you just need to configure an ip on it like any other interfaces, taking into account the MTU choose.

Then, you need to start a subnet manager on your IB network (if not, all interfaces will be in PULLING). You can start opensmd on main server, here batman. DO NOT start opensm/opensmd on another server. There can be only one (like highlanders) on the IB network. Beware, some advanced IB switch can also integrate a subnet manager, ensure you only use one.

If you want to start the subnet manager on master node (batman), use:

systemctl start opensmd
systemctl enable opensmd

Then I you should do a snapshot of your fabric (see below) to use it regularly to check if something has change in the fabric. This is a very simple and efficient fabric check.

Switches

 ibswitches 

HCAs in the fabric (Host Channel Adapters

 ibhosts 

Fabric topology

 ibnetdiscover 

Route through the fabric (slid is the LID of the source node and dlid is the LID of the destination node in the fabric)

 ibtracert slid dlid 

Link status of a node

 ibportstate lid port 

Counters of a node Note that this command can be used locally without arguments to obtain the informations of local port.

 perfquery lid port 

Data counters of a node

 ibdatacounts lid port 

Low level informations of a node

 smpquery switchinfo lid 

Low level informations of a port

 smpquery portinfo lid port 

Check the entire fabric

 ibdiagnet -v -r 

Check a specific route

 ibdiagpath -v -l slid dlid 

Take a snapshot of the fabric after installation, and check changes of the stack when something occurs

Create snapshot

 osmtest -f c 

Compare with snapshot

 osmtest -f v 

Check which links are experiencing significant errors

 ibdiagnet -c 100 -P all=1 

Check all ports (remove -v for only errors output)

 ibcheckstate -v 

Clearing error counters

 ibclearerrors 

Clearing data counters

 ibclearcounters 

Reset a port

 ibportstate lid port reset 

Setting port speed (value = 1 for low speed, 7 for high)

 ibportstate lid port speed <value> 

More informations, see :

• https://docs.oracle.com/cd/E18476_01/doc.220/e18478/fabric.htm
• https://wiki.archlinux.org/index.php/InfiniBand