Add InfiniBand notes

config/network/vyos.md

@@ -81,7 +81,7 @@ An example of a full configuration. Except intuitive stuff I forgot to mention.
     - **TODO** This can be done in the interface ethernet configs instead.
     - See the Linux router notes.
     - Enable GRO (example): `ethtool -K <if> gro on`
-    - Increase RX/TX buffer sizes (example): `ethtool -K <if> gro on`
+    - Increase RX/TX buffer sizes (example): `ethtool -G <if> tx 4096 rx 4096`
     - Enable scatter/gather aka vectored I/O (example): `ethtool -K <if> sg on`
     - Make any ethtool (e.g.) commands permanent by adding them to `/config/scripts/vyos-postconfig-bootup.script`.
 1. Commit and save: `commit` and `save`.

config/server/debian.md

@@ -124,7 +124,7 @@ If you didn't already configure this during the installation. Typically the case
 
 #### Network Manager
 
-Using ifupdown (default, alternative 1):
+Using ifupdown (alternative 1, default):
 
 1. For VLAN support, install `vlan`.
 1. For bonding/LACP support, install `ifenslave`.

config/server/linux-networking.md

@@ -0,0 +1,46 @@
+---
+title: Linux Server Networking
+breadcrumbs:
+- title: Configuration
+- title: Server
+---
+{% include header.md %}
+
+**TODO**:
+
+- Migrate stuff from Debian page.
+- Add link to Linux router page. Maybe combine.
+- Add ethtool notes from VyOS.
+
+## InfiniBand
+
+### Installation
+
+1. Install RDMA: `apt install rdma-core`
+1. Install user-space RDMA stuff: `apt install ibverbs-providers rdmacm-utils infiniband-diags ibverbs-utils`
+1. Install subnet manager (SM): `apt install opensm`
+    - Only one instance should be required on the subnet.
+1. Setup IPoIB:
+    - Just like for Ethernet. Just specify the IB interface as the L2 device.
+    - Use an appropriate MTU like 2044.
+1. Make sure ping and ping-pong is working.
+
+### Usage
+
+- Check L2 status: `ibstat`
+- Check L3 status: `ip a`
+- Show active subnet manager(s): `sminfo`
+
+#### Testing
+
+- Ping:
+    - Server: `ibping -S`
+    - Client: `ibping -G <guid>`
+- Ping-pong:
+    - Server: `ibv_rc_pingpong`
+    - Client: `ibv_rc_pingpong <ip>`
+- Other tools:
+    - qperf
+    - perftest
+
+{% include footer.md %}

index.md

@@ -18,8 +18,9 @@ Random collection of config notes and miscellaneous stuff. _Technically not a wi
 ### Server
 
 - [Debian Server](config/server/debian/)
-- [Linux Server Storage](config/server/linux-storage/)
 - [Linux Server Applications](config/server/linux-applications/)
+- [Linux Server Storage](config/server/linux-storage/)
+- [Linux Server Networking](config/server/linux-networking/)
 - [Proxmox VE](config/server/proxmox-ve/)
 - [Home Assistant](config/server/home-assistant/)
 - [Dell PowerEdge](config/server/dell-poweredge/)

se/frameworks/openmpi.md

@@ -33,6 +33,10 @@ A Message Passing Interface (MPI) implementation for C, Fortran, Java, etc.
 - Number of processes to use: `-n <n>`
 - Allow more processes than physical cores: `--oversubscribe`
 - Allow running as root (discouraged): `--allow-run-as-root`
-- Run on a specific TCP/IP subnet: `-mca btl_tcp_if_include <subnet>`
+- Specify which network type or subnet to use:
+    - Specify/include IP subnet (TCP): `--mca btl_tcp_if_<include|exclude> <subnet|device>,...`
+    - Disable TCP: `--mca btl ^tcp`
+    - Specify BTLs exactly: `--mca btl self,vader,tcp`
+    - *Something* (**TODO**): `--mca pml ob1`
 
 {% include footer.md %}