Update some network stuff

config/network/linux.md

@@ -50,14 +50,23 @@ Issues may also be related to stupid things like which ports you're using on the
 - Disable protocol hardware offloading as it typically causes more problems than it solves.
 - Make sure network interrupts from a given NIC are distributed across all cores.
     - See `/proc/interrupts`.
-- Enable or disable hardware offloading (needs testing):
-    - Enable/disable large receive offload (LRO) or generic receive offload (LRO): `ethtool -K <if> <lro|gro> on` (temporary)
+- Enable or disable hardware offloading (temporary):
+    - Enable/disable large receive offload (LRO) or generic receive offload (GRO): `ethtool -K <if> <lro|gro> on`
     - Enable/disable TX/RX checksum offload: `ethtool -K <if> tx on rx on`
     - Enable/disable scatter/gather aka vectored I/O: `ethtool -K <if> sg on`
     - (And some others.)
 - Change NIC RX/TX buffer sizes:
     - Show supported and current sizes: `ethtool -g <if>`
     - Set new sizes: `ethtool -G <if> tx 4096 rx 4096`
+- Change MTU:
+    - Typical Ethernet MTUs: 1500 (normal), 9000 (jumbo frames).
+    - Typical InfiniBand MTUs: 2044 (2048), 4092 (4096).
+    - Set: `ip link set <if> mtu <mtu>`
+- Increase TCP buffer sizes:
+    - **TODO**
+- **TODO**
+    - [https://fasterdata.es.net/host-tuning/linux/](https://fasterdata.es.net/host-tuning/linux/)
+    - [https://community.mellanox.com/s/article/performance-tuning-for-mellanox-adapters](https://community.mellanox.com/s/article/performance-tuning-for-mellanox-adapters)
 
 ## Notes
 

config/server/linux-networking.md

@@ -19,20 +19,29 @@ breadcrumbs:
 1. Install RDMA: `apt install rdma-core`
 1. Install user-space RDMA stuff: `apt install ibverbs-providers rdmacm-utils infiniband-diags ibverbs-utils`
 1. Install subnet manager (SM): `apt install opensm`
-    - Only one instance should be required on the subnet.
+    - Only one instance is required on the network, but multiple may be used for redundancy.
+    - A master SM is selected based on configured priority, with GUID as a tie breaker.
 1. Setup IPoIB:
     - Just like for Ethernet. Just specify the IB interface as the L2 device.
     - Use an appropriate MTU like 2044.
-1. Make sure ping and ping-pong is working.
+1. Make sure ping and ping-pong is working (see examples below).
 
 ### Usage
 
 - Show IPoIB status: `ip a`
-- Show port status: `ibstat`
-- Show hosts: `ibhosts`
-- Show switches: `ibswitches`
+- Show local devices:
+    - GUIDs: `ibv_devices`
+    - Basics (1): `ibstatus`
+    - Basics (2): `ibstat`
+    - Basics (3): `ibv_devinfo`
 - Show link statuses for network: `iblinkinfo`
+- Show subnet nodes:
+    - Hosts: `ibhosts`
+    - Switches: `ibswitches`
+    - Routers: `ibrouters`
 - Show active subnet manager(s): `sminfo`
+- Show subnet topology: `ibnetdiscover`
+- Show port counters: `perfquery`
 
 #### Testing
 
@@ -40,10 +49,14 @@ breadcrumbs:
     - Server: `ibping -S`
     - Client: `ibping -G <guid>`
 - Ping-pong:
-    - Server: `ibv_rc_pingpong`
-    - Client: `ibv_rc_pingpong <ip>`
+    - Server: `ibv_rc_pingpong -d <device> [-n <iters>]`
+    - Client: `ibv_rc_pingpong [-n <iters>] <ip>`
 - Other tools:
     - qperf
     - perftest
+- Diagnose with ibutils:
+    - Requires the `ibutils` package.
+    - Diagnose fabric: `ibdiagnet -ls 10 -lw 4x` (example)
+    - Diagnose path between two nodes: `ibdiagpath -l 65,1` (example)
 
 {% include footer.md %}