As requested, I’ve gathered some statistics from my setup to help identify any farming bottlenecks and figure out where we might squeeze more performance. Below are the core utilization and I/O wait percentages, along with other relevant system stats.
Let me know if more information is needed or if there’s any specific area you’d like to dive deeper into.
Snapshot build #386
Results: 6 Wins, 25 Slow Blocks (80.6%)
System Specs (24 * 15.36TB):
System:
Host: atc01 Kernel: 6.8.0-45-generic arch: x86_64 bits: 64 compiler: gcc v: 13.2.0
Console: pty pts/1 Distro: Ubuntu 24.04.1 LTS (Noble Numbat)
Machine:
Type: Server System: Dell product: PowerEdge R7515 v: N/A serial: <superuser required>
Mobo: Dell model: 035YY8 v: A03 serial: <superuser required>
part-nu: SKU=08FD;ModelName=PowerEdge R7515 UEFI: Dell v: 2.15.3 date: 05/15/2024
CPU:
Info: 24-core model: AMD EPYC 7F72 bits: 64 type: MT MCP arch: Zen 2 rev: 0 cache: L1: 1.5 MiB
L2: 12 MiB L3: 192 MiB
Speed (MHz): avg: 3600 high: 3603 min/max: 2500/3200 boost: enabled cores: 1: 3600 2: 3600
Flags: avx avx2 ht lm nx pae sse sse2 sse3 sse4_1 sse4_2 sse4a ssse3 svm
Network:
Device-1: Mellanox MT27800 Family [ConnectX-5] driver: mlx5_core v: kernel pcie: speed: 8 GT/s
lanes: 16 port: N/A bus-ID: 01:00.0 chip-ID: 15b3:1017 temp: 39.0 C
IF: enp1s0np0 state: up speed: 100000 Mbps duplex: full mac: ec:0d:9a:44:1b:e8
Drives:
Local Storage: total: 336.21 TiB used: 335.44 TiB (99.8%)
ID-1: /dev/nvme0n1 vendor: Intel model: SSDPE2NV153T8 size: 13.97 TiB speed: 31.6 Gb/s
lanes: 4 serial: PHLL15110044xxxxxx temp: 23.9 C
ID-2: /dev/nvme10n1 vendor: Intel model: SSDPE2NV153T8 size: 13.97 TiB speed: 31.6 Gb/s
lanes: 4 serial: PHLL1506009Exxxxxx temp: 22.9 C
ID-3: /dev/nvme11n1 vendor: Intel model: SSDPE2NV153T8 size: 13.97 TiB speed: 31.6 Gb/s
...
ID-23: /dev/nvme8n1 vendor: Intel model: SSDPE2NV153T8 size: 13.97 TiB speed: 31.6 Gb/s
lanes: 4 serial: PHLL15110096xxxxxx temp: 23.9 C
ID-24: /dev/nvme9n1 vendor: Intel model: SSDPE2NV153T8 size: 13.97 TiB speed: 31.6 Gb/s
lanes: 4 serial: PHLL1511006Qxxxxxx temp: 22.9 C
ID-25: /dev/sda vendor: Micron model: MTFDDAV480TDS size: 447.13 GiB speed: 6.0 Gb/s
serial: 221737xxxxxx
ID-26: /dev/sdb vendor: Micron model: MTFDDAV480TDS size: 447.13 GiB speed: 6.0 Gb/s
serial: 222138xxxxxx
Info:
Memory: total: 256 GiB available: 251.29 GiB used: 51.05 GiB (20.3%)
Processes: 680 Power: uptime: 4h 19m wakeups: 0 Init: systemd v: 255 target: graphical (5)
1. Core Utilization and I/O Wait (mpstat -P ALL 10 5):
Linux 6.8.0-45-generic (atc01) 09/21/2024 _x86_64_ (48 CPU)
07:57:10 PM CPU %usr %nice %sys %iowait %irq %soft %steal %guest %gnice %idle
07:57:20 PM all 33.19 0.00 66.77 0.01 0.00 0.03 0.00 0.00 0.00 0.00
07:57:20 PM 0 33.83 0.00 66.07 0.00 0.00 0.10 0.00 0.00 0.00 0.00
07:57:20 PM 1 33.87 0.00 66.13 0.00 0.00 0.00 0.00 0.00 0.00 0.00
07:57:20 PM 2 33.33 0.00 66.67 0.00 0.00 0.00 0.00 0.00 0.00 0.00
07:57:20 PM 3 34.23 0.00 65.67 0.00 0.00 0.10 0.00 0.00 0.00 0.00
07:57:20 PM 4 33.97 0.00 66.03 0.00 0.00 0.00 0.00 0.00 0.00 0.00
07:57:20 PM 5 33.97 0.00 66.03 0.00 0.00 0.00 0.00 0.00 0.00 0.00
07:57:20 PM 6 34.20 0.00 65.80 0.00 0.00 0.00 0.00 0.00 0.00 0.00
07:57:20 PM 7 33.77 0.00 66.23 0.00 0.00 0.00 0.00 0.00 0.00 0.00
07:57:20 PM 8 34.93 0.00 65.07 0.00 0.00 0.00 0.00 0.00 0.00 0.00
07:57:20 PM 9 33.90 0.00 66.00 0.00 0.00 0.10 0.00 0.00 0.00 0.00
07:57:20 PM 10 33.97 0.00 66.03 0.00 0.00 0.00 0.00 0.00 0.00 0.00
07:57:20 PM 11 33.60 0.00 66.40 0.00 0.00 0.00 0.00 0.00 0.00 0.00
07:57:20 PM 12 34.67 0.00 65.33 0.00 0.00 0.00 0.00 0.00 0.00 0.00
07:57:20 PM 13 33.57 0.00 66.43 0.00 0.00 0.00 0.00 0.00 0.00 0.00
07:57:20 PM 14 33.47 0.00 65.93 0.00 0.00 0.60 0.00 0.00 0.00 0.00
07:57:20 PM 15 33.87 0.00 66.13 0.00 0.00 0.00 0.00 0.00 0.00 0.00
07:57:20 PM 16 34.20 0.00 65.80 0.00 0.00 0.00 0.00 0.00 0.00 0.00
07:57:20 PM 17 32.73 0.00 67.17 0.00 0.00 0.10 0.00 0.00 0.00 0.00
07:57:20 PM 18 34.03 0.00 65.97 0.00 0.00 0.00 0.00 0.00 0.00 0.00
07:57:20 PM 19 33.93 0.00 66.07 0.00 0.00 0.00 0.00 0.00 0.00 0.00
07:57:20 PM 20 33.67 0.00 66.33 0.00 0.00 0.00 0.00 0.00 0.00 0.00
07:57:20 PM 21 34.37 0.00 65.63 0.00 0.00 0.00 0.00 0.00 0.00 0.00
07:57:20 PM 22 33.67 0.00 66.23 0.10 0.00 0.00 0.00 0.00 0.00 0.00
07:57:20 PM 23 0.00 0.00 100.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
07:57:20 PM 24 34.17 0.00 65.83 0.00 0.00 0.00 0.00 0.00 0.00 0.00
07:57:20 PM 25 33.97 0.00 66.03 0.00 0.00 0.00 0.00 0.00 0.00 0.00
07:57:20 PM 26 34.17 0.00 65.73 0.00 0.00 0.10 0.00 0.00 0.00 0.00
07:57:20 PM 27 34.33 0.00 65.57 0.00 0.00 0.10 0.00 0.00 0.00 0.00
07:57:20 PM 28 33.77 0.00 66.23 0.00 0.00 0.00 0.00 0.00 0.00 0.00
07:57:20 PM 29 33.33 0.00 66.67 0.00 0.00 0.00 0.00 0.00 0.00 0.00
07:57:20 PM 30 34.17 0.00 65.83 0.00 0.00 0.00 0.00 0.00 0.00 0.00
07:57:20 PM 31 34.53 0.00 65.37 0.00 0.00 0.10 0.00 0.00 0.00 0.00
07:57:20 PM 32 33.63 0.00 66.37 0.00 0.00 0.00 0.00 0.00 0.00 0.00
07:57:20 PM 33 34.30 0.00 65.70 0.00 0.00 0.00 0.00 0.00 0.00 0.00
07:57:20 PM 34 33.80 0.00 66.20 0.00 0.00 0.00 0.00 0.00 0.00 0.00
07:57:20 PM 35 33.77 0.00 66.23 0.00 0.00 0.00 0.00 0.00 0.00 0.00
07:57:20 PM 36 33.77 0.00 66.23 0.00 0.00 0.00 0.00 0.00 0.00 0.00
07:57:20 PM 37 33.33 0.00 66.67 0.00 0.00 0.00 0.00 0.00 0.00 0.00
07:57:20 PM 38 33.87 0.00 66.13 0.00 0.00 0.00 0.00 0.00 0.00 0.00
07:57:20 PM 39 33.23 0.00 66.67 0.00 0.00 0.10 0.00 0.00 0.00 0.00
07:57:20 PM 40 33.33 0.00 66.57 0.00 0.00 0.10 0.00 0.00 0.00 0.00
07:57:20 PM 41 33.47 0.00 66.53 0.00 0.00 0.00 0.00 0.00 0.00 0.00
07:57:20 PM 42 33.57 0.00 66.43 0.00 0.00 0.00 0.00 0.00 0.00 0.00
07:57:20 PM 43 34.40 0.00 65.60 0.00 0.00 0.00 0.00 0.00 0.00 0.00
07:57:20 PM 44 33.47 0.00 66.53 0.00 0.00 0.00 0.00 0.00 0.00 0.00
07:57:20 PM 45 34.17 0.00 65.83 0.00 0.00 0.00 0.00 0.00 0.00 0.00
07:57:20 PM 46 34.80 0.00 65.20 0.00 0.00 0.00 0.00 0.00 0.00 0.00
07:57:20 PM 47 33.97 0.00 65.73 0.20 0.00 0.10 0.00 0.00 0.00 0.00
...
2. Disk I/O Statistics (iostat -x 10 5):
avg-cpu: %user %nice %system %iowait %steal %idle
32.50 0.00 67.41 0.09 0.00 0.00
Device r/s rkB/s rrqm/s %rrqm r_await rareq-sz w/s wkB/s wrqm/s %wrqm w_await wareq-sz d/s dkB/s drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util
loop0 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
nvme0n1 9165.13 178515.88 0.00 0.00 0.25 19.48 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 2.30 67.17
nvme10n1 9300.10 181126.07 0.00 0.00 0.25 19.48 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 2.33 66.21
nvme11n1 10948.85 213458.94 0.00 0.00 0.25 19.50 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 2.75 70.74
nvme12n1 9169.13 178595.80 0.00 0.00 0.25 19.48 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 2.26 68.08
nvme13n1 9750.45 189838.56 0.00 0.00 0.25 19.47 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 2.40 65.71
nvme14n1 9460.64 184037.16 0.00 0.00 0.25 19.45 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 2.32 68.29
nvme15n1 9213.29 179365.43 0.00 0.00 0.25 19.47 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 2.26 67.22
nvme16n1 9164.04 178625.77 0.00 0.00 0.25 19.49 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 2.29 66.31
nvme17n1 9227.87 179955.24 0.00 0.00 0.24 19.50 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 2.26 68.10
nvme18n1 9863.84 192138.66 0.00 0.00 0.24 19.48 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 2.41 70.24
nvme19n1 10797.60 210507.09 0.00 0.00 0.25 19.50 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 2.67 67.73
nvme1n1 10440.16 202748.85 0.00 0.00 0.25 19.42 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 2.62 68.31
nvme20n1 11168.83 217640.36 0.00 0.00 0.25 19.49 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 2.83 70.64
nvme21n1 8963.94 174461.54 0.00 0.00 0.25 19.46 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 2.26 65.90
nvme22n1 8969.13 174574.23 0.00 0.00 0.25 19.46 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 2.25 66.40
nvme23n1 8801.20 171200.80 0.00 0.00 0.25 19.45 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 2.22 66.09
nvme2n1 10681.42 208169.03 0.00 0.00 0.25 19.49 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 2.70 71.16
nvme3n1 9558.54 186083.12 0.00 0.00 0.25 19.47 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 2.42 67.22
nvme4n1 10515.58 204753.65 0.00 0.00 0.25 19.47 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 2.67 70.59
nvme5n1 8705.00 169636.76 0.00 0.00 0.25 19.49 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 2.19 65.92
nvme6n1 8471.83 165021.38 0.00 0.00 0.25 19.48 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 2.12 64.22
nvme7n1 10104.30 196927.47 0.00 0.00 0.25 19.49 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 2.56 69.74
nvme8n1 8919.18 173804.20 0.00 0.00 0.25 19.49 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 2.22 66.11
nvme9n1 10489.41 204420.78 0.00 0.00 0.25 19.49 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 2.65 69.64
sda 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
sdb 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00 0.00
...
3. Process-Level Stats (top output):
Tasks: 681 total, 2 running, 679 sleeping, 0 stopped, 0 zombie
%Cpu(s): 42.3 us, 57.7 sy, 0.0 ni, 0.0 id, 0.0 wa, 0.0 hi, 0.0 si, 0.0 st
MiB Mem : 257323.5 total, 3632.2 free, 51408.1 used, 204943.5 buff/cache
MiB Swap: 0.0 total, 0.0 free, 0.0 used. 205915.5 avail Mem
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
20671 vexr 20 0 121.8g 45.3g 16896 S 4694 18.0 14,24 subspace-farmer
356 root 20 0 0 0 0 R 84.4 0.0 17:38.85 kswapd0
22848 vexr 20 0 12056 4992 3072 R 6.2 0.0 0:00.05 top
...
4. Quick Overview (vmstat 10 5):
procs -----------memory---------- ---swap-- -----io---- -system-- -------cpu-------
r b swpd free buff cache si so bi bo in cs us sy id wa st gu
666 80 0 4271120 311036 208549824 0 0 4095639 11795 683386 70 2 30 67 0 0 0
621 84 0 4261700 311040 209021296 0 0 3939737 1 259417 277616 30 65 4 1 0 0
755 15 0 4275596 311044 208252792 0 0 3350753 0 226500 241541 25 70 4 1 0 0
135 0 0 5442832 311060 207071604 0 0 367476 19 69083 36720 2 98 0 0 0 0
692 77 0 3556732 311072 208270848 0 0 5259680 4 317669 351835 41 59 0 0 0 0
5. I/O Wait and Disk Activity by Process (iotop):
Total DISK READ: 5.73 G/s | Total DISK WRITE: 0.00 B/s
Current DISK READ: 5.78 G/s | Current DISK WRITE: 0.00 B/s
...