Core Utilization & I/O Wait Stats for Farming Performance

As requested, I’ve gathered some statistics from my setup to help identify any farming bottlenecks and figure out where we might squeeze more performance. Below are the core utilization and I/O wait percentages, along with other relevant system stats.

Let me know if more information is needed or if there’s any specific area you’d like to dive deeper into.

Snapshot build #386

Results: 6 Wins, 25 Slow Blocks (80.6%)

System Specs (24 * 15.36TB):

System:
  Host: atc01 Kernel: 6.8.0-45-generic arch: x86_64 bits: 64 compiler: gcc v: 13.2.0
  Console: pty pts/1 Distro: Ubuntu 24.04.1 LTS (Noble Numbat)
Machine:
  Type: Server System: Dell product: PowerEdge R7515 v: N/A serial: <superuser required> 
  Mobo: Dell model: 035YY8 v: A03 serial: <superuser required>
    part-nu: SKU=08FD;ModelName=PowerEdge R7515 UEFI: Dell v: 2.15.3 date: 05/15/2024
CPU:
  Info: 24-core model: AMD EPYC 7F72 bits: 64 type: MT MCP arch: Zen 2 rev: 0 cache: L1: 1.5 MiB
    L2: 12 MiB L3: 192 MiB
  Speed (MHz): avg: 3600 high: 3603 min/max: 2500/3200 boost: enabled cores: 1: 3600 2: 3600
  Flags: avx avx2 ht lm nx pae sse sse2 sse3 sse4_1 sse4_2 sse4a ssse3 svm
Network:
  Device-1: Mellanox MT27800 Family [ConnectX-5] driver: mlx5_core v: kernel pcie: speed: 8 GT/s
    lanes: 16 port: N/A bus-ID: 01:00.0 chip-ID: 15b3:1017 temp: 39.0 C
  IF: enp1s0np0 state: up speed: 100000 Mbps duplex: full mac: ec:0d:9a:44:1b:e8
Drives:
  Local Storage: total: 336.21 TiB used: 335.44 TiB (99.8%)
  ID-1: /dev/nvme0n1 vendor: Intel model: SSDPE2NV153T8 size: 13.97 TiB speed: 31.6 Gb/s
    lanes: 4 serial: PHLL15110044xxxxxx temp: 23.9 C
  ID-2: /dev/nvme10n1 vendor: Intel model: SSDPE2NV153T8 size: 13.97 TiB speed: 31.6 Gb/s
    lanes: 4 serial: PHLL1506009Exxxxxx temp: 22.9 C
  ID-3: /dev/nvme11n1 vendor: Intel model: SSDPE2NV153T8 size: 13.97 TiB speed: 31.6 Gb/s
   ...
  ID-23: /dev/nvme8n1 vendor: Intel model: SSDPE2NV153T8 size: 13.97 TiB speed: 31.6 Gb/s
    lanes: 4 serial: PHLL15110096xxxxxx temp: 23.9 C
  ID-24: /dev/nvme9n1 vendor: Intel model: SSDPE2NV153T8 size: 13.97 TiB speed: 31.6 Gb/s
    lanes: 4 serial: PHLL1511006Qxxxxxx temp: 22.9 C
  ID-25: /dev/sda vendor: Micron model: MTFDDAV480TDS size: 447.13 GiB speed: 6.0 Gb/s
    serial: 221737xxxxxx
  ID-26: /dev/sdb vendor: Micron model: MTFDDAV480TDS size: 447.13 GiB speed: 6.0 Gb/s
    serial: 222138xxxxxx
Info:
  Memory: total: 256 GiB available: 251.29 GiB used: 51.05 GiB (20.3%)
  Processes: 680 Power: uptime: 4h 19m wakeups: 0 Init: systemd v: 255 target: graphical (5)

1. Core Utilization and I/O Wait (mpstat -P ALL 10 5):

Linux 6.8.0-45-generic (atc01) 	09/21/2024 	_x86_64_	(48 CPU)

07:57:10 PM  CPU    %usr   %nice    %sys %iowait    %irq   %soft  %steal  %guest  %gnice   %idle
07:57:20 PM  all   33.19    0.00   66.77    0.01    0.00    0.03    0.00    0.00    0.00    0.00
07:57:20 PM    0   33.83    0.00   66.07    0.00    0.00    0.10    0.00    0.00    0.00    0.00
07:57:20 PM    1   33.87    0.00   66.13    0.00    0.00    0.00    0.00    0.00    0.00    0.00
07:57:20 PM    2   33.33    0.00   66.67    0.00    0.00    0.00    0.00    0.00    0.00    0.00
07:57:20 PM    3   34.23    0.00   65.67    0.00    0.00    0.10    0.00    0.00    0.00    0.00
07:57:20 PM    4   33.97    0.00   66.03    0.00    0.00    0.00    0.00    0.00    0.00    0.00
07:57:20 PM    5   33.97    0.00   66.03    0.00    0.00    0.00    0.00    0.00    0.00    0.00
07:57:20 PM    6   34.20    0.00   65.80    0.00    0.00    0.00    0.00    0.00    0.00    0.00
07:57:20 PM    7   33.77    0.00   66.23    0.00    0.00    0.00    0.00    0.00    0.00    0.00
07:57:20 PM    8   34.93    0.00   65.07    0.00    0.00    0.00    0.00    0.00    0.00    0.00
07:57:20 PM    9   33.90    0.00   66.00    0.00    0.00    0.10    0.00    0.00    0.00    0.00
07:57:20 PM   10   33.97    0.00   66.03    0.00    0.00    0.00    0.00    0.00    0.00    0.00
07:57:20 PM   11   33.60    0.00   66.40    0.00    0.00    0.00    0.00    0.00    0.00    0.00
07:57:20 PM   12   34.67    0.00   65.33    0.00    0.00    0.00    0.00    0.00    0.00    0.00
07:57:20 PM   13   33.57    0.00   66.43    0.00    0.00    0.00    0.00    0.00    0.00    0.00
07:57:20 PM   14   33.47    0.00   65.93    0.00    0.00    0.60    0.00    0.00    0.00    0.00
07:57:20 PM   15   33.87    0.00   66.13    0.00    0.00    0.00    0.00    0.00    0.00    0.00
07:57:20 PM   16   34.20    0.00   65.80    0.00    0.00    0.00    0.00    0.00    0.00    0.00
07:57:20 PM   17   32.73    0.00   67.17    0.00    0.00    0.10    0.00    0.00    0.00    0.00
07:57:20 PM   18   34.03    0.00   65.97    0.00    0.00    0.00    0.00    0.00    0.00    0.00
07:57:20 PM   19   33.93    0.00   66.07    0.00    0.00    0.00    0.00    0.00    0.00    0.00
07:57:20 PM   20   33.67    0.00   66.33    0.00    0.00    0.00    0.00    0.00    0.00    0.00
07:57:20 PM   21   34.37    0.00   65.63    0.00    0.00    0.00    0.00    0.00    0.00    0.00
07:57:20 PM   22   33.67    0.00   66.23    0.10    0.00    0.00    0.00    0.00    0.00    0.00
07:57:20 PM   23    0.00    0.00  100.00    0.00    0.00    0.00    0.00    0.00    0.00    0.00
07:57:20 PM   24   34.17    0.00   65.83    0.00    0.00    0.00    0.00    0.00    0.00    0.00
07:57:20 PM   25   33.97    0.00   66.03    0.00    0.00    0.00    0.00    0.00    0.00    0.00
07:57:20 PM   26   34.17    0.00   65.73    0.00    0.00    0.10    0.00    0.00    0.00    0.00
07:57:20 PM   27   34.33    0.00   65.57    0.00    0.00    0.10    0.00    0.00    0.00    0.00
07:57:20 PM   28   33.77    0.00   66.23    0.00    0.00    0.00    0.00    0.00    0.00    0.00
07:57:20 PM   29   33.33    0.00   66.67    0.00    0.00    0.00    0.00    0.00    0.00    0.00
07:57:20 PM   30   34.17    0.00   65.83    0.00    0.00    0.00    0.00    0.00    0.00    0.00
07:57:20 PM   31   34.53    0.00   65.37    0.00    0.00    0.10    0.00    0.00    0.00    0.00
07:57:20 PM   32   33.63    0.00   66.37    0.00    0.00    0.00    0.00    0.00    0.00    0.00
07:57:20 PM   33   34.30    0.00   65.70    0.00    0.00    0.00    0.00    0.00    0.00    0.00
07:57:20 PM   34   33.80    0.00   66.20    0.00    0.00    0.00    0.00    0.00    0.00    0.00
07:57:20 PM   35   33.77    0.00   66.23    0.00    0.00    0.00    0.00    0.00    0.00    0.00
07:57:20 PM   36   33.77    0.00   66.23    0.00    0.00    0.00    0.00    0.00    0.00    0.00
07:57:20 PM   37   33.33    0.00   66.67    0.00    0.00    0.00    0.00    0.00    0.00    0.00
07:57:20 PM   38   33.87    0.00   66.13    0.00    0.00    0.00    0.00    0.00    0.00    0.00
07:57:20 PM   39   33.23    0.00   66.67    0.00    0.00    0.10    0.00    0.00    0.00    0.00
07:57:20 PM   40   33.33    0.00   66.57    0.00    0.00    0.10    0.00    0.00    0.00    0.00
07:57:20 PM   41   33.47    0.00   66.53    0.00    0.00    0.00    0.00    0.00    0.00    0.00
07:57:20 PM   42   33.57    0.00   66.43    0.00    0.00    0.00    0.00    0.00    0.00    0.00
07:57:20 PM   43   34.40    0.00   65.60    0.00    0.00    0.00    0.00    0.00    0.00    0.00
07:57:20 PM   44   33.47    0.00   66.53    0.00    0.00    0.00    0.00    0.00    0.00    0.00
07:57:20 PM   45   34.17    0.00   65.83    0.00    0.00    0.00    0.00    0.00    0.00    0.00
07:57:20 PM   46   34.80    0.00   65.20    0.00    0.00    0.00    0.00    0.00    0.00    0.00
07:57:20 PM   47   33.97    0.00   65.73    0.20    0.00    0.10    0.00    0.00    0.00    0.00

...

2. Disk I/O Statistics (iostat -x 10 5):

avg-cpu:  %user   %nice %system %iowait  %steal   %idle
          32.50    0.00   67.41    0.09    0.00    0.00

Device            r/s     rkB/s   rrqm/s  %rrqm r_await rareq-sz     w/s     wkB/s   wrqm/s  %wrqm w_await wareq-sz     d/s     dkB/s   drqm/s  %drqm d_await dareq-sz     f/s f_await  aqu-sz  %util
loop0            0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    0.00   0.00
nvme0n1       9165.13 178515.88     0.00   0.00    0.25    19.48    0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    2.30  67.17
nvme10n1      9300.10 181126.07     0.00   0.00    0.25    19.48    0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    2.33  66.21
nvme11n1      10948.85 213458.94     0.00   0.00    0.25    19.50    0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    2.75  70.74
nvme12n1      9169.13 178595.80     0.00   0.00    0.25    19.48    0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    2.26  68.08
nvme13n1      9750.45 189838.56     0.00   0.00    0.25    19.47    0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    2.40  65.71
nvme14n1      9460.64 184037.16     0.00   0.00    0.25    19.45    0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    2.32  68.29
nvme15n1      9213.29 179365.43     0.00   0.00    0.25    19.47    0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    2.26  67.22
nvme16n1      9164.04 178625.77     0.00   0.00    0.25    19.49    0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    2.29  66.31
nvme17n1      9227.87 179955.24     0.00   0.00    0.24    19.50    0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    2.26  68.10
nvme18n1      9863.84 192138.66     0.00   0.00    0.24    19.48    0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    2.41  70.24
nvme19n1      10797.60 210507.09     0.00   0.00    0.25    19.50    0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    2.67  67.73
nvme1n1       10440.16 202748.85     0.00   0.00    0.25    19.42    0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    2.62  68.31
nvme20n1      11168.83 217640.36     0.00   0.00    0.25    19.49    0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    2.83  70.64
nvme21n1      8963.94 174461.54     0.00   0.00    0.25    19.46    0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    2.26  65.90
nvme22n1      8969.13 174574.23     0.00   0.00    0.25    19.46    0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    2.25  66.40
nvme23n1      8801.20 171200.80     0.00   0.00    0.25    19.45    0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    2.22  66.09
nvme2n1       10681.42 208169.03     0.00   0.00    0.25    19.49    0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    2.70  71.16
nvme3n1       9558.54 186083.12     0.00   0.00    0.25    19.47    0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    2.42  67.22
nvme4n1       10515.58 204753.65     0.00   0.00    0.25    19.47    0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    2.67  70.59
nvme5n1       8705.00 169636.76     0.00   0.00    0.25    19.49    0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    2.19  65.92
nvme6n1       8471.83 165021.38     0.00   0.00    0.25    19.48    0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    2.12  64.22
nvme7n1       10104.30 196927.47     0.00   0.00    0.25    19.49    0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    2.56  69.74
nvme8n1       8919.18 173804.20     0.00   0.00    0.25    19.49    0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    2.22  66.11
nvme9n1       10489.41 204420.78     0.00   0.00    0.25    19.49    0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    2.65  69.64
sda              0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    0.00   0.00
sdb              0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00      0.00     0.00   0.00    0.00     0.00    0.00    0.00    0.00   0.00

...

3. Process-Level Stats (top output):

Tasks: 681 total,   2 running, 679 sleeping,   0 stopped,   0 zombie
%Cpu(s): 42.3 us, 57.7 sy,  0.0 ni,  0.0 id,  0.0 wa,  0.0 hi,  0.0 si,  0.0 st 
MiB Mem : 257323.5 total,   3632.2 free,  51408.1 used, 204943.5 buff/cache     
MiB Swap:      0.0 total,      0.0 free,      0.0 used. 205915.5 avail Mem 

    PID USER      PR  NI    VIRT    RES    SHR S  %CPU  %MEM     TIME+ COMMAND                                                                                                                                                                                                                                                                                                                                                              
  20671 vexr      20   0  121.8g  45.3g  16896 S  4694  18.0     14,24 subspace-farmer                                                                                                                                                                                                                                                                                                                                                      
    356 root      20   0       0      0      0 R  84.4   0.0  17:38.85 kswapd0                                                                                                                                                                                                                                                                                                                                                              
  22848 vexr      20   0   12056   4992   3072 R   6.2   0.0   0:00.05 top     
...

4. Quick Overview (vmstat 10 5):

procs -----------memory---------- ---swap-- -----io---- -system-- -------cpu-------
 r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa st gu
666 80      0 4271120 311036 208549824    0    0 4095639 11795 683386   70  2 30 67  0  0  0
621 84      0 4261700 311040 209021296    0    0 3939737     1 259417 277616 30 65  4  1  0  0
755 15      0 4275596 311044 208252792    0    0 3350753     0 226500 241541 25 70  4  1  0  0
135  0      0 5442832 311060 207071604    0    0 367476    19 69083 36720  2 98  0  0  0  0
692 77      0 3556732 311072 208270848    0    0 5259680     4 317669 351835 41 59  0  0  0  0

5. I/O Wait and Disk Activity by Process (iotop):

Total DISK READ:         5.73 G/s | Total DISK WRITE:         0.00 B/s
Current DISK READ:       5.78 G/s | Current DISK WRITE:       0.00 B/s
...
4 Likes

Update from Snapshot Build #390

The outcomes are nothing short of amazing! A total of 368TB is being successfully farmed on a single EPYC 7F72, with a peak CPU load of 68% and an average load of 64%. The memory footprint remains unchanged. There were some initial issues with multiple io_page_fault errors, but these were resolved by disabling IOMMU.

A one-hour results snapshot shows 124 wins, with zero slow blocks!

If you would find it helpful to receive the complete set of system stats as provided in the previous post, please let me know.

2 Likes