Farmer OOM with 130TB plots and 32G of ram

Support
1

Ubuntu 22.04 32G ram and 2G swap OOM first time when I had all 130TB plotted, farmer version may-06
cli:

screen -S subspace ./subspace-farmer-ubuntu-x86_64-skylake-gemini-3h-2024-may-06 farm --node-rpc-url ws://192.168.5.12:9944 --listen-on /ip4/0.0.0.0/tcp/32542 --listen-on /ip4/0.0.0.0/udp/32542/quic-v1 --reward-address st9PB769iY44Jno7So5NU1wSfE6MMSv3xULnu1Nn2NWAG1Cm3 path=/root/sub,size=3400GB path=/king2/sub,size=3990GB path=/king1/sub,size=3990GB path=/sam1/sub,size=3990GB path=/sam1/sub2,size=3990GB path=/sam2/sub,size=7990GB path=/sam3/sub,size=7990GB path=/sam4/sub,size=7990GB path=/sam5/sub,size=7990GB path=/sam6/sub,size=7990GB path=/sam7/sub,size=7990GB path=/sam8/sub,size=7990GB path=/sam9/sub,size=7990GB path=/sam10/sub,size=7990GB path=/sam11/sub,size=7990GB path=/sam12/sub,size=7990GB path=/sam13/sub,size=7990GB path=/sam14/sub,size=7990GB path=/sam15/sub,size=7990GB path=/sam16/sub,size=7990GB --metrics-endpoints 192.168.5.35:18585

Log from kernel

May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566264] CPU: 29 PID: 54658 Comm: farming-6.6 Not tainted 6.5.0-28-generic #29~22.04.1-Ubuntu
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566267] Hardware name: ASRock B650M Pro RS/B650M Pro RS, BIOS 2.08.AS01 01/31/2024
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566268] Call Trace:
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566270]  <TASK>
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566273]  dump_stack_lvl+0x48/0x70
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566279]  dump_stack+0x10/0x20
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566281]  dump_header+0x50/0x290
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566284]  oom_kill_process+0x10d/0x1c0
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566286]  out_of_memory+0x103/0x350
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566288]  __alloc_pages_may_oom+0x112/0x1e0
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566292]  __alloc_pages_slowpath.constprop.0+0x46f/0x9a0
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566295]  __alloc_pages+0x31d/0x350
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566298]  alloc_pages+0x91/0x1a0
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566302]  folio_alloc+0x1d/0x60
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566304]  filemap_alloc_folio+0x31/0x40
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566307]  filemap_get_pages+0x143/0x3b0
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566312]  filemap_read+0xe7/0x430
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566317]  generic_file_read_iter+0xbb/0x110
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566319]  ext4_file_read_iter+0x63/0x210
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566322]  ? security_file_permission+0x5f/0x70
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566326]  vfs_read+0x20a/0x360
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566331]  __x64_sys_pread64+0xa6/0xd0
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566334]  do_syscall_64+0x58/0x90
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566337]  ? srso_alias_return_thunk+0x5/0x7f
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566340]  ? common_interrupt+0x54/0xb0
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566342]  entry_SYSCALL_64_after_hwframe+0x6e/0xd8
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566345] RIP: 0033:0x7ed10571278f
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566360] Code: 08 89 3c 24 48 89 4c 24 18 e8 7d e2 f7 ff 4c 8b 54 24 18 48 8b 54 24 10 41 89 c0 48 8b 74 24 08 8b 3c 24 b8 11 00 00 00 0f 05 <48> 3d 00 f0 ff ff 77 31 44 89 c7 48 89 04 24 e8 bd e2 f7 ff 48 8b
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566362] RSP: 002b:00007ec43dbb9f30 EFLAGS: 00000293 ORIG_RAX: 0000000000000011
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566364] RAX: ffffffffffffffda RBX: 00007ed105712730 RCX: 00007ed10571278f
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566366] RDX: 0000000000004e80 RSI: 00007ec4e4085000 RDI: 0000000000000259
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566367] RBP: 0000000000000259 R08: 0000000000000000 R09: 0000000000000000
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566368] R10: 000000b337e6b1a0 R11: 0000000000000293 R12: 0000000000004e80
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566369] R13: 7fffffffffffffff R14: 000000b337e6b1a0 R15: 00007ec4e4085000
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566373]  </TASK>
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566374] Mem-Info:
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566376] active_anon:986196 inactive_anon:4498206 isolated_anon:0
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566376]  active_file:3191 inactive_file:13129 isolated_file:0
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566376]  unevictable:0 dirty:2 writeback:0
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566376]  slab_reclaimable:2175163 slab_unreclaimable:68615
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566376]  mapped:1177 shmem:146 pagetables:34523
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566376]  sec_pagetables:0 bounce:0
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566376]  kernel_misc_reclaimable:0
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566376]  free:67593 free_pcp:132 free_cma:0
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566380] Node 0 active_anon:3944784kB inactive_anon:17992824kB active_file:12764kB inactive_file:52516kB unevictable:0kB isolated(anon):0kB isolated(file):0kB mapped:4708kB dirty:8kB writeback:0kB shmem:584kB shmem_thp: 0kB shmem_pmdmappe>
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566385] Node 0 DMA free:11264kB boost:0kB min:32kB low:44kB high:56kB reserved_highatomic:0KB active_anon:0kB inactive_anon:0kB active_file:0kB inactive_file:0kB unevictable:0kB writepending:0kB present:15996kB managed:15360kB mlocked:0k>
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566390] lowmem_reserve[]: 0 2053 31099 31099 31099
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566394] Node 0 DMA32 free:143892kB boost:0kB min:4460kB low:6560kB high:8660kB reserved_highatomic:22528KB active_anon:349760kB inactive_anon:787684kB active_file:1416kB inactive_file:5216kB unevictable:0kB writepending:0kB present:22377>
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566398] lowmem_reserve[]: 0 0 29045 29045 29045
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566402] Node 0 Normal free:115216kB boost:124928kB min:188016kB low:217756kB high:247496kB reserved_highatomic:10240KB active_anon:3595548kB inactive_anon:17204180kB active_file:9284kB inactive_file:47828kB unevictable:0kB writepending:8>
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566406] lowmem_reserve[]: 0 0 0 0 0
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566409] Node 0 DMA: 0*4kB 0*8kB 0*16kB 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 1*1024kB (U) 1*2048kB (M) 2*4096kB (M) = 11264kB
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566420] Node 0 DMA32: 4327*4kB (UME) 257*8kB (UME) 209*16kB (UME) 1845*32kB (UME) 736*64kB (UME) 116*128kB (UM) 1*256kB (U) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 143956kB
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566434] Node 0 Normal: 22255*4kB (UME) 733*8kB (UME) 1573*16kB (UME) 11*32kB (UME) 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 120404kB
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566446] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566447] Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566449] 21323 total pagecache pages
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566449] 643 pages in swap cache
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566450] Free swap  = 88kB
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566451] Total swap = 2097148kB
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566452] 8157032 pages RAM
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566452] 0 pages HighMem/MovableOnly
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566453] 172697 pages reserved
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566454] 0 pages hwpoisoned
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566454] Tasks state (memory values in pages):
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566455] [  pid  ]   uid  tgid total_vm      rss pgtables_bytes swapents oom_score_adj name
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566459] [    550]     0   550    32553      904   258048      384          -250 systemd-journal
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566462] [    602]     0   602     6879     1152    81920      320         -1000 systemd-udevd
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566464] [   1042]   130  1042     2027      448    57344       64             0 rpcbind
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566466] [   1043]   108  1043     3709      704    69632       64          -900 systemd-oomd
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566468] [   1044]   101  1044     6418     1152    86016      704             0 systemd-resolve
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566470] [   1045]   103  1045    22347      704    77824       64             0 systemd-timesyn
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566472] [   1053]     0  1053     1282      256    49152        0             0 blkmapd
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566474] [   1054]     0  1054      777      384    45056        0             0 rpc.idmapd
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566476] [   1055]     0  1055     1366      832    45056        0             0 nfsdcld
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566478] [   1059]     0  1059    60721      448   110592      320             0 accounts-daemon
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566480] [   1060]     0  1060      704      320    49152        0             0 acpid
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566481] [   1063]   114  1063     1941      704    53248        0             0 avahi-daemon
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566483] [   1064]     0  1064     3094      704    65536        0             0 cron
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566485] [   1065]   102  1065     2768      640    61440      448          -900 dbus-daemon
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566487] [   1067]     0  1067   102865     1471   176128      320             0 NetworkManager
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566489] [   1074]     0  1074    20765      832    65536       64             0 irqbalance
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566491] [   1079]     0  1079    10973     1344   122880     1856             0 networkd-dispat
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566492] [   1081]     0  1081    61488     1728   110592      128             0 polkitd
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566494] [   1084]     0  1084    60735      768   106496      128             0 power-profiles-
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566496] [   1086]   104  1086    55601      960    81920      320             0 rsyslogd
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566499] [   1092]     0  1092     3138      561    65536      320             0 smartd
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566501] [   1094]     0  1094   866292     3333   540672     1088          -900 snapd
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566503] [   1095]     0  1095    59837     1024   102400      128             0 switcheroo-cont
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566505] [   1096]     0  1096    12100      448    98304      512             0 systemd-logind
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566507] [   1098]     0  1098   117277     1536   147456      960             0 udisksd
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566510] [   1100]     0  1100     4126      512    73728      128             0 wpa_supplicant
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566512] [   1105]   114  1105     1861      128    53248       64             0 avahi-daemon
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566514] [   1195]     0  1195    30259     1344   143360     1600             0 unattended-upgr
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566516] [   1203]     0  1203    79490     1280   118784      256             0 ModemManager
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566518] [   1220]     0  1220     3859     1088    73728      320         -1000 sshd
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566520] [   1319]     0  1319    61058      768   110592      192             0 gdm3
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566522] [   1354]  1000  1354     4625      896    77824      576             0 systemd
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566524] [   1361]  1000  1361    42605      914    98304      512             0 (sd-pam)
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566526] [   1423]  1000  1423    10615      768    86016      192             0 pipewire
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566527] [   1424]  1000  1424     6622      512    81920      192             0 pipewire-media-
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566529] [   1443]   116  1443    38501      512    65536        0             0 rtkit-daemon
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566531] [   1698]     0  1698    43334     1216   106496       64             0 gdm-session-wor
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566533] [   1717]     0  1717    61276      960   110592       64             0 upowerd
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566535] [   1720]  1000  1720    60937     1095    98304        0             0 gnome-keyring-d
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566536] [   1725]  1000  1725    41328     1024    90112       64             0 gdm-x-session
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566538] [   1727]  1000  1727   212333     1344   528384     9792             0 Xorg
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566540] [   1764]  1000  1764     2383      960    57344        0             0 dbus-daemon
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566542] [   1766]  1000  1766    60881      704   110592      192             0 gvfsd
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566544] [   1771]  1000  1771    95223      896    98304       64             0 gvfsd-fuse
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566545] [   1784]  1000  1784   162710     1468   364544     1990             0 tracker-miner-f
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566547] [   1794]  1000  1794    79799     1600   126976      256             0 gvfs-udisks2-vo
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566549] [   1799]  1000  1799    60107      896   102400      128             0 gvfs-gphoto2-vo
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566551] [   1803]  1000  1803    59835      960   110592        0             0 gvfs-mtp-volume
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566552] [   1807]  1000  1807    79525      832   122880      128             0 gvfs-afc-volume
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566554] [   1812]  1000  1812    59881      960   106496        0             0 gvfs-goa-volume
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566556] [   1816]  1000  1816   146172      896   290816     1600             0 goa-daemon
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566558] [   1826]  1000  1826    85320      704   151552      384             0 goa-identity-se
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566560] [   1836]  1000  1836    56480     1472   151552        0             0 gnome-session-b
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566562] [   1901]  1000  1901    77421     1088   102400      192             0 at-spi-bus-laun
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566564] [   1907]  1000  1907     2108      832    57344        0             0 dbus-daemon
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566568] [   1921]  1000  1921    23697     1024    90112       64             0 gnome-session-c
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566571] [   1933]  1000  1933   148954      640   204800      512             0 gnome-session-b
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566573] [   1952]  1000  1952  1086696     7581  1142784    24694             0 gnome-shell
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566575] [   1980]  1000  1980    59757     1024   106496      128             0 xdg-permission-
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566577] [   1982]  1000  1982   145718     1408   200704      192             0 gnome-shell-cal
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566579] [   1991]  1000  1991   268763      832   274432      704             0 evolution-sourc
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566581] [   1995]  1000  1995   272328     1280   208896      960             0 pulseaudio
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566583] [   1999]  1000  1999    39209     1024    73728      192             0 dconf-service
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566585] [   2003]     0  2003    75280      832   176128      704             0 packagekitd
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566586] [   2014]  1000  2014   329629     1278   319488      960             0 evolution-calen
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566588] [   2019]  1000  2019    97869      832   131072      256             0 gvfsd-trash
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566590] [   2026]  1000  2026   717125     1897   270336      384             0 gjs
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566592] [   2043]  1000  2043    40688      960    86016        0             0 at-spi2-registr
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566594] [   2053]  1000  2053   170836      640   262144      576             0 evolution-addre
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566596] [   2059]  1000  2059      723      256    40960        0             0 sh
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566598] [   2060]  1000  2060    78325     1088   110592       64             0 gsd-a11y-settin
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566599] [   2062]  1000  2062    79594     1344   126976      896             0 ibus-daemon
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566601] [   2063]  1000  2063   132188      883   212992     1856             0 gsd-color
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566603] [   2065]  1000  2065    94585     1024   176128      448             0 gsd-datetime
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566605] [   2067]  1000  2067    78744      832   110592      192             0 gsd-housekeepin
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566607] [   2068]  1000  2068    86079      969   167936     1600             0 gsd-keyboard
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566609] [   2070]  1000  2070   216969     1062   233472     1600             0 gsd-media-keys
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566611] [   2072]  1000  2072   113577      887   208896     1728             0 gsd-power
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566612] [   2075]  1000  2075    63183      832   131072      256             0 gsd-print-notif
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566614] [   2077]  1000  2077   115184     1024   122880        0             0 gsd-rfkill
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566616] [   2078]  1000  2078    59792     1024   106496      128             0 gsd-screensaver
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566618] [   2079]  1000  2079   117170      896   139264      128             0 gsd-sharing
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566620] [   2080]  1000  2080    78796      896   118784      256             0 gsd-smartcard
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566622] [   2082]  1000  2082    80554     1024   126976      128             0 gsd-sound
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566624] [   2087]  1000  2087    86166     1278   176128     1280             0 gsd-wacom
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566625] [   2094]  1000  2094    86569     1147   180224     1600             0 gsd-xsettings
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566627] [   2112]  1000  2112   187584     1132   385024     3392             0 evolution-alarm
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566629] [   2114]  1000  2114    58069     1280    86016      192             0 gsd-disk-utilit
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566632] [   2162]  1000  2162    41593     1152    94208       64             0 ibus-memconf
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566634] [   2182]  1000  2182    68858     1407   176128     2240             0 ibus-extension-
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566636] [   2200]   123  2200    62089     1142   114688      832             0 colord
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566638] [   2205]  1000  2205    49273     1594   155648     1152             0 ibus-x11
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566640] [   2208]  1000  2208    60038     1216    98304        0             0 ibus-portal
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566643] [   2237]  1000  2237   719204     1698   278528      576             0 gjs
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566645] [   2251]  1000  2251   119807      448   151552      576             0 xdg-desktop-por
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566647] [   2278]  1000  2278    41591     1152    94208      128             0 ibus-engine-sim
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566649] [   2280]  1000  2280   116513     1024   139264        0             0 xdg-document-po
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566651] [   2288]  1000  2288      699      320    49152        0             0 fusermount3
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566653] [   2295]  1000  2295   314530     1182   540672     4864             0 xdg-desktop-por
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566655] [   2305]  1000  2305    86310     1024   159744      384             0 gsd-printer
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566657] [   2326]  1000  2326    86245     1408   172032     1088             0 xdg-desktop-por
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566658] [   2327]  1000  2327     9737      192   114688      256             0 snapd-desktop-i
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566660] [   2406]  1000  2406   228829     1366   524288     4544             0 snapd-desktop-i
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566662] [   2485]   131  2485     1194      384    49152       64             0 rpc.statd
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566664] [   2486]     0  2486     1653     1528    53248      192             0 rpc.mountd
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566666] [   2487]   113  2487     3272      771    65536        0             0 kerneloops
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566667] [   2491]   113  2491     3272      448    65536       64             0 kerneloops
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566669] [   2522]  1000  2522    41473     1024    94208      128             0 gvfsd-metadata
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566671] [   2527]  1000  2527   123634     1536   200704     1152             0 update-notifier
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566673] [   2695]     0  2695     4540     1152    77824      192             0 systemd
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566675] [   2696]     0  2696    42606      889    98304      576             0 (sd-pam)
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566676] [   2703]     0  2703     9737      256   110592      256             0 snapd-desktop-i
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566678] [   2745]     0  2745     2122      704    57344      128             0 dbus-daemon
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566680] [   2751]     0  2751   116513      704   147456        0             0 xdg-document-po
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566682] [   2762]     0  2762    59757     1088   106496       64             0 xdg-permission-
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566684] [   2773]     0  2773      699      256    45056        0             0 fusermount3
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566686] [   2853]     0  2853    88065      448   184320     1024             0 snapd-desktop-i
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566688] [  14168]     0 14168     4366     1216    73728       64             0 sshd
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566690] [  14590]     0 14590     3504     1152    69632        0             0 bash
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566691] [  21385]  1000 21385   257963     2188   565248    17024             0 update-manager
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566694] [  54064]     0 54064     3125      576    69632        0             0 screen
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566695] [  54065]     0 54065     3333      384    53248        0             0 screen
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566697] [  54066]     0 54066 24279572  5464698 124502016   390293             0 subspace-farmer
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566699] [  56263]     0 56263    18941     1408   139264        0             0 cupsd
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566701] [  56265]     0 56265    43159      960   106496      128             0 cups-browsed
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.566703] oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=/,mems_allowed=0,global_oom,task_memcg=/user.slice/user-0.slice/session-638.scope,task=subspace-farmer,pid=54066,uid=0
May 14 07:05:47 ploter-asrock-7950x3d kernel: [1619176.567070] Out of memory: Killed process 54066 (subspace-farmer) total-vm:97118288kB, anon-rss:21854952kB, file-rss:3840kB, shmem-rss:0kB, UID:0 pgtables:121584kB oom_score_adj:0
May 14 07:05:50 ploter-asrock-7950x3d kernel: [1619180.273396] oom_reaper: reaped process 54066 (subspace-farmer), now anon-rss:0kB, file-rss:3072kB, shmem-rss:0kB
May 14 17:04:52 ploter-asrock-7950x3d kernel: [1655122.055161] loop5: detected capacity change from 0 to 130960
2

Okay, so just a bunch of farms with defaults. What CPU is this on? You can definitely limit plotting memory usage by constraining sector or record encoding parameters.

Remove UniqueRecordBinaryHeap · Issue #1769 · subspace/subspace · GitHub once implemented will also help with memory usage a bit.

But yeah, 130T is a lot of space and there is some bookkeeping that farmer stores in memory in order to work efficiently. It is not the highest priority to optimize right now, but I’ll keep it in mind.

UPD: I see from machine name it is likely 7950X, which should only have 2 L3 cache groups due to 2 chiplets used.

3

Yeah 7950x3d
I might not crash anymore with more swap will see, and if does I will limit constining sector or write for farmer to restart as even now it was first time so no issue

4

You said on Discord that after restart it uses ~20G of RAM. Is that while it is plotting or without plotting at all?

~20000M/~130000=~154kiB of data per sector (very rough math), which is way more than I would expect. But if it is still plotting then a significant chunk of it might be plotting part and it will settle lower once plotting is done.

Quite the capacity you have on that poor 7950X :slightly_smiling_face:

5

Yeah at least htop reported 73% mem use while replotting

Now replot finished just farming 130TB and htop report 57% of 32G use from farmer process

6

That is still high at ~140 kiB per sector

7

Would you mind sharing with me how could you have 130TB of SSD on your single PC? What is the mainboard and types of SSD and connection? Thank you.

8

Any pc, just using HBA card with 16x SSD with 8TB and one nvme, HBA card is in primary 16x slot but i guess could work in 2nd too

9

Thanks for the info.

Each 8TB once fully plotted, will require 120 MBps audit. You have 16 x SSD, so it would be 16 x 120 MBps = 1,920 MBps. Are you sure the bottle neck is not at your HBA card?

10

Hba is 8x pci 3.0 and can do 12GB/s
model: LSI SAS 9300-16I

11

The specs says it has 12Gbps. So 12Gbps = 1,500MBps only. There would be bottle neck, just I’m not sure if it has huge impact on your reward or not.

12

Then its 12Gbps per connector so 4x (I bet its this as it has 4 SAS ports) or 16x (as each SAS can connect 4 sata), as it do not state for details

13

I checked the code and indeed farmer will keep 131084 bytes of metadata per sector in memory in order to farm efficiently, this is a fixed size in addition to everything else that farmer might do (it will consume more during plotting for example).

There isn’t much that can be done to avoid that. We could potentially read it from disk, but then farming would require ~5x amount of reads. Memory-mapping is also a possibility to allow OS to cache it transparently and free when it really needs to, but then when it really needs to farming will suffer anyway.

So the conclusion is that you will need quite a bit of RAM for 130T worth of plotted space and I don’t think it is completely unreasonable.

14

It might be enough 16G+ 16G swap as it works for last 6 days, also systemd to restart on crash would help if OOM happen

even now mem use is same (no replot 56% use)

still puzzled why OOM killed it while process used “only” ~24GB at that time

15

Release gemini-3h-2024-jun-11 · subspace/subspace · GitHub includes some optimizations that should reduce peak memory usage for both node and farmer, curious if you see a meaningful difference

16

Updated, but as I had no more OOM with more swap I presume I will not have issue anymore

I plan to add more storage in 3 weeks as I will move nvme from other PC to USB enclosure so then will test more plot space with 32G of ram

17

I’m still curious if you see any meaningful difference when replotting kicks in or otherwise.

18

Still can see difference ~72-73% mem use or ~23G while replotting, if there where peaks now or in past not sure how to monitor those