Memory allocation of bytes failed (Advanced CLI May-01)

Support
1

Issue Report

Initial plotting failed with error “memory allocation of bytes failed”.

Node-1

PS C:\Subspace> .\subspace-farmer-windows-x86_64-skylake-gemini-3h-2024-may-01.exe farm --record-encoding-concurrency 3 path=D:\Subspace\farms_99,size=1000GB
2024-05-01T21:25:45.224455Z INFO subspace_farmer::commands::farm: Preparing plotting thread pools plotting_thread_pool_core_indices=[CpuCoreSet { cores: CpuSet(0-23), … }] replotting_thread_pool_core_indices=[CpuCoreSet { cores: CpuSet(0-11), … }]
…
2024-05-02T05:55:49.005445Z INFO {farm_index=0}: subspace_farmer::single_disk_farm::plotting: Plotting sector (64.21% complete) sector_index=601
memory allocation of memory allocation of 10486721048672memory allocation of bytes failed
bytes failed

Node-2

PS C:\Subspace> .\subspace-farmer-windows-x86_64-skylake-gemini-3h-2024-may-01.exe farm --reward-address --record-encoding-concurrency 4 path=H:\Subspace\farms_20,size=1000GB
2024-05-01T21:18:40.382750Z INFO subspace_farmer::commands::farm: Preparing plotting thread pools plotting_thread_pool_core_indices=[CpuCoreSet { cores: CpuSet(0-23), … }] replotting_thread_pool_core_indices=[CpuCoreSet { cores: CpuSet(0-11), … }]
2024-05-02T02:28:12.701999Z INFO {farm_index=0}: subspace_farmer::single_disk_farm::plotting: Plotting sector (4.27% complete) sector_index=40
2024-05-02T02:30:30.979817Z INFO {farm_index=0}: subspace_farmer::single_disk_farm::plotting: Plotting sector (4.38% complete) sector_index=41
memory allocation of memory allocation of memory allocation of memory allocation of memory allocation of
thread ’
thread '10487121048712104871210487121048712tokio-runtime-workertokio-runtime-worker bytes failed
bytes failed

PS C:\Subspace> .\subspace-farmer-windows-x86_64-skylake-gemini-3h-2024-may-01.exe farm path=H:\Subspace\farms_20,size=1000GB
2024-05-02T15:09:14.476366Z INFO subspace_farmer::commands::farm: Preparing plotting thread pools plotting_thread_pool_core_indices=[CpuCoreSet { cores: CpuSet(0-23), … }] replotting_thread_pool_core_indices=[CpuCoreSet { cores: CpuSet(0-11), … }]
2024-05-02T16:40:04.524814Z INFO {farm_index=0}: subspace_farmer::single_disk_farm::plotting: Plotting sector (5.77% complete) sector_index=54
memory allocation of 1048672 bytes failed

Environment

subspace-farmer-windows-x86_64-skylake-gemini-3h-2024-may-01.exe

Node-1
I7-13700/32GB/8TB SSD/WIN11 23H2

Node-2
I7-13700F/32GB/16TB SSD/WIN11 23H2

Node-3
Threadripper 2950X/128GB/32TB SSD/WIN11 23H2

Problem

Memory allocation failed during initial plotting on servers on Node-1 and Node-2.
Node-3 works fine.
Looks like a memory leak. When stoped plotting on Node-3 total memory consumpion drops from 59GB to 14GB.

2

Thanks so much for reporting this! @nazar-pc

3

I do not expect there to be a memory leak, but there might be a higher memory usage due to higher plotting concurrency in last releases.

How much memory does farmer use on i7 machines? Do you have anything else running there that can consume a lot of memory? I need more information.

4

I did some more research and found that memory keeps growing during plotting. On I7 machines farmer reached peak of 20Gb in 3 hours after start and then error appeared. On threadripper farmer reached 53GB after 10 hours and keep it even after plotting has finished (see screenshots).

Threadripper

farmer
farmer1159×609 63.5 KB

memory
memory1268×726 67.9 KB

My previous version was Mar-29 and everithing works fine. Peak memory consumtion for node + farmer was around 15Gb on each machine. Also I tried to decrise “–record-encoding-concurrency” from 4 to 3 but I don’t see that its helps to fix problem.

5

Got it, will take a closer look then

6

I initiated build from the mainbranch that has some tweaks, can you check if it helps with memory usage during plotting when it is built?: Snapshot build · subspace/subspace@3494834 · GitHub

It should also significantly reduce memory usage once plotting is over instead of keeping memory to itself.

7

Thanks! I will try it today and provide results.

UPD:
This build looks much better for me. After 4 hours plotting farmer consume ~14Gb and not growing. Total memory consumtion (win + node + farmer) on I7 system is about 25Gb, so errors are gone.

2 Likes
8

The change of test build is included in the latest release Release gemini-3h-2024-may-06 · subspace/subspace · GitHub

1 Like