Fake display of high RAM usage or RAM leak on Windows by Subspace farmer

Support
121

yeah build failed, will wait for update

122

Updated link to build that should finally succeed, sorry

123

I updated but I can not see issue nor changes, I am not monitoring memory issue in details so if it save few GB I would not know
No issue with rejected blocks also

ssd/nvme still paged at 97% controller load while plotting, but no issue with it (read speed is fine, the expected one for farming size)

124

Iā€™ve missed 2/4, and the 2 misses are at the SATA SSD (only 2TB, an enterprise grade Samsung SSD). Itā€™s weird this version, there is no log for prove time.

RAM usage is down to 50% of the last version, only 13GB/64GB.

2024-03-07T00:58:19.110361Z  INFO {disk_farm_index=3}: subspace_farmer::single_disk_farm::plotting: Plotting sector (85.27% complete) sector_index=1517
2024-03-07T00:58:19.191045Z  INFO {disk_farm_index=0}: subspace_farmer::single_disk_farm::plotting: Replotting sector (0.00% complete) sector_index=683
2024-03-07T00:58:34.745132Z  INFO subspace_farmer::farmer_cache: Synchronizing piece cache
2024-03-07T00:58:34.774952Z  INFO subspace_farmer::farmer_cache: Finished piece cache synchronization
2024-03-07T01:09:05.945196Z  INFO {disk_farm_index=7}: subspace_farmer::single_disk_farm::plotting: Plotting sector (41.59% complete) sector_index=1519
2024-03-07T01:12:32.120452Z  INFO {disk_farm_index=1}: subspace_farmer::single_disk_farm::plotting: Plotting sector (85.22% complete) sector_index=1516
2024-03-07T01:15:47.637842Z  INFO {disk_farm_index=4}: subspace_farmer::single_disk_farm::plotting: Plotting sector (92.02% complete) sector_index=1637
2024-03-07T01:19:08.635659Z  INFO {disk_farm_index=2}: subspace_farmer::single_disk_farm::plotting: Plotting sector (85.22% complete) sector_index=1516
2024-03-07T01:21:39.639887Z  WARN {disk_farm_index=6}: subspace_farmer::single_disk_farm::farming: Proving for solution skipped due to farming time limit slot=2916517 sector_index=303
2024-03-07T01:22:21.704218Z  INFO {disk_farm_index=5}: subspace_farmer::single_disk_farm::plotting: Plotting sector (85.22% complete) sector_index=1516
2024-03-07T01:25:34.693264Z  INFO {disk_farm_index=6}: subspace_farmer::single_disk_farm::plotting: Plotting sector (85.16% complete) sector_index=1515
2024-03-07T01:28:46.502120Z  INFO {disk_farm_index=3}: subspace_farmer::single_disk_farm::plotting: Plotting sector (85.33% complete) sector_index=1518
2024-03-07T01:31:20.957210Z  WARN {disk_farm_index=6}: subspace_farmer::single_disk_farm::farming: Failed to send solution to node, skipping further proving for this slot slot=2917098 sector_index=1040 error=RPC call failed: ErrorObject { code: ServerError(-32001), message: "Custom error: Solution was ignored", data: None }
2024-03-07T01:31:59.233284Z  INFO {disk_farm_index=0}: subspace_farmer::single_disk_farm::plotting: Replotting sector (4.00% complete) sector_index=853
2024-03-07T01:35:11.121763Z  INFO {disk_farm_index=7}: subspace_farmer::single_disk_farm::plotting: Plotting sector (41.62% complete) sector_index=1520
2024-03-07T01:38:26.327516Z  INFO {disk_farm_index=1}: subspace_farmer::single_disk_farm::plotting: Plotting sector (85.27% complete) sector_index=1517
2024-03-07T01:41:37.050834Z  INFO {disk_farm_index=4}: subspace_farmer::single_disk_farm::plotting: Plotting sector (92.07% complete) sector_index=1638
2024-03-07T01:44:52.530583Z  INFO {disk_farm_index=2}: subspace_farmer::single_disk_farm::plotting: Plotting sector (85.27% complete) sector_index=1517
2024-03-07T01:48:04.151042Z  INFO {disk_farm_index=5}: subspace_farmer::single_disk_farm::plotting: Plotting sector (85.27% complete) sector_index=1517
2024-03-07T01:51:18.974262Z  INFO {disk_farm_index=6}: subspace_farmer::single_disk_farm::plotting: Plotting sector (85.22% complete) sector_index=1516
2024-03-07T01:54:32.144641Z  INFO {disk_farm_index=3}: subspace_farmer::single_disk_farm::plotting: Plotting sector (85.39% complete) sector_index=1519
2024-03-07T01:57:43.948997Z  INFO {disk_farm_index=0}: subspace_farmer::single_disk_farm::plotting: Replotting sector (8.00% complete) sector_index=407
2024-03-07T01:58:21.501076Z  INFO {disk_farm_index=7}: subspace_farmer::reward_signing: Successfully signed reward hash 0x83333430b6f294e571093c4a7cfea1f632cc2029a51038e013432775c4b1b849
2024-03-07T02:01:00.487184Z  INFO {disk_farm_index=7}: subspace_farmer::single_disk_farm::plotting: Plotting sector (41.65% complete) sector_index=1521
2024-03-07T02:04:12.936125Z  INFO {disk_farm_index=1}: subspace_farmer::single_disk_farm::plotting: Plotting sector (85.33% complete) sector_index=1518
2024-03-07T02:07:24.671109Z  INFO {disk_farm_index=4}: subspace_farmer::single_disk_farm::plotting: Plotting sector (92.13% complete) sector_index=1639
2024-03-07T02:10:39.693247Z  INFO {disk_farm_index=2}: subspace_farmer::single_disk_farm::plotting: Plotting sector (85.33% complete) sector_index=1518
2024-03-07T02:13:51.823188Z  INFO {disk_farm_index=5}: subspace_farmer::single_disk_farm::plotting: Plotting sector (85.33% complete) sector_index=1518
2024-03-07T02:17:07.161238Z  INFO {disk_farm_index=6}: subspace_farmer::single_disk_farm::plotting: Plotting sector (85.27% complete) sector_index=1517
2024-03-07T02:20:20.046625Z  INFO {disk_farm_index=3}: subspace_farmer::single_disk_farm::plotting: Plotting sector (85.44% complete) sector_index=1520
2024-03-07T02:20:27.034483Z  INFO {disk_farm_index=7}: subspace_farmer::reward_signing: Successfully signed reward hash 0x8ba533792702e3a956553b506be33a86611768ae0cae5b831a1720017bc2a919
2024-03-07T02:23:34.705862Z  INFO {disk_farm_index=0}: subspace_farmer::single_disk_farm::plotting: Replotting sector (12.00% complete) sector_index=452
2024-03-07T02:26:44.022331Z  INFO {disk_farm_index=7}: subspace_farmer::single_disk_farm::plotting: Plotting sector (41.68% complete) sector_index=1522
2024-03-07T02:29:58.601899Z  INFO {disk_farm_index=1}: subspace_farmer::single_disk_farm::plotting: Plotting sector (85.39% complete) sector_index=1519
2024-03-07T02:33:10.031931Z  INFO {disk_farm_index=4}: subspace_farmer::single_disk_farm::plotting: Plotting sector (92.19% complete) sector_index=1640
2024-03-07T02:36:24.558207Z  INFO {disk_farm_index=2}: subspace_farmer::single_disk_farm::plotting: Plotting sector (85.39% complete) sector_index=1519
125

Memory usage reduction is the only thing that I expected to change in this version, especially when plotting is not yet finished, so looks like it was confirmed. Farming code remained the same as it was before.

The log was for debugging purpose, I removed it because it is not supposed to go into release version.

126

Do you folks happen to have more than one farm on a physical disk? Especially those that sometimes fail to prove in time?

127

I have 16 farms, I donā€™t think there is any of them that miss reward more often than others. Or I let any of them to have a high miss reward. I control this by lowering record concurrency, or even plot thread.

Iā€™m in asia, controlling the CPU temperature is my biggest challenge. I notice if we let the CPU too hot, it will miss reward more frequently.

So I have no answer for your question. But I tested the latest build (Snapshot build Ā· subspace/subspace@21de4db Ā· GitHub), and missed 2/4. Itā€™s super high. Actually it missed 2/2 on SATA disks, the other 2 win was on NVMe. On the same machine, if I run Feb 19 release, the reward missing rate is only 0.5% across all SATA/NVMe disks.

128

I believe you, but that is not the question I asked

129

I read your question again, seems I misunderstand.

So your question is whether I have 2/3/4 farms on a physical disk, e.g. 4 x 2TB farms on 8TB disk. No, I donā€™t.

But I know Qwinn, he has 8 x 2TB farms on his 16TB disk (this is an NVMe U2 disk), it works well. He never miss any reward.

130

Okay, if you have issues with SATA devices, how to you connect them? Are they connected through motherboardā€™s SATA ports or you use some expansion card or port multiplexer? Would be great if you can provide exact models of such cards if you do use them.

I suspect you might overwhelm SATA controller or something, but I have ideas on how to potentially address that too.

131

On the machine I used to test, I have 3 x Samsung PM883 (MZ7LH1T9HMLT) and each of them connect directly to the Asrock Z690 chipset mobo via SATA port. Both the mobo/chipset and the SATA disk are pretty good ones, I believe.

1 Like
132

Got it, I think this thread about memory usage can be considered resolved (at least for now), Iā€™ll look into reward misses more closely in the near future as well.

1 Like
133

Fix for memory usage was shipped in Release gemini-3h-2024-mar-08 Ā· subspace/subspace Ā· GitHub and Release 0.1.6 Ā· subspace/space-acres Ā· GitHub