CPU (re)plotting hangs, a restart solves the symptom

Issue Report

Environment

  • Operating System: Happens on Debian, Ubuntu, and I believe (based on other reports) Windows.
  • Space Acres/Advanced CLI/Docker: Advanced CLI

Problem

The farmer stops (re)plotting, although it should (re)plot (progress hasn’t reached 100%). (It is not predictable when. Usually after a few hours, sometimes after days).

If a farmer stops plotting in this way he still does reward signing. Here is an example:

2024-11-15T03:41:07.586355Z  INFO {farm_index=1}:{sector_index=144}: subspace_farmer::single_disk_farm::plotting: Replotting sector (1.38% complete)
2024-11-15T03:44:55.221300Z  INFO {farm_index=1}:{sector_index=261}: subspace_farmer::single_disk_farm::plotting: Replotting sector (1.61% complete)
2024-11-15T03:46:10.821002Z  INFO {farm_index=0}:{sector_index=365}: subspace_farmer::single_disk_farm::plotting: Replotting sector (1.25% complete)
2024-11-15T03:51:01.556366Z  INFO {farm_index=0}:{sector_index=190}: subspace_farmer::single_disk_farm::plotting: Replotting sector (1.46% complete)
2024-11-15T04:30:48.531536Z  INFO {farm_index=1}: subspace_farmer::single_disk_farm::reward_signing: Successfully signed reward hash 0x3956c8ac6f239111ae886cdddf99ae0694595ccaf02ba543e059cb41c67fdcbc

However, reward signing isn’t a requirement for this issue:

2024-11-13T19:32:42.806799Z  INFO {farm_index=0}:{sector_index=1998}: subspace_farmer::single_disk_farm::plotting: Plotting sector (97.04% complete)
2024-11-13T19:35:02.836240Z  INFO {farm_index=1}:{sector_index=33}: subspace_farmer::single_disk_farm::plotting: Replotting sector (0.18% complete)
2024-11-13T19:37:11.205735Z  INFO {farm_index=1}:{sector_index=131}: subspace_farmer::single_disk_farm::plotting: Replotting sector (0.35% complete)
2024-11-13T19:40:20.354073Z  INFO {farm_index=0}:{sector_index=1999}: subspace_farmer::single_disk_farm::plotting: Plotting sector (97.09% complete)
2024-11-13T20:02:18.660366Z  INFO {farm_index=0}:{sector_index=2000}: subspace_farmer::single_disk_farm::plotting: Plotting sector (97.13% complete)

The last log entry was 45 minutes old, so nothing happened for 45 minutes after the last entry. When a farmer is restarted in this state, it almost always starts (re)plotting successfully within 2 minutes.

We have checked (almost(?)) all possible reasons, like network connection/port forwarding, resources, cpu and memory usage, etc. See here for more details, command line parameters, logs and screenshots: farmer-support | Autonomys Network on Discord.

Expected behavior

The farmer will plot at least as continuously as when the farmer is periodically restarted when the system load becomes too low.

Sorry,

I didn’t know you already knew:

No worries, thanks for creating forum post.
I’ll update here once I have a potential fix.

Here is a test build that should address piece cache sync speed and downloading speeds in general: Snapshot build · autonomys/subspace@d303125 · GitHub

Let me know if it helps.

Should be fixed in Release mainnet-2024-nov-18 · autonomys/subspace · GitHub and Release 0.2.4 · autonomys/space-acres · GitHub

1 Like