Farmer bloats to 32GB RAM and then stops

Support
1

I was able to sync with a plot size of 4T successfully but now I can’t get the farmer to work, it just gorges itself with 32 GB of RAM and then throws the following in the logs.

2022-06-08T10:03:37.063871Z  INFO subspace_farmer::farming: Salt will be updated, recommitting in background next_salt=79f25142c329a0d0
2022-06-08T10:03:37.063982Z  INFO subspace_farmer::farming: Salt will be updated, recommitting in background next_salt=79f25142c329a0d0
2022-06-08T10:03:37.063860Z  INFO subspace_farmer::farming: Salt updated, recommitting in background new_salt=0000000000000000
2022-06-08T10:03:37.064083Z  INFO subspace_farmer::farming: Finished recommitment salt=0000000000000000 took_seconds=0.00022365500626619905
2022-06-08T10:03:37.064096Z  INFO subspace_farmer::farming: Salt will be updated, recommitting in background next_salt=79f25142c329a0d0
2022-06-08T10:03:37.063821Z  INFO subspace_farmer::farming: Salt updated, recommitting in background new_salt=0000000000000000
2022-06-08T10:03:37.064154Z  INFO subspace_farmer::farming: Finished recommitment salt=0000000000000000 took_seconds=0.0003373810031916946
2022-06-08T10:03:37.064169Z  INFO subspace_farmer::farming: Salt will be updated, recommitting in background next_salt=79f25142c329a0d0
2022-06-08T10:07:33.115613Z  INFO subspace_farmer::farming: Successfully signed reward hash 0x845398e1de9f515d093104041bdb2bf9c4422ac12d1b14e792ac5be3b8da0cba
2022-06-08T10:17:06.180733Z  INFO subspace_farmer::farming: Successfully signed reward hash 0xd887d366200e97dbfb54791bc226ee3bac98aee0eb7f2470c3e97e6739d593b6
2022-06-08T10:17:24.130810Z  INFO subspace_farmer::farming: Successfully signed reward hash 0x9ffcf707e15974a3e4ccb09f0ab93ca8bbcebd2bea0e761c8f1166ed530146d8
2022-06-08T10:17:38.088642Z  INFO subspace_farmer::farming: Successfully signed reward hash 0xc2f3c670be7b7772f5dfb614347a9f7b701de471c97c82f15e53ff27bbaeeef0
2022-06-08T10:26:24.186605Z  INFO subspace_farmer::farming: Successfully signed reward hash 0x8afdcb308c89f0a83c4df847a5c316063157c2f230aa18511ba05ac3ec3480cb
2022-06-08T10:29:26.594383Z  INFO subspace_farmer::farming: Successfully signed reward hash 0x7e922570dba13a42ab9135184cf62c256bc6a30bfd3f57d6dd39dcfc5d2b5aca
2022-06-08T11:29:06.364433Z ERROR jsonrpsee_core::client::async_client::helpers: Dropping subscription Str("JiQCCiUl0HQbdhhc") error: TrySendError { kind: Full }

This is happening every time I start the farmer. I’m using subspace-farmer-ubuntu-x86_64-gemini-1b-2022-june-05 on Ubuntu 20.04.4 LTS

Thanks for your help

2

Somehow it working fine in the last couple of hours. I’ll post here when this happens again.

3

I’ve seen the RAM usage go up like when the following happens:

  • Node completes sync and initial plot complete
  • After sync completes, the farmer begins the salt update process
  • The salt update process requires a lot of I/O throughput. If the farm is on a single HDD, it will start chugging because it’s trying to keep in sync and update salt at the same time (my guess, just an observation)
  • The farm starts to fall behind causing memory usage to increase steadily.

How were you able to sync a 4T node? How long did this take? Single HDD?

4

Thanks teslak. Yes I think the instances I saw where after the salt update and I do see the node getting behind during the updates.

It took me about 3 days to plot the 4T, with some idle time in the middle where syncing wasn’t progressing. It’s my old chia HDD plotting machine with 10x 10K RPM HDDs in RAID0.

5

Thanks! So yeah, a 10x 10K RPM RAID array would seem to be fast enough, at least eventually.

See my other post about syncing a 2TB node on an SSD - it was one of my leftover Chia plotting SSDs:

I’m really skeptical that a single cheap external USB HDD has enough IOPs and throughput to complete plot/sync/salt update.

6

Hopefully this issue should be mitigated with the latest release.

Here is the PR: Recommitments improvements by nazar-pc · Pull Request #575 · subspace/subspace · GitHub