Hey all, so per my previous post I was struggling to get one of my node/farmer pairs past the “salt recommitment” phase that occurs not long after the node finishes its initial syncing/plotting. It’s a 600GB farm located on an external 2.0 TB HDD (USB 3.0).
Symptoms: When salt recommitment starts, multiple threads simultaneously try to access the HDD causing it to thrash. The sustained read throughput was about 20 MB/sec (according to iostat). Unfortunately this created an I/O bottleneck for the accompanying node which could no longer stay in sync (Waiting for farmer to receive and acknowledge archived segment). This caused the node to buffer up blocks in RAM while waiting for I/O. Eventually the machine runs out of RAM and the process crashed.
I tried an overnight experiment: Let salt recommitment start and as soon as it does, bring down the accompanying node (docker container stop node). Sure enough after about 6 hours, the salt recommitment process completed. I just brought the node back up and the farmer began slowly creating new plot segments. It’s currently about 6000 blocks behind the top of the chain and syncing very slowly. I guess we’ll see where this goes…