My farms are newly created, I waited until node was synced to start them. I see the line for “Synchronizing cache” in the farmer log but I never see the line for “Finished cache initialization” (on 3 separate farmers on 3 separate servers), and my plotting time is crazy slow, first sectors took 3 hours each and after 16 hours or so I’m getting one every 40 minutes and it’s no longer speeding up.
Things I have already tried:
Restarting node with --db-cache 4096
Disabled swap
Turned off all farmers except the one local to the node
Ran scrub on the farm (passed)
Still never get “finished cache initialization”
BTW: When I was first syncing the node, I started the farmer and let it go for a while despite the fact that it was waiting for node sync, and that one time I noticed I did get the finished cache initialization notice after 3 hours. But I also got a notice that a plot was complete before the node was done syncing, which troubled me, so I wiped and restarted the farm. Perhaps it can only finish cache initialization when it is also not plotting, but I would really like to find a solution to this that does not require me to restart the node from scratch (another 2 days), and I don’t think it should’ve taken 3 hours even in that circumstance anyway.
(note - I only recently added the --execution wasm in an attempt to fix the problem, but initially I didn’t have it as it is not in the current advanced cli documentation. It didn’t help.)
Farmer string just has the --reward-address and a single path set
Commands were previously a little more complicated than this with farmers on other servers remote to the node, requiring a few rpc switches, but I’ve shut all those off and simplified as much as possible hoping to resolve, nothing helps.
BTW: When I was first syncing the node, I started the farmer and let it go for a while despite the fact that it was waiting for node sync, and that one time I noticed I did get the finished cache initialization notice after 3 hours. But I also got a notice that a plot was complete before the node was done syncing, which troubled me, so I wiped and restarted the farm.
Thank you for sharing that information. Just to clarify, the Farmer is designed to begin plotting even while the node is syncing. So, it’s not unusual for the farmer to start and pause its plotting process multiple times until the node fully syncs. If you notice the farmer completing its tasks before the node has finished syncing, there’s no need to be concerned or restart. It’s best to let it continue and complete on its own.
In regards to the core issue you are facing, I know you mentioned you are never getting “Finished cache initialization” but are you seeing any other logs related to piece cache other than it starting?
Hello. Good to know that plot outside of node sync wasn’t unexpected behavior. Wish I hadn’t wiped, heh.
Here’s what I have from my latest restart of the farmer local to the node, with the observer added lines removed. (Also removing DNS listening lines as they have IP info, tell me if you need them)
2023-10-30T17:27:16.704496Z INFO subspace_farmer::commands::farm: Collecting already plotted pieces (this will take some time)…
2023-10-30T17:27:16.704555Z INFO subspace_farmer::piece_cache: Initializing piece cache
2023-10-30T17:27:16.714274Z INFO subspace_farmer::commands::farm: Finished collecting already plotted pieces successfully
2023-10-30T17:27:16.715167Z INFO single_disk_farm{disk_farm_index=0}: subspace_farmer::single_disk_farm: Subscribing to slot info notifications
2023-10-30T17:27:16.715251Z INFO single_disk_farm{disk_farm_index=0}: subspace_farmer::reward_signing: Subscribing to reward signing notifications
2023-10-30T17:27:16.717749Z INFO single_disk_farm{disk_farm_index=0}: subspace_farmer::single_disk_farm::plotting: Subscribing to archived segments
2023-10-30T17:27:16.718073Z INFO single_disk_farm{disk_farm_index=0}: subspace_farmer::single_disk_farm::plotting: Node is not synced yet, pausing plotting until sync status changes
2023-10-30T17:27:43.577794Z INFO subspace_networking::node_runner: Public address status changed. old=Unknown new=Private
2023-10-30T17:27:43.942856Z INFO subspace_farmer::piece_cache: Synchronizing cache
2023-10-30T17:32:12.763090Z INFO single_disk_farm{disk_farm_index=0}: subspace_farmer::single_disk_farm::plotting: Node is synced, resuming plotting
2023-10-30T18:28:17.970098Z INFO single_disk_farm{disk_farm_index=0}: subspace_farmer::single_disk_farm::plotting: Sector plotted successfully (0.55%) sector_index=18
2023-10-30T19:17:11.346401Z INFO single_disk_farm{disk_farm_index=0}: subspace_farmer::single_disk_farm::plotting: Sector plotted successfully (0.57%) sector_index=19
As you can see, sectors are still taking about 40-50 minutes to plot, and no line about initializing cache.
FYI, longest I’ve let a farmer run where I never got the finished initialization line was about 16 hours. I’m not going to do any more restarts until you guys tell me to (and do let me know if I should just move to 3g when that drops)
Morning update: Plot times went back up to around 1h30m around midnight but then started to improve. By morning they’re between 20-30 minutes each, which is… better. All 3 boxes have signed at least 1 reward hash. Still never seen the cache initialized line on any of the logs.
3g addresses this issue by first populating cache and only then starting farming. This way you have a higher chance of local cache hit and plotting speed should greatly improve.
Please note that in 3g unless you start the farmer with --farm-during-initial-plotting, farming will only start after initial plotting is complete. This makes plotting faster, but if you strongly prefer to try to farm while plotting you can try it as well (you might see that both plotting is slower and farming is not very successful at the same time, which is why it is not the default).
My 7T SSD is estimated to take 15 days to complete plotting on 3F. At the current speed on 3g, it requires 90 days. By then, if the testnet ends, I might not even finish the plotting for mining.
That seems really odd. I’m estimating I’ll be finished plotting a 4TB drive in 12 days on 3g. Plots a sector every 4m45s, very consistently (Ubuntu, 10900 CPU). Another farm (Windows, 7950x) plots every 3m10s, should be done with 2TB in 4 days.
I’d open another thread to report that though, as the topic for this one is no longer applicable.
Hey, brother, I also encountered the same problem on gemini-3h. The plotting sector is very slow. It takes nearly an hour to complete one sector. How long does it take for your 1TB SSD to complete the plotting sector?
For details, you can see what I mentioned