We had an issue originally created as a response to Can set up farm to connect to specified farm on internal network? that suggested a hierarchical architecture of connecting multiple farmers together. After thinking about it more, I realized we should probably decompose previously monolithic farme…

Hard to answer without knowing what machines you have, what networking you have between them, etc. This thread is already quite long, please create a separate topic and describe all the machines you have, networking between them and what components each of them is running.

System Specs Alpha has a 12c Ryzen and runs only node, cache and controller + nats. Farmers are all 7950X or HP Z840s with a v4 Xeon. Errors Controller 2024-06-03T12:57:01.149958Z INFO async_nats: event: slow consumers for subscription 1 The above messages was getting spammed until I restarte…

I have detailed logs with a similar issue, I’ll ping you if more information is needed

Turns out I can benefit from logs with RUST_LOG=info,subspace_farmer=trace, thanks!

It took about two days to replicate the slow consumers for subscription 1 errors. I rotated the logs and was able to capture it with just under one day of logs. I then realized I only had the trace on the controller, cache, and plotter. (Standard logging on the farmers). The logs are pretty good siz…

I’ll pull that build and turn on trace logging for all the cluster components.

I see with your logs where the issue is. It basically appears during higher plotting concurrency. It will eventually time out and resolve itself, but I’ll think how to handle it more gracefully still. If you have a lot of plotting capacity you should consider running more than one controller with a…

Okay, I had Constrain farmer piece getter concurrency by nazar-pc · Pull Request #2830 · subspace/subspace · GitHub hanging in a branch for some time and wasn’t sure if it’ll be helpful or not, but I think it will be and will give you more flexibility if you decide to add even more plotting capacity…

[image] nazar-pc: If you have a lot of plotting capacity you should consider running more than one controller with additional cache to spread the load across multiple instances so plotters are not so bottlenecked. I can do that, but currently (and in those logs) I am only running a single plo…

Farming cluster

Research

nazar-pc June 7, 2024, 10:00am 83

I’m wondering if you’ll be able to reproduce it with There are too many warning logs in the farmer cluster - #22 by duanyz_aiyo

Topic		Replies	Views
Farming cluster documents Support	12	309	May 23, 2024
Abnormal plotting progress in farmer cluster Support	21	354	June 27, 2024
Cluster farmer timeout while plotting Support nodes , cli , farmer	1	42	June 19, 2025
Farmer jun 11 stop farming after replot start sometimes Support	33	437	June 19, 2024
Farmer stops signing rewards once it starts plotting Incentivized Testnet gemini , cli	22	686	September 17, 2023

Farming cluster

Related topics