We are testing subspace new feature - farming cluster.
Can Dev team share its design docs or videos helping to better understand how it work inside? such as the detail about how controller, cache, plotter, farmer work together with NATs.
(I have trouble to access discord)
Have you checked here? Farming cluster - #44 by nazar-pc there is a lot of info from others testing it.
Yes, I have checked it out but it is not well organized for a new feature.
That feature has not shipped yet, you need to understand that it is early preview and you’re expected to do some reading and experimentation before full proper documentation arrives.
Farming cluster farmer Log shows “Timed out without ping from plotter” frequently.
The plotting never completes (regular farmer takes 200 seconds to plot 1 sector).
./subspace-farmer cluster --nats-server nats://192.168.0.2:4222 farmer --reward-address st9XXXX path=/home/fact/subspace/farm,size=300GiB
2024-05-13T22:40:38.948482Z INFO async_nats: event: connected
2024-05-13T22:40:38.948539Z INFO async_nats: event: connected
2024-05-13T22:40:38.948888Z INFO async_nats: event: connected
2024-05-13T22:40:38.948913Z INFO async_nats: event: connected
2024-05-13T22:40:38.949089Z INFO async_nats: event: connected
2024-05-13T22:40:38.949137Z INFO async_nats: event: connected
2024-05-13T22:40:38.949154Z INFO async_nats: event: connected
2024-05-13T22:40:38.949664Z INFO async_nats: event: connected
2024-05-13T22:40:40.047791Z INFO {farm_index=0}: subspace_farmer::single_disk_farm::plot_cache: Checking plot cache contents, this can take a while
2024-05-13T22:40:40.050085Z INFO {farm_index=0}: subspace_farmer::single_disk_farm::plot_cache: Finished checking plot cache contents
2024-05-13T22:40:40.051916Z INFO {farm_index=0}: subspace_farmer::single_disk_farm: Benchmarking faster proving method
2024-05-13T22:40:42.773150Z INFO {farm_index=0}: subspace_farmer::single_disk_farm: Faster proving method found fastest_mode=ConcurrentChunks
2024-05-13T22:40:42.836397Z INFO {farm_index=0}: subspace_farmer::commands::cluster::farmer: Farm 0:
2024-05-13T22:40:42.836422Z INFO {farm_index=0}: subspace_farmer::commands::cluster::farmer: ID: 01HSSQNHR8ZDTM9G0MWXE0DYD6
2024-05-13T22:40:42.836428Z INFO {farm_index=0}: subspace_farmer::commands::cluster::farmer: Genesis hash: 0x0c121c75f4ef450f40619e1fca9d1e8e7fbabc42c895bc4790801e85d5a91c34
2024-05-13T22:40:42.836437Z INFO {farm_index=0}: subspace_farmer::commands::cluster::farmer: Public key: 0xccc9f0d7b7cdd905d3d6f4ea0cbbecfe851be6f124d8119670fcf91b34acea6b
2024-05-13T22:40:42.836450Z INFO {farm_index=0}: subspace_farmer::commands::cluster::farmer: Allocated space: 300.0 GiB (322.1 GB)
2024-05-13T22:40:42.836455Z INFO {farm_index=0}: subspace_farmer::commands::cluster::farmer: Directory: /home/fact/subspace/farm
2024-05-13T22:40:42.920249Z INFO {farm_index=0}: subspace_farmer::single_disk_farm::farming: Subscribing to slot info notifications
2024-05-13T22:40:42.920283Z INFO {farm_index=0}: subspace_farmer::reward_signing: Subscribing to reward signing notifications
2024-05-13T22:40:42.923447Z INFO {farm_index=0}: subspace_farmer::single_disk_farm::plotting: Subscribing to archived segments
2024-05-13T22:40:42.926821Z INFO {farm_index=0}: subspace_farmer::single_disk_farm::plotting: Plotting sector (99.01% complete) sector_index=301
2024-05-13T22:40:42.928840Z INFO {farm_index=0}: subspace_farmer::single_disk_farm::plotting: Plotting sector (99.34% complete) sector_index=302
2024-05-13T22:40:42.930768Z INFO {farm_index=0}: subspace_farmer::single_disk_farm::plotting: Plotting sector (99.67% complete) sector_index=303
2024-05-13T22:40:52.928272Z WARN {farm_index=0}: subspace_farmer::single_disk_farm::plotting: Failed to plot sector, retrying in 1s sector_index=301 error=Low-level plotting error: Timed out without ping from plotter
2024-05-13T22:40:52.931972Z WARN {farm_index=0}: subspace_farmer::single_disk_farm::plotting: Failed to plot sector, retrying in 1s sector_index=303 error=Low-level plotting error: Timed out without ping from plotter
2024-05-13T22:40:53.930086Z INFO {farm_index=0}: subspace_farmer::single_disk_farm::plotting: Plotting sector retry sector_index=301
2024-05-13T22:40:53.933836Z INFO {farm_index=0}: subspace_farmer::single_disk_farm::plotting: Plotting sector retry sector_index=303
2024-05-13T22:41:03.935659Z WARN {farm_index=0}: subspace_farmer::single_disk_farm::plotting: Failed to plot sector, retrying in 1s sector_index=303 error=Low-level plotting error: Timed out without ping from plotter
2024-05-13T22:41:05.256981Z INFO {farm_index=0}: subspace_farmer::single_disk_farm::plotting: Plotting sector retry sector_index=303
2024-05-13T22:41:15.258222Z WARN {farm_index=0}: subspace_farmer::single_disk_farm::plotting: Failed to plot sector, retrying in 1s sector_index=303 error=Low-level plotting error: Timed out without ping from plotter
2024-05-13T22:41:16.260708Z INFO {farm_index=0}: subspace_farmer::single_disk_farm::plotting: Plotting sector retry sector_index=303
2024-05-13T22:41:26.261824Z WARN {farm_index=0}: subspace_farmer::single_disk_farm::plotting: Failed to plot sector, retrying in 1s sector_index=303 error=Low-level plotting error: Timed out without ping from plotter
2024-05-13T22:41:27.264539Z INFO {farm_index=0}: subspace_farmer::single_disk_farm::plotting: Plotting sector retry sector_index=303
2024-05-13T22:41:37.265542Z WARN {farm_index=0}: subspace_farmer::single_disk_farm::plotting: Failed to plot sector, retrying in 1s sector_index=303 error=Low-level plotting error: Timed out without ping from plotter
2024-05-13T22:41:38.267672Z INFO {farm_index=0}: subspace_farmer::single_disk_farm::plotting: Plotting sector retry sector_index=303
2024-05-13T22:41:42.957202Z WARN subspace_farmer::cluster::nats_client: Acknowledgement wait timed out request_type=subspace_farmer::cluster::farmer::ClusterFarmerPlottedSectorsRequest response_type=core::result::Result<subspace_farmer_components::plotting::PlottedSector, alloc::string::String>
2024-05-13T22:41:43.042476Z WARN subspace_farmer::cluster::nats_client: Acknowledgement wait timed out request_type=subspace_farmer::cluster::farmer::ClusterFarmerPlottedSectorsRequest response_type=core::result::Result<subspace_farmer_components::plotting::PlottedSector, alloc::string::String>
2024-05-13T22:41:48.269453Z WARN {farm_index=0}: subspace_farmer::single_disk_farm::plotting: Failed to plot sector, retrying in 1s sector_index=303 error=Low-level plotting error: Timed out without ping from plotter
2024-05-13T22:41:49.272713Z INFO {farm_index=0}: subspace_farmer::single_disk_farm::plotting: Plotting sector retry sector_index=303
2024-05-13T22:41:59.274503Z WARN {farm_index=0}: subspace_farmer::single_disk_farm::plotting: Failed to plot sector, retrying in 1s sector_index=303 error=Low-level plotting error: Timed out without ping from plotter
2024-05-13T22:42:00.276414Z INFO {farm_index=0}: subspace_farmer::single_disk_farm::plotting: Plotting sector retry sector_index=303
2024-05-13T22:42:10.277172Z WARN {farm_index=0}: subspace_farmer::single_disk_farm::plotting: Failed to plot sector, retrying in 1s sector_index=303 error=Low-level plotting error: Timed out without ping from plotter
2024-05-13T22:42:11.279396Z INFO {farm_index=0}: subspace_farmer::single_disk_farm::plotting: Plotting sector retry sector_index=303
2024-05-13T22:42:21.280389Z WARN {farm_index=0}: subspace_farmer::single_disk_farm::plotting: Failed to plot sector, retrying in 1s sector_index=303 error=Low-level plotting error: Timed out without ping from plotter
2024-05-13T22:42:22.285920Z INFO {farm_index=0}: subspace_farmer::single_disk_farm::plotting: Plotting sector retry sector_index=303
2024-05-13T22:42:32.287669Z WARN {farm_index=0}: subspace_farmer::single_disk_farm::plotting: Failed to plot sector, retrying in 1s sector_index=303 error=Low-level plotting error: Timed out without ping from plotter
2024-05-13T22:42:33.291103Z INFO {farm_index=0}: subspace_farmer::single_disk_farm::plotting: Plotting sector retry sector_index=303
Try latest build: Farming cluster - #53 by nazar-pc
It should work much better.
What hardware are you using for each farming cluster component?
ARM64 8 cores，12GB DDR，Ubuntu 18.04
Hi Nazar, could you please send me some video or document links about the subspace-farmer (regular farmer not cluster)? I am lost in subspace-farmer RUST code.
Is it the same as latest build: git pull the latest code in farming-cluster branch, and build it with cargo build --profile production --bin subspace-farmer?
What ARM64 cores exactly, do you have a specific model or name of the SBC/server you’re using? Are you running all cluster components on the same machine?
https://docs.subspace.network/ is what you want to read. No need to read Rust code. If you want code docs then Index of crates is probably where you should look, but some of the documentation might be outdated.
I build and deployed the latest farming-cluster branch code. and run on my another x64 server, also report timeout warnings.
NATs runs on other server, while controller, cache, plotter, farmer run on this x64 server.
x64 server cpu info:
Architecture: x86_64
CPU op-mode(s): 32-bit, 64-bit
Address sizes: 39 bits physical, 48 bits virtual
Byte Order: Little Endian
CPU(s): 12
On-line CPU(s) list: 0-11
Vendor ID: GenuineIntel
Model name: 12th Gen Intel(R) Core(TM) i5-12400F
CPU family: 6
Model: 151
Thread(s) per core: 2
Core(s) per socket: 6
Socket(s): 1
Stepping: 5
CPU max MHz: 4400.0000
CPU min MHz: 800.0000
BogoMIPS: 4992.00
Flags: fpu vme de pse tsc msr pae mce cx8 apic sep mtrr pge mca cm
ov pat pse36 clflush dts acpi mmx fxsr sse sse2 ss ht tm pb
e syscall nx pdpe1gb rdtscp lm constant_tsc art arch_perfmo
n pebs bts rep_good nopl xtopology nonstop_tsc cpuid aperfm
perf tsc_known_freq pni pclmulqdq dtes64 monitor ds_cpl vmx
est tm2 ssse3 sdbg fma cx16 xtpr pdcm sse4_1 sse4_2 x2apic
movbe popcnt tsc_deadline_timer aes xsave avx f16c rdrand
lahf_lm abm 3dnowprefetch cpuid_fault ssbd ibrs ibpb stibp
ibrs_enhanced tpr_shadow flexpriority ept vpid ept_ad fsgsb
ase tsc_adjust bmi1 avx2 smep bmi2 erms invpcid rdseed adx
smap clflushopt clwb intel_pt sha_ni xsaveopt xsavec xgetbv
1 xsaves split_lock_detect avx_vnni dtherm ida arat pln pts
hwp hwp_notify hwp_act_window hwp_epp hwp_pkg_req hfi vnmi
umip pku ospke waitpkg gfni vaes vpclmulqdq rdpid movdiri
movdir64b fsrm md_clear serialize arch_lbr ibt flush_l1d ar
ch_capabilities
Virtualization features:
Virtualization: VT-x
Caches (sum of all):
L1d: 288 KiB (6 instances)
L1i: 192 KiB (6 instances)
L2: 7.5 MiB (6 instances)
L3: 18 MiB (1 instance)
NUMA:
NUMA node(s): 1
NUMA node0 CPU(s): 0-11
Vulnerabilities:
Gather data sampling: Not affected
Itlb multihit: Not affected
L1tf: Not affected
Mds: Not affected
Meltdown: Not affected
Mmio stale data: Not affected
Retbleed: Not affected
Spec rstack overflow: Not affected
Spec store bypass: Mitigation; Speculative Store Bypass disabled via prctl
Spectre v1: Mitigation; usercopy/swapgs barriers and __user pointer san
itization
Spectre v2: Mitigation; Enhanced / Automatic IBRS, IBPB conditional, RS
B filling, PBRSB-eIBRS SW sequence
Srbds: Not affected
Tsx async abort: Not affected
Latest code log:
2024-05-14T13:00:20.777302Z INFO async_nats: event: connected
2024-05-14T13:00:20.777354Z INFO async_nats: event: connected
2024-05-14T13:00:20.777376Z INFO async_nats: event: connected
2024-05-14T13:00:20.777386Z INFO async_nats: event: connected
2024-05-14T13:00:20.777391Z INFO async_nats: event: connected
2024-05-14T13:00:20.777394Z INFO async_nats: event: connected
2024-05-14T13:00:20.777405Z INFO async_nats: event: connected
2024-05-14T13:00:20.777549Z INFO async_nats: event: connected
2024-05-14T13:00:21.827205Z INFO {farm_index=0}: subspace_farmer::single_disk_farm::plot_cache: Checking plot cache contents, this can take a while
2024-05-14T13:00:21.829385Z INFO {farm_index=0}: subspace_farmer::single_disk_farm::plot_cache: Finished checking plot cache contents
2024-05-14T13:00:21.830731Z INFO {farm_index=0}: subspace_farmer::single_disk_farm: Benchmarking faster proving method
2024-05-14T13:00:24.476067Z INFO {farm_index=0}: subspace_farmer::single_disk_farm: Faster proving method found fastest_mode=ConcurrentChunks
2024-05-14T13:00:24.530805Z INFO {farm_index=0}: subspace_farmer::commands::cluster::farmer: Farm 0:
2024-05-14T13:00:24.530833Z INFO {farm_index=0}: subspace_farmer::commands::cluster::farmer: ID: 01HSSQNHR8ZDTM9G0MWXE0DYD6
2024-05-14T13:00:24.530839Z INFO {farm_index=0}: subspace_farmer::commands::cluster::farmer: Genesis hash: 0x0c121c75f4ef450f40619e1fca9d1e8e7fbabc42c895bc4790801e85d5a91c34
2024-05-14T13:00:24.530846Z INFO {farm_index=0}: subspace_farmer::commands::cluster::farmer: Public key: 0xccc9f0d7b7cdd905d3d6f4ea0cbbecfe851be6f124d8119670fcf91b34acea6b
2024-05-14T13:00:24.530857Z INFO {farm_index=0}: subspace_farmer::commands::cluster::farmer: Allocated space: 300.0 GiB (322.1 GB)
2024-05-14T13:00:24.530862Z INFO {farm_index=0}: subspace_farmer::commands::cluster::farmer: Directory: /home/fact/subspace/farm
2024-05-14T13:00:24.613823Z INFO {farm_index=0}: subspace_farmer::single_disk_farm::farming: Subscribing to slot info notifications
2024-05-14T13:00:24.613856Z INFO {farm_index=0}: subspace_farmer::reward_signing: Subscribing to reward signing notifications
2024-05-14T13:00:24.615787Z INFO {farm_index=0}: subspace_farmer::single_disk_farm::plotting: Subscribing to archived segments
2024-05-14T13:00:24.617945Z INFO {farm_index=0}: subspace_farmer::single_disk_farm::plotting: Plotting sector (99.01% complete) sector_index=301
2024-05-14T13:00:24.620497Z INFO {farm_index=0}: subspace_farmer::single_disk_farm::plotting: Plotting sector (99.34% complete) sector_index=302
2024-05-14T13:00:24.622425Z INFO {farm_index=0}: subspace_farmer::single_disk_farm::plotting: Plotting sector (99.67% complete) sector_index=303
2024-05-14T13:00:34.619383Z WARN {farm_index=0}: subspace_farmer::single_disk_farm::plotting: Failed to plot sector, retrying in 1s sector_index=301 error=Low-level plotting error: Timed out without ping from plotter
2024-05-14T13:00:34.621701Z WARN {farm_index=0}: subspace_farmer::single_disk_farm::plotting: Failed to plot sector, retrying in 1s sector_index=302 error=Low-level plotting error: Timed out without ping from plotter
2024-05-14T13:00:34.624077Z WARN {farm_index=0}: subspace_farmer::single_disk_farm::plotting: Failed to plot sector, retrying in 1s sector_index=303 error=Low-level plotting error: Timed out without ping from plotter
2024-05-14T13:00:35.622328Z INFO {farm_index=0}:{public_key=ccc9f0d7b7cdd905d3d6f4ea0cbbecfe851be6f124d8119670fcf91b34acea6b sector_index=301}:{public_key=ccc9f0d7b7cdd905d3d6f4ea0cbbecfe851be6f124d8119670fcf91b34acea6b sector_index=302}: subspace_farmer::single_disk_farm::plotting: Plotting sector retry sector_index=301
2024-05-14T13:00:35.622986Z INFO {farm_index=0}: subspace_farmer::single_disk_farm::plotting: Plotting sector retry sector_index=302
2024-05-14T13:00:35.626273Z INFO {farm_index=0}: subspace_farmer::single_disk_farm::plotting: Plotting sector retry sector_index=303
2024-05-14T13:00:45.623723Z WARN {farm_index=0}: subspace_farmer::single_disk_farm::plotting: Failed to plot sector, retrying in 1s sector_index=301 error=Low-level plotting error: Timed out without ping from plotter
2024-05-14T13:00:45.624023Z WARN {farm_index=0}: subspace_farmer::single_disk_farm::plotting: Failed to plot sector, retrying in 1s sector_index=302 error=Low-level plotting error: Timed out without ping from plotter
2024-05-14T13:00:45.627370Z WARN {farm_index=0}: subspace_farmer::single_disk_farm::plotting: Failed to plot sector, retrying in 1s sector_index=303 error=Low-level plotting error: Timed out without ping from plotter
2024-05-14T13:00:46.626285Z INFO {farm_index=0}:{public_key=ccc9f0d7b7cdd905d3d6f4ea0cbbecfe851be6f124d8119670fcf91b34acea6b sector_index=301}: subspace_farmer::single_disk_farm::plotting: Plotting sector retry sector_index=302
2024-05-14T13:00:46.626439Z INFO {farm_index=0}: subspace_farmer::single_disk_farm::plotting: Plotting sector retry sector_index=301
2024-05-14T13:00:46.629878Z INFO {farm_index=0}: subspace_farmer::single_disk_farm::plotting: Plotting sector retry sector_index=303
2024-05-14T13:00:56.627887Z WARN {farm_index=0}: subspace_farmer::single_disk_farm::plotting: Failed to plot sector, retrying in 1s sector_index=301 error=Low-level plotting error: Timed out without ping from plotter
2024-05-14T13:00:56.627919Z WARN {farm_index=0}: subspace_farmer::single_disk_farm::plotting: Failed to plot sector, retrying in 1s sector_index=302 error=Low-level plotting error: Timed out without ping from plotter
2024-05-14T13:00:56.631286Z WARN {farm_index=0}: subspace_farmer::single_disk_farm::plotting: Failed to plot sector, retrying in 1s sector_index=303 error=Low-level plotting error: Timed out without ping from plotter
2024-05-14T13:00:57.630725Z INFO {farm_index=0}:{public_key=ccc9f0d7b7cdd905d3d6f4ea0cbbecfe851be6f124d8119670fcf91b34acea6b sector_index=301}: subspace_farmer::single_disk_farm::plotting: Plotting sector retry sector_index=302
2024-05-14T13:00:57.630905Z INFO {farm_index=0}: subspace_farmer::single_disk_farm::plotting: Plotting sector retry sector_index=301
2024-05-14T13:00:57.633128Z INFO {farm_index=0}: subspace_farmer::single_disk_farm::plotting: Plotting sector retry sector_index=303
2024-05-14T13:01:07.632059Z WARN {farm_index=0}: subspace_farmer::single_disk_farm::plotting: Failed to plot sector, retrying in 1s sector_index=301 error=Low-level plotting error: Timed out without ping from plotter
2024-05-14T13:01:07.632087Z WARN {farm_index=0}: subspace_farmer::single_disk_farm::plotting: Failed to plot sector, retrying in 1s sector_index=302 error=Low-level plotting error: Timed out without ping from plotter
2024-05-14T13:01:07.634511Z WARN {farm_index=0}: subspace_farmer::single_disk_farm::plotting: Failed to plot sector, retrying in 1s sector_index=303 error=Low-level plotting error: Timed out without ping from plotter
2024-05-14T13:01:08.635073Z INFO {farm_index=0}:{public_key=ccc9f0d7b7cdd905d3d6f4ea0cbbecfe851be6f124d8119670fcf91b34acea6b sector_index=301}:{public_key=ccc9f0d7b7cdd905d3d6f4ea0cbbecfe851be6f124d8119670fcf91b34acea6b sector_index=302}:{public_key=ccc9f0d7b7cdd905d3d6f4ea0cbbecfe851be6f124d8119670fcf91b34acea6b sector_index=303}: subspace_farmer::single_disk_farm::plotting: Plotting sector retry sector_index=301
2024-05-14T13:01:08.635352Z INFO {farm_index=0}:{public_key=ccc9f0d7b7cdd905d3d6f4ea0cbbecfe851be6f124d8119670fcf91b34acea6b sector_index=301}:{public_key=ccc9f0d7b7cdd905d3d6f4ea0cbbecfe851be6f124d8119670fcf91b34acea6b sector_index=302}:{public_key=ccc9f0d7b7cdd905d3d6f4ea0cbbecfe851be6f124d8119670fcf91b34acea6b sector_index=303}: subspace_farmer::single_disk_farm::plotting: Plotting sector retry sector_index=302
2024-05-14T13:01:08.636213Z INFO {farm_index=0}: subspace_farmer::single_disk_farm::plotting: Plotting sector retry sector_index=303
It is really difficult when you don’t answer questions I’m asking 
Please run all components separately for now and if the issue persists, run them with RUST_LOG=info,subspace_farmer=trace so I have more information to work with. See how vexr reports bugs in farming cluster thread. Please include full commands you’re using to run things and if you build it from source then tell me how exactly are you doing it and exact commit from which you’ve built it.
Also please format messages like I edited above with spoilers and code blocks, it is much easier to read it that way comparing to a wall of text that everyone has to scroll through.
12400F should be fine and it shoud be possible to make it work. However the plotter doesn’t seem to be in logs above despite your claim, which likely indicates plotter is only running on some other machine, likely that underpowered ARM64 computer that you persistently refuse to provide any details on.
With newest commit in farming-cluster branch, this farmer timeout issue has gone.