Jun-18 subspace farmer plotter ping timeout

Support
1

Issue Report

Farmer log keep showing plotter timeout

Environment

  • Operating System: Ubuntu 18.04
  • Pulsar/Advanced CLI/Docker: CLI

Problem

Farmer keep reporting plotter timeout. This leads to unable to plot at all.
Since Jun-24 found this issue and everything works fine before.

[Paste any errors or relevant logs here]
node@node-MS-7D73:~/subspace$ ./subspace-farmer-ubuntu-x86_64-skylake-gemini-3h-2024-jun-18 cluster --nats-server nats://192.168.0.39:4222 farmer --reward-address stXXX path=/mnt/sda1,size=3.3TiB                                                                                          2024-06-24T07:59:39.715548Z  INFO subspace_farmer::cluster::controller: Downloading all segment headers from controller...
2024-06-24T07:59:39.715617Z  INFO async_nats: event: connected
2024-06-24T07:59:39.757011Z  INFO subspace_farmer::cluster::controller: Downloaded all segment headers from node successfully
2024-06-24T07:59:40.359632Z  INFO {farm_index=0}: subspace_farmer::single_disk_farm::plot_cache: Checking plot cache contents, this can take a while
2024-06-24T07:59:40.360899Z  INFO {farm_index=0}: subspace_farmer::single_disk_farm::plot_cache: Finished checking plot cache contents
2024-06-24T07:59:40.362788Z  INFO {farm_index=0}: subspace_farmer::single_disk_farm: Benchmarking faster proving method
2024-06-24T07:59:47.998540Z  INFO {farm_index=0}: subspace_farmer::single_disk_farm: Faster proving method found fastest_mode=ConcurrentChunks
2024-06-24T07:59:48.071880Z  INFO {farm_index=0}: subspace_farmer::commands::cluster::farmer:   ID: 01J064A906XK3GT42E4FA676XF2024-06-24T07:59:48.071885Z  INFO {farm_index=0}: subspace_farmer::commands::cluster::farmer:   Genesis hash: 0x0c121c75f4ef450f40619e1fca9d1e8e7fbabc42c895bc4790801e85d5a91c342024-06-24T07:59:48.071896Z  INFO {farm_index=0}: subspace_farmer::commands::cluster::farmer:   Public key: 0xee4c831be6acf47d9c6552aa8f15155caa1051a635798098b6a324c24d6bb97c2024-06-24T07:59:48.071904Z  INFO {farm_index=0}: subspace_farmer::commands::cluster::farmer:   Allocated space: 3.3 TiB (3.6 TB)2024-06-24T07:59:48.071908Z  INFO {farm_index=0}: subspace_farmer::commands::cluster::farmer:   Directory: /mnt/sda12024-06-24T07:59:48.102218Z  INFO {farm_index=0}: subspace_farmer::single_disk_farm::plotting: Subscribing to archived segments2024-06-24T07:59:48.102265Z  INFO {farm_index=0}: subspace_farmer::single_disk_farm::farming: Subscribing to slot info notificationsng notifications2024-06-24T07:59:48.126596Z  INFO {farm_index=0}:{sector_index=185}: subspace_farmer::single_disk_farm::plotting: Plotting sector (5.39% complete)2024-06-24T08:00:49.928957Z  INFO {farm_index=0}:{sector_index=186}: subspace_farmer::single_disk_farm::plotting: Plotting sector (5.42% complete)2024-06-24T08:00:49.945535Z  INFO {farm_index=0}:{sector_index=187}: subspace_farmer::single_disk_farm::plotting: Plotting sector (5.45% complete)2024-06-24T08:00:49.962459Z  INFO {farm_index=0}:{sector_index=188}: subspace_farmer::single_disk_farm::plotting: Plotting sector (5.48% complete)2024-06-24T08:00:49.979279Z  INFO {farm_index=0}:{sector_index=189}: subspace_farmer::single_disk_farm::plotting: Plotting sector (5.51% complete)2024-06-24T08:00:49.996013Z  INFO {farm_index=0}:{sector_index=190}: subspace_farmer::single_disk_farm::plotting: Plotting sector (5.54% complete)2024-06-24T08:00:50.013163Z  INFO {farm_index=0}:{sector_index=191}: subspace_farmer::single_disk_farm::plotting: Plotting sector (5.57% complete)2024-06-24T08:00:50.651375Z  INFO {farm_index=0}:{sector_index=192}: subspace_farmer::single_disk_farm::plotting: Plotting sector (5.59% complete)2024-06-24T08:00:50.695628Z  INFO {farm_index=0}:{sector_index=195}: subspace_farmer::single_disk_farm::plotting: Plotting sector (5.68% complete)2024-06-24T08:00:50.742299Z  INFO {farm_index=0}:{sector_index=198}: subspace_farmer::single_disk_farm::plotting: Plotting sector (5.77% complete)2024-06-24T08:00:51.057832Z  INFO {farm_index=0}:{sector_index=199}: subspace_farmer::single_disk_farm::plotting: Plotting sector (5.80% complete)2024-06-24T08:00:51.074374Z  INFO {farm_index=0}:{sector_index=200}: subspace_farmer::single_disk_farm::plotting: Plotting sector (5.83% complete)2024-06-24T08:00:51.539351Z  INFO {farm_index=0}:{sector_index=201}: subspace_farmer::single_disk_farm::plotting: Plotting sector (5.86% complete)2024-06-24T08:00:51.555016Z  INFO {farm_index=0}:{sector_index=202}: subspace_farmer::single_disk_farm::plotting: Plotting sector (5.89% complete)^C2024-06-24T08:01:48.457893Z  INFO subspace_farmer::utils: Received SIGINT, shutting down farmer...node@node-MS-7D73:~/subspace$ ./subspace-farmer-ubuntu-x86_64-skylake-gemini-3h-2024-jun-18 cluster --nats-server nats://192.168.0.39:4222 farmer --reward-address st7Qmat3inHZtQojUS6w1BoBNKH2Hef6zjtPRRQgEcBVF6J4A path=/mnt/sda1,size=3.3TiB2024-06-24T08:01:49.520810Z  INFO subspace_farmer::cluster::controller: Downloading all segment headers from controller...2024-06-24T08:01:49.520847Z  INFO async_nats: event: connected2024-06-24T08:01:49.544495Z  INFO subspace_farmer::cluster::controller: Downloaded all segment headers from node successfully2024-06-24T08:01:50.060371Z  INFO {farm_index=0}: subspace_farmer::single_disk_farm::plot_cache: Checking plot cache contents, this can take a while2024-06-24T08:01:50.061394Z  INFO {farm_index=0}: subspace_farmer::single_disk_farm::plot_cache: Finished checking plot cache contents2024-06-24T08:01:50.063108Z  INFO {farm_index=0}: subspace_farmer::single_disk_farm: Benchmarking faster proving method
^Cnode@node-MS-7D73:~/subspace$ ./subspace-farmer-ubuntu-x86_64-skylake-gemini-3h-2024-jun-18 cluster --nats-server nats://192.168.0.39:4222 farmer --reward-address st7Qmat3inHZtQojUS6w1BoBNKH2Hef6zjtPRRQgEcBVF6J4A path=/mnt/sda1,size=3.3TiB2024-06-24T08:02:04.768352Z  INFO subspace_farmer::cluster::controller: Downloading all segment headers from controller...2024-06-24T08:02:04.768374Z  INFO async_nats: event: connected2024-06-24T08:02:04.791665Z  INFO subspace_farmer::cluster::controller: Downloaded all segment headers from node successfully2024-06-24T08:02:05.389814Z  INFO {farm_index=0}: subspace_farmer::single_disk_farm::plot_cache: Checking plot cache contents, this can take a while2024-06-24T08:02:05.391064Z  INFO {farm_index=0}: subspace_farmer::single_disk_farm::plot_cache: Finished checking plot cache contents2024-06-24T08:02:05.392845Z  INFO {farm_index=0}: subspace_farmer::single_disk_farm: Benchmarking faster proving method
2024-06-24T08:02:13.186912Z  INFO {farm_index=0}: subspace_farmer::single_disk_farm: Faster proving method found fastest_mode=ConcurrentChunks
2024-06-24T08:02:13.242884Z  INFO {farm_index=0}: subspace_farmer::commands::cluster::farmer: Farm 0:2024-06-24T08:02:13.242897Z  INFO {farm_index=0}: subspace_farmer::commands::cluster::farmer:   ID: 01J064A906XK3GT42E4FA676XF2024-06-24T08:02:13.242903Z  INFO {farm_index=0}: subspace_farmer::commands::cluster::farmer:   Genesis hash: 0x0c121c75f4ef450f40619e1fca9d1e8e7fbabc42c895bc4790801e85d5a91c342024-06-24T08:02:13.242909Z  INFO {farm_index=0}: subspace_farmer::commands::cluster::farmer:   Public key: 0xee4c831be6acf47d9c6552aa8f15155caa1051a635798098b6a324c24d6bb97c2024-06-24T08:02:13.242923Z  INFO {farm_index=0}: subspace_farmer::commands::cluster::farmer:   Allocated space: 3.3 TiB (3.6 TB)
2024-06-24T08:02:13.242928Z  INFO {farm_index=0}: subspace_farmer::commands::cluster::farmer:   Directory: /mnt/sda1
2024-06-24T08:02:13.274021Z  INFO {farm_index=0}: subspace_farmer::single_disk_farm::plotting: Subscribing to archived segments
2024-06-24T08:02:13.274069Z  INFO {farm_index=0}: subspace_farmer::single_disk_farm::farming: Subscribing to slot info notifications
2024-06-24T08:02:13.274081Z  INFO {farm_index=0}: subspace_farmer::reward_signing: Subscribing to reward signing notifications
2024-06-24T08:02:13.291113Z  INFO {farm_index=0}:{sector_index=185}: subspace_farmer::single_disk_farm::plotting: Plotting sector (5.39% complete)
2024-06-24T08:02:13.797141Z  INFO {farm_index=0}:{sector_index=186}: subspace_farmer::single_disk_farm::plotting: Plotting sector (5.42% complete)
2024-06-24T08:02:14.267570Z  INFO {farm_index=0}:{sector_index=187}: subspace_farmer::single_disk_farm::plotting: Plotting sector (5.45% complete)
2024-06-24T08:02:14.737584Z  INFO {farm_index=0}:{sector_index=188}: subspace_farmer::single_disk_farm::plotting: Plotting sector (5.48% complete)
2024-06-24T08:02:15.204026Z  INFO {farm_index=0}:{sector_index=189}: subspace_farmer::single_disk_farm::plotting: Plotting sector (5.51% complete)
2024-06-24T08:02:15.535879Z  INFO {farm_index=0}:{sector_index=190}: subspace_farmer::single_disk_farm::plotting: Plotting sector (5.54% complete)
2024-06-24T08:02:15.537588Z  INFO {farm_index=0}:{sector_index=191}: subspace_farmer::single_disk_farm::plotting: Plotting sector (5.57% complete)
2024-06-24T08:02:15.539312Z  INFO {farm_index=0}:{sector_index=192}: subspace_farmer::single_disk_farm::plotting: Plotting sector (5.59% complete)
2024-06-24T08:02:15.541028Z  INFO {farm_index=0}:{sector_index=193}: subspace_farmer::single_disk_farm::plotting: Plotting sector (5.62% complete)
2024-06-24T08:02:15.542486Z  INFO {farm_index=0}:{sector_index=194}: subspace_farmer::single_disk_farm::plotting: Plotting sector (5.65% complete)
2024-06-24T08:02:15.544107Z  INFO {farm_index=0}:{sector_index=195}: subspace_farmer::single_disk_farm::plotting: Plotting sector (5.68% complete)
2024-06-24T08:02:15.545275Z  INFO {farm_index=0}:{sector_index=196}: subspace_farmer::single_disk_farm::plotting: Plotting sector (5.71% complete)
2024-06-24T08:02:15.963604Z  INFO {farm_index=0}:{sector_index=197}: subspace_farmer::single_disk_farm::plotting: Plotting sector (5.74% complete)
2024-06-24T08:02:15.965468Z  INFO {farm_index=0}:{sector_index=198}: subspace_farmer::single_disk_farm::plotting: Plotting sector (5.77% complete)
2024-06-24T08:02:15.966964Z  INFO {farm_index=0}:{sector_index=199}: subspace_farmer::single_disk_farm::plotting: Plotting sector (5.80% complete)
2024-06-24T08:02:15.968312Z  INFO {farm_index=0}:{sector_index=200}: subspace_farmer::single_disk_farm::plotting: Plotting sector (5.83% complete)
2024-06-24T08:02:15.969813Z  INFO {farm_index=0}:{sector_index=201}: subspace_farmer::single_disk_farm::plotting: Plotting sector (5.86% complete)
2024-06-24T08:02:17.678285Z  INFO {farm_index=0}:{sector_index=202}: subspace_farmer::single_disk_farm::plotting: Plotting sector (5.89% complete)
2024-06-24T08:02:17.679932Z  INFO {farm_index=0}:{sector_index=203}: subspace_farmer::single_disk_farm::plotting: Plotting sector (5.91% complete)
2024-06-24T08:02:18.830132Z  INFO {farm_index=0}:{sector_index=204}: subspace_farmer::single_disk_farm::plotting: Plotting sector (5.94% complete)
2024-06-24T08:05:03.614223Z  WARN {farm_index=0}:{sector_index=185}: subspace_farmer::single_disk_farm::plotting: Failed to plot sector, retrying in 1s error=Low-level plotting error: Timed out without ping from plotter
2024-06-24T08:05:04.103522Z  WARN {farm_index=0}:{sector_index=186}: subspace_farmer::single_disk_farm::plotting: Failed to plot sector, retrying in 1s error=Low-level plotting error: Timed out without ping from plotter
2024-06-24T08:05:04.572873Z  WARN {farm_index=0}:{sector_index=187}: subspace_farmer::single_disk_farm::plotting: Failed to plot sector, retrying in 1s error=Low-level plotting error: Timed out without ping from plotter
2024-06-24T08:05:05.038472Z  WARN {farm_index=0}:{sector_index=188}: subspace_farmer::single_disk_farm::plotting: Failed to plot sector, retrying in 1s error=Low-level plotting error: Timed out without ping from plotter
2024-06-24T08:05:05.506776Z  WARN {farm_index=0}:{sector_index=189}: subspace_farmer::single_disk_farm::plotting: Failed to plot sector, retrying in 1s error=Low-level plotting error: Timed out without ping from plotter
2024-06-24T08:05:05.545246Z  WARN {farm_index=0}:{sector_index=191}: subspace_farmer::single_disk_farm::plotting: Failed to plot sector, retrying in 1s error=Low-level plotting error: Timed out without ping from plotter
2024-06-24T08:05:05.545273Z  WARN {farm_index=0}:{sector_index=190}: subspace_farmer::single_disk_farm::plotting: Failed to plot sector, retrying in 1s error=Low-level plotting error: Timed out without ping from plotter
2024-06-24T08:05:05.549126Z  WARN {farm_index=0}:{sector_index=193}: subspace_farmer::single_disk_farm::plotting: Failed to plot sector, retrying in 1s error=Low-level plotting error: Timed out without ping from plotter
2024-06-24T08:05:05.551341Z  WARN {farm_index=0}:{sector_index=194}: subspace_farmer::single_disk_farm::plotting: Failed to plot sector, retrying in 1s error=Low-level plotting error: Timed out without ping from plotter
2024-06-24T08:05:05.552782Z  WARN {farm_index=0}:{sector_index=196}: subspace_farmer::single_disk_farm::plotting: Failed to plot sector, retrying in 1s error=Low-level plotting error: Timed out without ping from plotter
2024-06-24T08:05:05.971455Z  WARN {farm_index=0}:{sector_index=197}: subspace_farmer::single_disk_farm::plotting: Failed to plot sector, retrying in 1s error=Low-level plotting error: Timed out without ping from plotter
2024-06-24T08:05:05.973650Z  WARN {farm_index=0}:{sector_index=198}: subspace_farmer::single_disk_farm::plotting: Failed to plot sector, retrying in 1s error=Low-level plotting error: Timed out without ping from plotter
2024-06-24T08:05:05.974544Z  WARN {farm_index=0}:{sector_index=199}: subspace_farmer::single_disk_farm::plotting: Failed to plot sector, retrying in 1s error=Low-level plotting error: Timed out without ping from plotter
2024-06-24T08:05:05.976655Z  WARN {farm_index=0}:{sector_index=201}: subspace_farmer::single_disk_farm::plotting: Failed to plot sector, retrying in 1s error=Low-level plotting error: Timed out without ping from plotter
2024-06-24T08:05:07.687240Z  WARN {farm_index=0}:{sector_index=202}: subspace_farmer::single_disk_farm::plotting: Failed to plot sector, retrying in 1s error=Low-level plotting error: Timed out without ping from plotter
2024-06-24T08:05:08.838602Z  WARN {farm_index=0}:{sector_index=204}: subspace_farmer::single_disk_farm::plotting: Failed to plot sector, retrying in 1s error=Low-level plotting error: Timed out without ping from plotter
2024-06-24T08:05:45.546573Z  WARN {farm_index=0}:{sector_index=192}: subspace_farmer::single_disk_farm::plotting: Failed to plot sector, retrying in 1s error=Low-level plotting error: Timed out without ping from plotter
2024-06-24T08:05:47.688083Z  WARN {farm_index=0}:{sector_index=203}: subspace_farmer::single_disk_farm::plotting: Failed to plot sector, retrying in 1s error=Low-level plotting error: Timed out without ping from plotter
2

Looks like something somewhere might be overloaded. What does network bandwidth look like? It would also be helpful to know what and how is running here, see Timed out without ping from plotter for a great example of reporting with important details present. There is also a test build there that may potentially help.

3

I have 10 servers running plotter process, and timeout occurs.
But if I run only 2 plotters, no RPC timeout warnings.

Is there a limitation on total number of connection on NATs?
Can I run two or more NATS server in subspace cluster for my case?

4 
[Paste any errors or relevant logs here]
NATs Server log has warnings:
[1] 2024/06/24 11:23:16.059252 [INF] 192.168.0.30:57484 - cid:407 - Slow Consumer Detected: WriteDeadline of 10s exceeded with 459 chunks of 29365342 total bytes.
[1] 2024/06/24 11:23:44.826526 [INF] 192.168.0.37:36014 - cid:405 - Slow Consumer Detected: WriteDeadline of 10s exceeded with 394 chunks of 25170666 total bytes.
[1] 2024/06/24 11:26:38.164135 [INF] 192.168.0.31:38454 - cid:408 - Slow Consumer Detected: WriteDeadline of 10s exceeded with 410 chunks of 26219335 total bytes.
[1] 2024/06/24 11:26:57.873645 [INF] 192.168.0.32:38826 - cid:403 - Slow Consumer Detected: WriteDeadline of 10s exceeded with 378 chunks of 24122084 total bytes.
[1] 2024/06/24 11:27:07.125728 [INF] 192.168.0.38:39604 - cid:411 - Slow Consumer Detected: WriteDeadline of 10s exceeded with 442 chunks of 28316499 total bytes.
[1] 2024/06/24 11:27:40.774242 [INF] 192.168.0.33:48860 - cid:412 - Slow Consumer Detected: WriteDeadline of 10s exceeded with 507 chunks of 32511436 total bytes.
[1] 2024/06/24 11:28:22.355598 [INF] 192.168.0.36:53310 - cid:413 - Slow Consumer Detected: WriteDeadline of 10s exceeded with 426 chunks of 27268091 total bytes.
[1] 2024/06/24 11:30:29.342680 [INF] 192.168.0.35:59254 - cid:398 - Slow Consumer Detected: WriteDeadline of 10s exceeded with 507 chunks of 32511436 total bytes.
[1] 2024/06/24 11:30:30.645076 [INF] 192.168.0.37:36036 - cid:415 - Slow Consumer Detected: WriteDeadline of 10s exceeded with 378 chunks of 24122084 total bytes.
[1] 2024/06/24 11:31:40.310326 [INF] 192.168.0.34:58016 - cid:410 - Slow Consumer Detected: WriteDeadline of 10s exceeded with 443 chunks of 28316673 total bytes.
[1] 2024/06/24 11:32:16.798575 [INF] 192.168.0.38:39622 - cid:418 - Slow Consumer Detected: WriteDeadline of 10s exceeded with 442 chunks of 28316499 total bytes.
[1] 2024/06/24 11:32:19.370884 [INF] 192.168.0.32:38864 - cid:417 - Slow Consumer Detected: WriteDeadline of 10s exceeded with 394 chunks of 25170816 total bytes.
[1] 2024/06/24 11:32:54.124996 [INF] 192.168.0.33:48876 - cid:419 - Slow Consumer Detected: WriteDeadline of 10s exceeded with 507 chunks of 32511436 total bytes.
[1] 2024/06/24 11:36:51.679004 [INF] 192.168.0.34:58046 - cid:423 - Slow Consumer Detected: WriteDeadline of 10s exceeded with 507 chunks of 32511436 total bytes.
[1] 2024/06/24 11:37:06.576659 [INF] 192.168.0.37:36054 - cid:422 - Slow Consumer Detected: WriteDeadline of 10s exceeded with 507 chunks of 32511436 total bytes.
[1] 2024/06/24 11:37:36.164943 [INF] 192.168.0.30:57498 - cid:414 - Slow Consumer Detected: WriteDeadline of 10s exceeded with 394 chunks of 25170666 total bytes.
[1] 2024/06/24 11:38:10.614318 [INF] 192.168.0.33:48894 - cid:426 - Slow Consumer Detected: WriteDeadline of 10s exceeded with 507 chunks of 32511436 total bytes.
[1] 2024/06/24 11:40:41.376313 [INF] 192.168.0.32:38878 - cid:425 - Slow Consumer Detected: WriteDeadline of 10s exceeded with 507 chunks of 32511436 total bytes.
[1] 2024/06/24 11:42:07.500396 [INF] 192.168.0.36:53328 - cid:420 - Slow Consumer Detected: WriteDeadline of 10s exceeded with 507 chunks of 32511436 total bytes.
[1] 2024/06/24 11:42:33.598518 [INF] 192.168.0.37:36076 - cid:428 - Slow Consumer Detected: WriteDeadline of 10s exceeded with 507 chunks of 32511436 total bytes.
[1] 2024/06/24 11:43:14.623524 [INF] 192.168.0.30:57540 - cid:429 - Slow Consumer Detected: WriteDeadline of 10s exceeded with 509 chunks of 32512196 total bytes.
[1] 2024/06/24 11:45:51.549990 [INF] 192.168.0.32:38904 - cid:431 - Slow Consumer Detected: WriteDeadline of 10s exceeded with 491 chunks of 31462680 total bytes.
[1] 2024/06/24 11:46:08.958069 [INF] 192.168.0.29:43032 - cid:409 - Slow Consumer Detected: WriteDeadline of 10s exceeded with 507 chunks of 32511436 total bytes.
[1] 2024/06/24 11:46:09.078952 [INF] 192.168.0.34:58060 - cid:427 - Slow Consumer Detected: WriteDeadline of 10s exceeded with 459 chunks of 29365429 total bytes.
[1] 2024/06/24 11:52:40.660827 [INF] 192.168.0.31:38480 - cid:416 - Slow Consumer Detected: WriteDeadline of 10s exceeded with 507 chunks of 32511436 total bytes.
[1] 2024/06/24 12:14:16.017898 [INF] 192.168.0.29:43112 - cid:440 - Slow Consumer Detected: WriteDeadline of 10s exceeded with 475 chunks of 30413924 total bytes.
[1] 2024/06/24 12:14:16.367959 [INF] 192.168.0.31:38608 - cid:444 - Slow Consumer Detected: WriteDeadline of 10s exceeded with 507 chunks of 32511436 total bytes.
[1] 2024/06/24 12:15:19.751013 [INF] 192.168.0.29:43190 - cid:447 - Slow Consumer Detected: WriteDeadline of 10s exceeded with 507 chunks of 32511436 total bytes.
[1] 2024/06/24 12:15:24.048282 [INF] 192.168.0.32:38996 - cid:443 - Slow Consumer Detected: WriteDeadline of 10s exceeded with 491 chunks of 31462680 total bytes.
5

Those timeouts seem to indicate that readers of messages can’t process them in time.
What hardware is all this running on, do you see network bandwidth pegged on NATS server to the max by any chance? How are those machines connected to each other? I need more information to understand what is going on.

There is no inherent limitation except hardware capabilities, NATS should be able to scale with CPU cores and network speed. You can run more than one and it might help depending on network topology, just make sure to create a cluster and not multiple independent NATS servers or you will have issues: Clustering | NATS Docs

6

The network bandwidth is close to 120MB/s bottleneck when plotter finished a sector.
The timeout issue is fixed when I move NATS service from node to farmer node.
It seems that NATS should avoid to run with node, since both of them have network requirement.

7

This is a helpful information, thank you!
I think there is still opportunity to opimize bandwidth spikes further, I’ll look into it going forward.

8

Move NATS service from node to farmer node, it did fix the RPC timeout issue but the plotting speed is much slower than before.
About 4 mins each sector when nats runs along with node,controller and cache. when nats runs with only farmer, about 40 mins or more for each sector to plot.
Is there any constrains for subspace cluster services? Should nats must run with controller and cache at the same server for performance?