# Jun-18 subspace farmer plotter ping timeout **URL:** \ **Category:** Support\ **Created:** [June 24, 2024, 8:09am UTC](https://forum.autonomys.xyz/t/jun-18-subspace-farmer-plotter-ping-timeout/4283 "2024-06-24T08:09:01Z")\ **Posts on this page:** 8\ **Page:** 1
**Author:** ![keens312](https://avatars.discourse-cdn.com/v4/letter/k/90ced4/32.png) [@keens312](https://forum.autonomys.xyz/u/keens312)\ **Post date:** [June 24, 2024, 8:09am UTC](https://forum.autonomys.xyz/t/jun-18-subspace-farmer-plotter-ping-timeout/4283/1 "2024-06-24T08:09:01Z")
## Issue Report Farmer log keep showing plotter timeout ### Environment - **Operating System:** Ubuntu 18.04 - **Pulsar/Advanced CLI/Docker:** CLI ### Problem Farmer keep reporting plotter timeout. This leads to unable to plot at all. Since Jun-24 found this issue and everything works fine before. ```auto [Paste any errors or relevant logs here] node@node-MS-7D73:~/subspace$ ./subspace-farmer-ubuntu-x86_64-skylake-gemini-3h-2024-jun-18 cluster --nats-server nats://192.168.0.39:4222 farmer --reward-address stXXX path=/mnt/sda1,size=3.3TiB 2024-06-24T07:59:39.715548Z INFO subspace_farmer::cluster::controller: Downloading all segment headers from controller... 2024-06-24T07:59:39.715617Z INFO async_nats: event: connected 2024-06-24T07:59:39.757011Z INFO subspace_farmer::cluster::controller: Downloaded all segment headers from node successfully 2024-06-24T07:59:40.359632Z INFO {farm_index=0}: subspace_farmer::single_disk_farm::plot_cache: Checking plot cache contents, this can take a while 2024-06-24T07:59:40.360899Z INFO {farm_index=0}: subspace_farmer::single_disk_farm::plot_cache: Finished checking plot cache contents 2024-06-24T07:59:40.362788Z INFO {farm_index=0}: subspace_farmer::single_disk_farm: Benchmarking faster proving method 2024-06-24T07:59:47.998540Z INFO {farm_index=0}: subspace_farmer::single_disk_farm: Faster proving method found fastest_mode=ConcurrentChunks 2024-06-24T07:59:48.071880Z INFO {farm_index=0}: subspace_farmer::commands::cluster::farmer: ID: 01J064A906XK3GT42E4FA676XF2024-06-24T07:59:48.071885Z INFO {farm_index=0}: subspace_farmer::commands::cluster::farmer: Genesis hash: 0x0c121c75f4ef450f40619e1fca9d1e8e7fbabc42c895bc4790801e85d5a91c342024-06-24T07:59:48.071896Z INFO {farm_index=0}: subspace_farmer::commands::cluster::farmer: Public key: 0xee4c831be6acf47d9c6552aa8f15155caa1051a635798098b6a324c24d6bb97c2024-06-24T07:59:48.071904Z INFO {farm_index=0}: subspace_farmer::commands::cluster::farmer: Allocated space: 3.3 TiB (3.6 TB)2024-06-24T07:59:48.071908Z INFO {farm_index=0}: subspace_farmer::commands::cluster::farmer: Directory: /mnt/sda12024-06-24T07:59:48.102218Z INFO {farm_index=0}: subspace_farmer::single_disk_farm::plotting: Subscribing to archived segments2024-06-24T07:59:48.102265Z INFO {farm_index=0}: subspace_farmer::single_disk_farm::farming: Subscribing to slot info notificationsng notifications2024-06-24T07:59:48.126596Z INFO {farm_index=0}:{sector_index=185}: subspace_farmer::single_disk_farm::plotting: Plotting sector (5.39% complete)2024-06-24T08:00:49.928957Z INFO {farm_index=0}:{sector_index=186}: subspace_farmer::single_disk_farm::plotting: Plotting sector (5.42% complete)2024-06-24T08:00:49.945535Z INFO {farm_index=0}:{sector_index=187}: subspace_farmer::single_disk_farm::plotting: Plotting sector (5.45% complete)2024-06-24T08:00:49.962459Z INFO {farm_index=0}:{sector_index=188}: subspace_farmer::single_disk_farm::plotting: Plotting sector (5.48% complete)2024-06-24T08:00:49.979279Z INFO {farm_index=0}:{sector_index=189}: subspace_farmer::single_disk_farm::plotting: Plotting sector (5.51% complete)2024-06-24T08:00:49.996013Z INFO {farm_index=0}:{sector_index=190}: subspace_farmer::single_disk_farm::plotting: Plotting sector (5.54% complete)2024-06-24T08:00:50.013163Z INFO {farm_index=0}:{sector_index=191}: subspace_farmer::single_disk_farm::plotting: Plotting sector (5.57% complete)2024-06-24T08:00:50.651375Z INFO {farm_index=0}:{sector_index=192}: subspace_farmer::single_disk_farm::plotting: Plotting sector (5.59% complete)2024-06-24T08:00:50.695628Z INFO {farm_index=0}:{sector_index=195}: subspace_farmer::single_disk_farm::plotting: Plotting sector (5.68% complete)2024-06-24T08:00:50.742299Z INFO {farm_index=0}:{sector_index=198}: subspace_farmer::single_disk_farm::plotting: Plotting sector (5.77% complete)2024-06-24T08:00:51.057832Z INFO {farm_index=0}:{sector_index=199}: subspace_farmer::single_disk_farm::plotting: Plotting sector (5.80% complete)2024-06-24T08:00:51.074374Z INFO {farm_index=0}:{sector_index=200}: subspace_farmer::single_disk_farm::plotting: Plotting sector (5.83% complete)2024-06-24T08:00:51.539351Z INFO {farm_index=0}:{sector_index=201}: subspace_farmer::single_disk_farm::plotting: Plotting sector (5.86% complete)2024-06-24T08:00:51.555016Z INFO {farm_index=0}:{sector_index=202}: subspace_farmer::single_disk_farm::plotting: Plotting sector (5.89% complete)^C2024-06-24T08:01:48.457893Z INFO subspace_farmer::utils: Received SIGINT, shutting down farmer...node@node-MS-7D73:~/subspace$ ./subspace-farmer-ubuntu-x86_64-skylake-gemini-3h-2024-jun-18 cluster --nats-server nats://192.168.0.39:4222 farmer --reward-address st7Qmat3inHZtQojUS6w1BoBNKH2Hef6zjtPRRQgEcBVF6J4A path=/mnt/sda1,size=3.3TiB2024-06-24T08:01:49.520810Z INFO subspace_farmer::cluster::controller: Downloading all segment headers from controller...2024-06-24T08:01:49.520847Z INFO async_nats: event: connected2024-06-24T08:01:49.544495Z INFO subspace_farmer::cluster::controller: Downloaded all segment headers from node successfully2024-06-24T08:01:50.060371Z INFO {farm_index=0}: subspace_farmer::single_disk_farm::plot_cache: Checking plot cache contents, this can take a while2024-06-24T08:01:50.061394Z INFO {farm_index=0}: subspace_farmer::single_disk_farm::plot_cache: Finished checking plot cache contents2024-06-24T08:01:50.063108Z INFO {farm_index=0}: subspace_farmer::single_disk_farm: Benchmarking faster proving method ^Cnode@node-MS-7D73:~/subspace$ ./subspace-farmer-ubuntu-x86_64-skylake-gemini-3h-2024-jun-18 cluster --nats-server nats://192.168.0.39:4222 farmer --reward-address st7Qmat3inHZtQojUS6w1BoBNKH2Hef6zjtPRRQgEcBVF6J4A path=/mnt/sda1,size=3.3TiB2024-06-24T08:02:04.768352Z INFO subspace_farmer::cluster::controller: Downloading all segment headers from controller...2024-06-24T08:02:04.768374Z INFO async_nats: event: connected2024-06-24T08:02:04.791665Z INFO subspace_farmer::cluster::controller: Downloaded all segment headers from node successfully2024-06-24T08:02:05.389814Z INFO {farm_index=0}: subspace_farmer::single_disk_farm::plot_cache: Checking plot cache contents, this can take a while2024-06-24T08:02:05.391064Z INFO {farm_index=0}: subspace_farmer::single_disk_farm::plot_cache: Finished checking plot cache contents2024-06-24T08:02:05.392845Z INFO {farm_index=0}: subspace_farmer::single_disk_farm: Benchmarking faster proving method 2024-06-24T08:02:13.186912Z INFO {farm_index=0}: subspace_farmer::single_disk_farm: Faster proving method found fastest_mode=ConcurrentChunks 2024-06-24T08:02:13.242884Z INFO {farm_index=0}: subspace_farmer::commands::cluster::farmer: Farm 0:2024-06-24T08:02:13.242897Z INFO {farm_index=0}: subspace_farmer::commands::cluster::farmer: ID: 01J064A906XK3GT42E4FA676XF2024-06-24T08:02:13.242903Z INFO {farm_index=0}: subspace_farmer::commands::cluster::farmer: Genesis hash: 0x0c121c75f4ef450f40619e1fca9d1e8e7fbabc42c895bc4790801e85d5a91c342024-06-24T08:02:13.242909Z INFO {farm_index=0}: subspace_farmer::commands::cluster::farmer: Public key: 0xee4c831be6acf47d9c6552aa8f15155caa1051a635798098b6a324c24d6bb97c2024-06-24T08:02:13.242923Z INFO {farm_index=0}: subspace_farmer::commands::cluster::farmer: Allocated space: 3.3 TiB (3.6 TB) 2024-06-24T08:02:13.242928Z INFO {farm_index=0}: subspace_farmer::commands::cluster::farmer: Directory: /mnt/sda1 2024-06-24T08:02:13.274021Z INFO {farm_index=0}: subspace_farmer::single_disk_farm::plotting: Subscribing to archived segments 2024-06-24T08:02:13.274069Z INFO {farm_index=0}: subspace_farmer::single_disk_farm::farming: Subscribing to slot info notifications 2024-06-24T08:02:13.274081Z INFO {farm_index=0}: subspace_farmer::reward_signing: Subscribing to reward signing notifications 2024-06-24T08:02:13.291113Z INFO {farm_index=0}:{sector_index=185}: subspace_farmer::single_disk_farm::plotting: Plotting sector (5.39% complete) 2024-06-24T08:02:13.797141Z INFO {farm_index=0}:{sector_index=186}: subspace_farmer::single_disk_farm::plotting: Plotting sector (5.42% complete) 2024-06-24T08:02:14.267570Z INFO {farm_index=0}:{sector_index=187}: subspace_farmer::single_disk_farm::plotting: Plotting sector (5.45% complete) 2024-06-24T08:02:14.737584Z INFO {farm_index=0}:{sector_index=188}: subspace_farmer::single_disk_farm::plotting: Plotting sector (5.48% complete) 2024-06-24T08:02:15.204026Z INFO {farm_index=0}:{sector_index=189}: subspace_farmer::single_disk_farm::plotting: Plotting sector (5.51% complete) 2024-06-24T08:02:15.535879Z INFO {farm_index=0}:{sector_index=190}: subspace_farmer::single_disk_farm::plotting: Plotting sector (5.54% complete) 2024-06-24T08:02:15.537588Z INFO {farm_index=0}:{sector_index=191}: subspace_farmer::single_disk_farm::plotting: Plotting sector (5.57% complete) 2024-06-24T08:02:15.539312Z INFO {farm_index=0}:{sector_index=192}: subspace_farmer::single_disk_farm::plotting: Plotting sector (5.59% complete) 2024-06-24T08:02:15.541028Z INFO {farm_index=0}:{sector_index=193}: subspace_farmer::single_disk_farm::plotting: Plotting sector (5.62% complete) 2024-06-24T08:02:15.542486Z INFO {farm_index=0}:{sector_index=194}: subspace_farmer::single_disk_farm::plotting: Plotting sector (5.65% complete) 2024-06-24T08:02:15.544107Z INFO {farm_index=0}:{sector_index=195}: subspace_farmer::single_disk_farm::plotting: Plotting sector (5.68% complete) 2024-06-24T08:02:15.545275Z INFO {farm_index=0}:{sector_index=196}: subspace_farmer::single_disk_farm::plotting: Plotting sector (5.71% complete) 2024-06-24T08:02:15.963604Z INFO {farm_index=0}:{sector_index=197}: subspace_farmer::single_disk_farm::plotting: Plotting sector (5.74% complete) 2024-06-24T08:02:15.965468Z INFO {farm_index=0}:{sector_index=198}: subspace_farmer::single_disk_farm::plotting: Plotting sector (5.77% complete) 2024-06-24T08:02:15.966964Z INFO {farm_index=0}:{sector_index=199}: subspace_farmer::single_disk_farm::plotting: Plotting sector (5.80% complete) 2024-06-24T08:02:15.968312Z INFO {farm_index=0}:{sector_index=200}: subspace_farmer::single_disk_farm::plotting: Plotting sector (5.83% complete) 2024-06-24T08:02:15.969813Z INFO {farm_index=0}:{sector_index=201}: subspace_farmer::single_disk_farm::plotting: Plotting sector (5.86% complete) 2024-06-24T08:02:17.678285Z INFO {farm_index=0}:{sector_index=202}: subspace_farmer::single_disk_farm::plotting: Plotting sector (5.89% complete) 2024-06-24T08:02:17.679932Z INFO {farm_index=0}:{sector_index=203}: subspace_farmer::single_disk_farm::plotting: Plotting sector (5.91% complete) 2024-06-24T08:02:18.830132Z INFO {farm_index=0}:{sector_index=204}: subspace_farmer::single_disk_farm::plotting: Plotting sector (5.94% complete) 2024-06-24T08:05:03.614223Z WARN {farm_index=0}:{sector_index=185}: subspace_farmer::single_disk_farm::plotting: Failed to plot sector, retrying in 1s error=Low-level plotting error: Timed out without ping from plotter 2024-06-24T08:05:04.103522Z WARN {farm_index=0}:{sector_index=186}: subspace_farmer::single_disk_farm::plotting: Failed to plot sector, retrying in 1s error=Low-level plotting error: Timed out without ping from plotter 2024-06-24T08:05:04.572873Z WARN {farm_index=0}:{sector_index=187}: subspace_farmer::single_disk_farm::plotting: Failed to plot sector, retrying in 1s error=Low-level plotting error: Timed out without ping from plotter 2024-06-24T08:05:05.038472Z WARN {farm_index=0}:{sector_index=188}: subspace_farmer::single_disk_farm::plotting: Failed to plot sector, retrying in 1s error=Low-level plotting error: Timed out without ping from plotter 2024-06-24T08:05:05.506776Z WARN {farm_index=0}:{sector_index=189}: subspace_farmer::single_disk_farm::plotting: Failed to plot sector, retrying in 1s error=Low-level plotting error: Timed out without ping from plotter 2024-06-24T08:05:05.545246Z WARN {farm_index=0}:{sector_index=191}: subspace_farmer::single_disk_farm::plotting: Failed to plot sector, retrying in 1s error=Low-level plotting error: Timed out without ping from plotter 2024-06-24T08:05:05.545273Z WARN {farm_index=0}:{sector_index=190}: subspace_farmer::single_disk_farm::plotting: Failed to plot sector, retrying in 1s error=Low-level plotting error: Timed out without ping from plotter 2024-06-24T08:05:05.549126Z WARN {farm_index=0}:{sector_index=193}: subspace_farmer::single_disk_farm::plotting: Failed to plot sector, retrying in 1s error=Low-level plotting error: Timed out without ping from plotter 2024-06-24T08:05:05.551341Z WARN {farm_index=0}:{sector_index=194}: subspace_farmer::single_disk_farm::plotting: Failed to plot sector, retrying in 1s error=Low-level plotting error: Timed out without ping from plotter 2024-06-24T08:05:05.552782Z WARN {farm_index=0}:{sector_index=196}: subspace_farmer::single_disk_farm::plotting: Failed to plot sector, retrying in 1s error=Low-level plotting error: Timed out without ping from plotter 2024-06-24T08:05:05.971455Z WARN {farm_index=0}:{sector_index=197}: subspace_farmer::single_disk_farm::plotting: Failed to plot sector, retrying in 1s error=Low-level plotting error: Timed out without ping from plotter 2024-06-24T08:05:05.973650Z WARN {farm_index=0}:{sector_index=198}: subspace_farmer::single_disk_farm::plotting: Failed to plot sector, retrying in 1s error=Low-level plotting error: Timed out without ping from plotter 2024-06-24T08:05:05.974544Z WARN {farm_index=0}:{sector_index=199}: subspace_farmer::single_disk_farm::plotting: Failed to plot sector, retrying in 1s error=Low-level plotting error: Timed out without ping from plotter 2024-06-24T08:05:05.976655Z WARN {farm_index=0}:{sector_index=201}: subspace_farmer::single_disk_farm::plotting: Failed to plot sector, retrying in 1s error=Low-level plotting error: Timed out without ping from plotter 2024-06-24T08:05:07.687240Z WARN {farm_index=0}:{sector_index=202}: subspace_farmer::single_disk_farm::plotting: Failed to plot sector, retrying in 1s error=Low-level plotting error: Timed out without ping from plotter 2024-06-24T08:05:08.838602Z WARN {farm_index=0}:{sector_index=204}: subspace_farmer::single_disk_farm::plotting: Failed to plot sector, retrying in 1s error=Low-level plotting error: Timed out without ping from plotter 2024-06-24T08:05:45.546573Z WARN {farm_index=0}:{sector_index=192}: subspace_farmer::single_disk_farm::plotting: Failed to plot sector, retrying in 1s error=Low-level plotting error: Timed out without ping from plotter 2024-06-24T08:05:47.688083Z WARN {farm_index=0}:{sector_index=203}: subspace_farmer::single_disk_farm::plotting: Failed to plot sector, retrying in 1s error=Low-level plotting error: Timed out without ping from plotter ``` ---
**Author:** ![nazar-pc](https://yyz1.discourse-cdn.com/flex011/user_avatar/forum.autonomys.xyz/nazar-pc/32/6_2.png) [@nazar-pc](https://forum.autonomys.xyz/u/nazar-pc)\ **Post date:** [June 24, 2024, 11:23am UTC](https://forum.autonomys.xyz/t/jun-18-subspace-farmer-plotter-ping-timeout/4283/2 "2024-06-24T11:23:27Z")
Looks like something somewhere might be overloaded. What does network bandwidth look like? It would also be helpful to know what and how is running here, see [Timed out without ping from plotter](https://forum.autonomys.xyz/t/timed-out-without-ping-from-plotter/4262) for a great example of reporting with important details present. There is also a test build there that may potentially help. ---
**Author:** ![keens312](https://avatars.discourse-cdn.com/v4/letter/k/90ced4/32.png) [@keens312](https://forum.autonomys.xyz/u/keens312)\ **Post date:** [June 24, 2024, 12:12pm UTC](https://forum.autonomys.xyz/t/jun-18-subspace-farmer-plotter-ping-timeout/4283/3 "2024-06-24T12:12:08Z")
I have 10 servers running plotter process, and timeout occurs. But if I run only 2 plotters, no RPC timeout warnings. Is there a limitation on total number of connection on NATs? Can I run two or more NATS server in subspace cluster for my case? ---
**Author:** ![keens312](https://avatars.discourse-cdn.com/v4/letter/k/90ced4/32.png) [@keens312](https://forum.autonomys.xyz/u/keens312)\ **Post date:** [June 24, 2024, 12:21pm UTC](https://forum.autonomys.xyz/t/jun-18-subspace-farmer-plotter-ping-timeout/4283/4 "2024-06-24T12:21:45Z")
```auto [Paste any errors or relevant logs here] NATs Server log has warnings: [1] 2024/06/24 11:23:16.059252 [INF] 192.168.0.30:57484 - cid:407 - Slow Consumer Detected: WriteDeadline of 10s exceeded with 459 chunks of 29365342 total bytes. [1] 2024/06/24 11:23:44.826526 [INF] 192.168.0.37:36014 - cid:405 - Slow Consumer Detected: WriteDeadline of 10s exceeded with 394 chunks of 25170666 total bytes. [1] 2024/06/24 11:26:38.164135 [INF] 192.168.0.31:38454 - cid:408 - Slow Consumer Detected: WriteDeadline of 10s exceeded with 410 chunks of 26219335 total bytes. [1] 2024/06/24 11:26:57.873645 [INF] 192.168.0.32:38826 - cid:403 - Slow Consumer Detected: WriteDeadline of 10s exceeded with 378 chunks of 24122084 total bytes. [1] 2024/06/24 11:27:07.125728 [INF] 192.168.0.38:39604 - cid:411 - Slow Consumer Detected: WriteDeadline of 10s exceeded with 442 chunks of 28316499 total bytes. [1] 2024/06/24 11:27:40.774242 [INF] 192.168.0.33:48860 - cid:412 - Slow Consumer Detected: WriteDeadline of 10s exceeded with 507 chunks of 32511436 total bytes. [1] 2024/06/24 11:28:22.355598 [INF] 192.168.0.36:53310 - cid:413 - Slow Consumer Detected: WriteDeadline of 10s exceeded with 426 chunks of 27268091 total bytes. [1] 2024/06/24 11:30:29.342680 [INF] 192.168.0.35:59254 - cid:398 - Slow Consumer Detected: WriteDeadline of 10s exceeded with 507 chunks of 32511436 total bytes. [1] 2024/06/24 11:30:30.645076 [INF] 192.168.0.37:36036 - cid:415 - Slow Consumer Detected: WriteDeadline of 10s exceeded with 378 chunks of 24122084 total bytes. [1] 2024/06/24 11:31:40.310326 [INF] 192.168.0.34:58016 - cid:410 - Slow Consumer Detected: WriteDeadline of 10s exceeded with 443 chunks of 28316673 total bytes. [1] 2024/06/24 11:32:16.798575 [INF] 192.168.0.38:39622 - cid:418 - Slow Consumer Detected: WriteDeadline of 10s exceeded with 442 chunks of 28316499 total bytes. [1] 2024/06/24 11:32:19.370884 [INF] 192.168.0.32:38864 - cid:417 - Slow Consumer Detected: WriteDeadline of 10s exceeded with 394 chunks of 25170816 total bytes. [1] 2024/06/24 11:32:54.124996 [INF] 192.168.0.33:48876 - cid:419 - Slow Consumer Detected: WriteDeadline of 10s exceeded with 507 chunks of 32511436 total bytes. [1] 2024/06/24 11:36:51.679004 [INF] 192.168.0.34:58046 - cid:423 - Slow Consumer Detected: WriteDeadline of 10s exceeded with 507 chunks of 32511436 total bytes. [1] 2024/06/24 11:37:06.576659 [INF] 192.168.0.37:36054 - cid:422 - Slow Consumer Detected: WriteDeadline of 10s exceeded with 507 chunks of 32511436 total bytes. [1] 2024/06/24 11:37:36.164943 [INF] 192.168.0.30:57498 - cid:414 - Slow Consumer Detected: WriteDeadline of 10s exceeded with 394 chunks of 25170666 total bytes. [1] 2024/06/24 11:38:10.614318 [INF] 192.168.0.33:48894 - cid:426 - Slow Consumer Detected: WriteDeadline of 10s exceeded with 507 chunks of 32511436 total bytes. [1] 2024/06/24 11:40:41.376313 [INF] 192.168.0.32:38878 - cid:425 - Slow Consumer Detected: WriteDeadline of 10s exceeded with 507 chunks of 32511436 total bytes. [1] 2024/06/24 11:42:07.500396 [INF] 192.168.0.36:53328 - cid:420 - Slow Consumer Detected: WriteDeadline of 10s exceeded with 507 chunks of 32511436 total bytes. [1] 2024/06/24 11:42:33.598518 [INF] 192.168.0.37:36076 - cid:428 - Slow Consumer Detected: WriteDeadline of 10s exceeded with 507 chunks of 32511436 total bytes. [1] 2024/06/24 11:43:14.623524 [INF] 192.168.0.30:57540 - cid:429 - Slow Consumer Detected: WriteDeadline of 10s exceeded with 509 chunks of 32512196 total bytes. [1] 2024/06/24 11:45:51.549990 [INF] 192.168.0.32:38904 - cid:431 - Slow Consumer Detected: WriteDeadline of 10s exceeded with 491 chunks of 31462680 total bytes. [1] 2024/06/24 11:46:08.958069 [INF] 192.168.0.29:43032 - cid:409 - Slow Consumer Detected: WriteDeadline of 10s exceeded with 507 chunks of 32511436 total bytes. [1] 2024/06/24 11:46:09.078952 [INF] 192.168.0.34:58060 - cid:427 - Slow Consumer Detected: WriteDeadline of 10s exceeded with 459 chunks of 29365429 total bytes. [1] 2024/06/24 11:52:40.660827 [INF] 192.168.0.31:38480 - cid:416 - Slow Consumer Detected: WriteDeadline of 10s exceeded with 507 chunks of 32511436 total bytes. [1] 2024/06/24 12:14:16.017898 [INF] 192.168.0.29:43112 - cid:440 - Slow Consumer Detected: WriteDeadline of 10s exceeded with 475 chunks of 30413924 total bytes. [1] 2024/06/24 12:14:16.367959 [INF] 192.168.0.31:38608 - cid:444 - Slow Consumer Detected: WriteDeadline of 10s exceeded with 507 chunks of 32511436 total bytes. [1] 2024/06/24 12:15:19.751013 [INF] 192.168.0.29:43190 - cid:447 - Slow Consumer Detected: WriteDeadline of 10s exceeded with 507 chunks of 32511436 total bytes. [1] 2024/06/24 12:15:24.048282 [INF] 192.168.0.32:38996 - cid:443 - Slow Consumer Detected: WriteDeadline of 10s exceeded with 491 chunks of 31462680 total bytes. ``` ---
**Author:** ![nazar-pc](https://yyz1.discourse-cdn.com/flex011/user_avatar/forum.autonomys.xyz/nazar-pc/32/6_2.png) [@nazar-pc](https://forum.autonomys.xyz/u/nazar-pc)\ **Post date:** [June 24, 2024, 8:10pm UTC](https://forum.autonomys.xyz/t/jun-18-subspace-farmer-plotter-ping-timeout/4283/5 "2024-06-24T20:10:22Z")
Those timeouts seem to indicate that readers of messages can’t process them in time. What hardware is all this running on, do you see network bandwidth pegged on NATS server to the max by any chance? How are those machines connected to each other? I need more information to understand what is going on. > [@keens312](#): > > Is there a limitation on total number of connection on NATs? > Can I run two or more NATS server in subspace cluster for my case? There is no inherent limitation except hardware capabilities, NATS should be able to scale with CPU cores and network speed. You can run more than one and it might help depending on network topology, just make sure to create a cluster and not multiple independent NATS servers or you will have issues: [Clustering | NATS Docs](https://docs.nats.io/running-a-nats-service/configuration/clustering) ---
**Author:** ![keens312](https://avatars.discourse-cdn.com/v4/letter/k/90ced4/32.png) [@keens312](https://forum.autonomys.xyz/u/keens312)\ **Post date:** [June 26, 2024, 12:10am UTC](https://forum.autonomys.xyz/t/jun-18-subspace-farmer-plotter-ping-timeout/4283/6 "2024-06-26T00:10:58Z")
The network bandwidth is close to 120MB/s bottleneck when plotter finished a sector. The timeout issue is fixed when I move NATS service from node to farmer node. It seems that NATS should avoid to run with node, since both of them have network requirement. ---
**Author:** ![nazar-pc](https://yyz1.discourse-cdn.com/flex011/user_avatar/forum.autonomys.xyz/nazar-pc/32/6_2.png) [@nazar-pc](https://forum.autonomys.xyz/u/nazar-pc)\ **Post date:** [June 26, 2024, 4:43pm UTC](https://forum.autonomys.xyz/t/jun-18-subspace-farmer-plotter-ping-timeout/4283/7 "2024-06-26T16:43:47Z")
This is a helpful information, thank you! I think there is still opportunity to opimize bandwidth spikes further, I’ll look into it going forward. ---
**Author:** ![Quanjun\_Yu](https://yyz1.discourse-cdn.com/flex011/user_avatar/forum.autonomys.xyz/quanjun_yu/32/2089_2.png) [@Quanjun\_Yu](https://forum.autonomys.xyz/u/Quanjun_Yu)\ **Post date:** [June 26, 2024, 8:11pm UTC](https://forum.autonomys.xyz/t/jun-18-subspace-farmer-plotter-ping-timeout/4283/8 "2024-06-26T20:11:28Z")
Move NATS service from node to farmer node, it did fix the RPC timeout issue but the plotting speed is much slower than before. About 4 mins each sector when nats runs along with node,controller and cache. when nats runs with only farmer, about 40 mins or more for each sector to plot. Is there any constrains for subspace cluster services? Should nats must run with controller and cache at the same server for performance?