Issue Report
Farmer log keep showing plotter timeout
Environment
- Operating System: Ubuntu 18.04
- Pulsar/Advanced CLI/Docker: CLI
Problem
Farmer keep reporting plotter timeout. This leads to unable to plot at all.
Since Jun-24 found this issue and everything works fine before.
[Paste any errors or relevant logs here]
node@node-MS-7D73:~/subspace$ ./subspace-farmer-ubuntu-x86_64-skylake-gemini-3h-2024-jun-18 cluster --nats-server nats://192.168.0.39:4222 farmer --reward-address stXXX path=/mnt/sda1,size=3.3TiB 2024-06-24T07:59:39.715548Z INFO subspace_farmer::cluster::controller: Downloading all segment headers from controller...
2024-06-24T07:59:39.715617Z INFO async_nats: event: connected
2024-06-24T07:59:39.757011Z INFO subspace_farmer::cluster::controller: Downloaded all segment headers from node successfully
2024-06-24T07:59:40.359632Z INFO {farm_index=0}: subspace_farmer::single_disk_farm::plot_cache: Checking plot cache contents, this can take a while
2024-06-24T07:59:40.360899Z INFO {farm_index=0}: subspace_farmer::single_disk_farm::plot_cache: Finished checking plot cache contents
2024-06-24T07:59:40.362788Z INFO {farm_index=0}: subspace_farmer::single_disk_farm: Benchmarking faster proving method
2024-06-24T07:59:47.998540Z INFO {farm_index=0}: subspace_farmer::single_disk_farm: Faster proving method found fastest_mode=ConcurrentChunks
2024-06-24T07:59:48.071880Z INFO {farm_index=0}: subspace_farmer::commands::cluster::farmer: ID: 01J064A906XK3GT42E4FA676XF2024-06-24T07:59:48.071885Z INFO {farm_index=0}: subspace_farmer::commands::cluster::farmer: Genesis hash: 0x0c121c75f4ef450f40619e1fca9d1e8e7fbabc42c895bc4790801e85d5a91c342024-06-24T07:59:48.071896Z INFO {farm_index=0}: subspace_farmer::commands::cluster::farmer: Public key: 0xee4c831be6acf47d9c6552aa8f15155caa1051a635798098b6a324c24d6bb97c2024-06-24T07:59:48.071904Z INFO {farm_index=0}: subspace_farmer::commands::cluster::farmer: Allocated space: 3.3 TiB (3.6 TB)2024-06-24T07:59:48.071908Z INFO {farm_index=0}: subspace_farmer::commands::cluster::farmer: Directory: /mnt/sda12024-06-24T07:59:48.102218Z INFO {farm_index=0}: subspace_farmer::single_disk_farm::plotting: Subscribing to archived segments2024-06-24T07:59:48.102265Z INFO {farm_index=0}: subspace_farmer::single_disk_farm::farming: Subscribing to slot info notificationsng notifications2024-06-24T07:59:48.126596Z INFO {farm_index=0}:{sector_index=185}: subspace_farmer::single_disk_farm::plotting: Plotting sector (5.39% complete)2024-06-24T08:00:49.928957Z INFO {farm_index=0}:{sector_index=186}: subspace_farmer::single_disk_farm::plotting: Plotting sector (5.42% complete)2024-06-24T08:00:49.945535Z INFO {farm_index=0}:{sector_index=187}: subspace_farmer::single_disk_farm::plotting: Plotting sector (5.45% complete)2024-06-24T08:00:49.962459Z INFO {farm_index=0}:{sector_index=188}: subspace_farmer::single_disk_farm::plotting: Plotting sector (5.48% complete)2024-06-24T08:00:49.979279Z INFO {farm_index=0}:{sector_index=189}: subspace_farmer::single_disk_farm::plotting: Plotting sector (5.51% complete)2024-06-24T08:00:49.996013Z INFO {farm_index=0}:{sector_index=190}: subspace_farmer::single_disk_farm::plotting: Plotting sector (5.54% complete)2024-06-24T08:00:50.013163Z INFO {farm_index=0}:{sector_index=191}: subspace_farmer::single_disk_farm::plotting: Plotting sector (5.57% complete)2024-06-24T08:00:50.651375Z INFO {farm_index=0}:{sector_index=192}: subspace_farmer::single_disk_farm::plotting: Plotting sector (5.59% complete)2024-06-24T08:00:50.695628Z INFO {farm_index=0}:{sector_index=195}: subspace_farmer::single_disk_farm::plotting: Plotting sector (5.68% complete)2024-06-24T08:00:50.742299Z INFO {farm_index=0}:{sector_index=198}: subspace_farmer::single_disk_farm::plotting: Plotting sector (5.77% complete)2024-06-24T08:00:51.057832Z INFO {farm_index=0}:{sector_index=199}: subspace_farmer::single_disk_farm::plotting: Plotting sector (5.80% complete)2024-06-24T08:00:51.074374Z INFO {farm_index=0}:{sector_index=200}: subspace_farmer::single_disk_farm::plotting: Plotting sector (5.83% complete)2024-06-24T08:00:51.539351Z INFO {farm_index=0}:{sector_index=201}: subspace_farmer::single_disk_farm::plotting: Plotting sector (5.86% complete)2024-06-24T08:00:51.555016Z INFO {farm_index=0}:{sector_index=202}: subspace_farmer::single_disk_farm::plotting: Plotting sector (5.89% complete)^C2024-06-24T08:01:48.457893Z INFO subspace_farmer::utils: Received SIGINT, shutting down farmer...node@node-MS-7D73:~/subspace$ ./subspace-farmer-ubuntu-x86_64-skylake-gemini-3h-2024-jun-18 cluster --nats-server nats://192.168.0.39:4222 farmer --reward-address st7Qmat3inHZtQojUS6w1BoBNKH2Hef6zjtPRRQgEcBVF6J4A path=/mnt/sda1,size=3.3TiB2024-06-24T08:01:49.520810Z INFO subspace_farmer::cluster::controller: Downloading all segment headers from controller...2024-06-24T08:01:49.520847Z INFO async_nats: event: connected2024-06-24T08:01:49.544495Z INFO subspace_farmer::cluster::controller: Downloaded all segment headers from node successfully2024-06-24T08:01:50.060371Z INFO {farm_index=0}: subspace_farmer::single_disk_farm::plot_cache: Checking plot cache contents, this can take a while2024-06-24T08:01:50.061394Z INFO {farm_index=0}: subspace_farmer::single_disk_farm::plot_cache: Finished checking plot cache contents2024-06-24T08:01:50.063108Z INFO {farm_index=0}: subspace_farmer::single_disk_farm: Benchmarking faster proving method
^Cnode@node-MS-7D73:~/subspace$ ./subspace-farmer-ubuntu-x86_64-skylake-gemini-3h-2024-jun-18 cluster --nats-server nats://192.168.0.39:4222 farmer --reward-address st7Qmat3inHZtQojUS6w1BoBNKH2Hef6zjtPRRQgEcBVF6J4A path=/mnt/sda1,size=3.3TiB2024-06-24T08:02:04.768352Z INFO subspace_farmer::cluster::controller: Downloading all segment headers from controller...2024-06-24T08:02:04.768374Z INFO async_nats: event: connected2024-06-24T08:02:04.791665Z INFO subspace_farmer::cluster::controller: Downloaded all segment headers from node successfully2024-06-24T08:02:05.389814Z INFO {farm_index=0}: subspace_farmer::single_disk_farm::plot_cache: Checking plot cache contents, this can take a while2024-06-24T08:02:05.391064Z INFO {farm_index=0}: subspace_farmer::single_disk_farm::plot_cache: Finished checking plot cache contents2024-06-24T08:02:05.392845Z INFO {farm_index=0}: subspace_farmer::single_disk_farm: Benchmarking faster proving method
2024-06-24T08:02:13.186912Z INFO {farm_index=0}: subspace_farmer::single_disk_farm: Faster proving method found fastest_mode=ConcurrentChunks
2024-06-24T08:02:13.242884Z INFO {farm_index=0}: subspace_farmer::commands::cluster::farmer: Farm 0:2024-06-24T08:02:13.242897Z INFO {farm_index=0}: subspace_farmer::commands::cluster::farmer: ID: 01J064A906XK3GT42E4FA676XF2024-06-24T08:02:13.242903Z INFO {farm_index=0}: subspace_farmer::commands::cluster::farmer: Genesis hash: 0x0c121c75f4ef450f40619e1fca9d1e8e7fbabc42c895bc4790801e85d5a91c342024-06-24T08:02:13.242909Z INFO {farm_index=0}: subspace_farmer::commands::cluster::farmer: Public key: 0xee4c831be6acf47d9c6552aa8f15155caa1051a635798098b6a324c24d6bb97c2024-06-24T08:02:13.242923Z INFO {farm_index=0}: subspace_farmer::commands::cluster::farmer: Allocated space: 3.3 TiB (3.6 TB)
2024-06-24T08:02:13.242928Z INFO {farm_index=0}: subspace_farmer::commands::cluster::farmer: Directory: /mnt/sda1
2024-06-24T08:02:13.274021Z INFO {farm_index=0}: subspace_farmer::single_disk_farm::plotting: Subscribing to archived segments
2024-06-24T08:02:13.274069Z INFO {farm_index=0}: subspace_farmer::single_disk_farm::farming: Subscribing to slot info notifications
2024-06-24T08:02:13.274081Z INFO {farm_index=0}: subspace_farmer::reward_signing: Subscribing to reward signing notifications
2024-06-24T08:02:13.291113Z INFO {farm_index=0}:{sector_index=185}: subspace_farmer::single_disk_farm::plotting: Plotting sector (5.39% complete)
2024-06-24T08:02:13.797141Z INFO {farm_index=0}:{sector_index=186}: subspace_farmer::single_disk_farm::plotting: Plotting sector (5.42% complete)
2024-06-24T08:02:14.267570Z INFO {farm_index=0}:{sector_index=187}: subspace_farmer::single_disk_farm::plotting: Plotting sector (5.45% complete)
2024-06-24T08:02:14.737584Z INFO {farm_index=0}:{sector_index=188}: subspace_farmer::single_disk_farm::plotting: Plotting sector (5.48% complete)
2024-06-24T08:02:15.204026Z INFO {farm_index=0}:{sector_index=189}: subspace_farmer::single_disk_farm::plotting: Plotting sector (5.51% complete)
2024-06-24T08:02:15.535879Z INFO {farm_index=0}:{sector_index=190}: subspace_farmer::single_disk_farm::plotting: Plotting sector (5.54% complete)
2024-06-24T08:02:15.537588Z INFO {farm_index=0}:{sector_index=191}: subspace_farmer::single_disk_farm::plotting: Plotting sector (5.57% complete)
2024-06-24T08:02:15.539312Z INFO {farm_index=0}:{sector_index=192}: subspace_farmer::single_disk_farm::plotting: Plotting sector (5.59% complete)
2024-06-24T08:02:15.541028Z INFO {farm_index=0}:{sector_index=193}: subspace_farmer::single_disk_farm::plotting: Plotting sector (5.62% complete)
2024-06-24T08:02:15.542486Z INFO {farm_index=0}:{sector_index=194}: subspace_farmer::single_disk_farm::plotting: Plotting sector (5.65% complete)
2024-06-24T08:02:15.544107Z INFO {farm_index=0}:{sector_index=195}: subspace_farmer::single_disk_farm::plotting: Plotting sector (5.68% complete)
2024-06-24T08:02:15.545275Z INFO {farm_index=0}:{sector_index=196}: subspace_farmer::single_disk_farm::plotting: Plotting sector (5.71% complete)
2024-06-24T08:02:15.963604Z INFO {farm_index=0}:{sector_index=197}: subspace_farmer::single_disk_farm::plotting: Plotting sector (5.74% complete)
2024-06-24T08:02:15.965468Z INFO {farm_index=0}:{sector_index=198}: subspace_farmer::single_disk_farm::plotting: Plotting sector (5.77% complete)
2024-06-24T08:02:15.966964Z INFO {farm_index=0}:{sector_index=199}: subspace_farmer::single_disk_farm::plotting: Plotting sector (5.80% complete)
2024-06-24T08:02:15.968312Z INFO {farm_index=0}:{sector_index=200}: subspace_farmer::single_disk_farm::plotting: Plotting sector (5.83% complete)
2024-06-24T08:02:15.969813Z INFO {farm_index=0}:{sector_index=201}: subspace_farmer::single_disk_farm::plotting: Plotting sector (5.86% complete)
2024-06-24T08:02:17.678285Z INFO {farm_index=0}:{sector_index=202}: subspace_farmer::single_disk_farm::plotting: Plotting sector (5.89% complete)
2024-06-24T08:02:17.679932Z INFO {farm_index=0}:{sector_index=203}: subspace_farmer::single_disk_farm::plotting: Plotting sector (5.91% complete)
2024-06-24T08:02:18.830132Z INFO {farm_index=0}:{sector_index=204}: subspace_farmer::single_disk_farm::plotting: Plotting sector (5.94% complete)
2024-06-24T08:05:03.614223Z WARN {farm_index=0}:{sector_index=185}: subspace_farmer::single_disk_farm::plotting: Failed to plot sector, retrying in 1s error=Low-level plotting error: Timed out without ping from plotter
2024-06-24T08:05:04.103522Z WARN {farm_index=0}:{sector_index=186}: subspace_farmer::single_disk_farm::plotting: Failed to plot sector, retrying in 1s error=Low-level plotting error: Timed out without ping from plotter
2024-06-24T08:05:04.572873Z WARN {farm_index=0}:{sector_index=187}: subspace_farmer::single_disk_farm::plotting: Failed to plot sector, retrying in 1s error=Low-level plotting error: Timed out without ping from plotter
2024-06-24T08:05:05.038472Z WARN {farm_index=0}:{sector_index=188}: subspace_farmer::single_disk_farm::plotting: Failed to plot sector, retrying in 1s error=Low-level plotting error: Timed out without ping from plotter
2024-06-24T08:05:05.506776Z WARN {farm_index=0}:{sector_index=189}: subspace_farmer::single_disk_farm::plotting: Failed to plot sector, retrying in 1s error=Low-level plotting error: Timed out without ping from plotter
2024-06-24T08:05:05.545246Z WARN {farm_index=0}:{sector_index=191}: subspace_farmer::single_disk_farm::plotting: Failed to plot sector, retrying in 1s error=Low-level plotting error: Timed out without ping from plotter
2024-06-24T08:05:05.545273Z WARN {farm_index=0}:{sector_index=190}: subspace_farmer::single_disk_farm::plotting: Failed to plot sector, retrying in 1s error=Low-level plotting error: Timed out without ping from plotter
2024-06-24T08:05:05.549126Z WARN {farm_index=0}:{sector_index=193}: subspace_farmer::single_disk_farm::plotting: Failed to plot sector, retrying in 1s error=Low-level plotting error: Timed out without ping from plotter
2024-06-24T08:05:05.551341Z WARN {farm_index=0}:{sector_index=194}: subspace_farmer::single_disk_farm::plotting: Failed to plot sector, retrying in 1s error=Low-level plotting error: Timed out without ping from plotter
2024-06-24T08:05:05.552782Z WARN {farm_index=0}:{sector_index=196}: subspace_farmer::single_disk_farm::plotting: Failed to plot sector, retrying in 1s error=Low-level plotting error: Timed out without ping from plotter
2024-06-24T08:05:05.971455Z WARN {farm_index=0}:{sector_index=197}: subspace_farmer::single_disk_farm::plotting: Failed to plot sector, retrying in 1s error=Low-level plotting error: Timed out without ping from plotter
2024-06-24T08:05:05.973650Z WARN {farm_index=0}:{sector_index=198}: subspace_farmer::single_disk_farm::plotting: Failed to plot sector, retrying in 1s error=Low-level plotting error: Timed out without ping from plotter
2024-06-24T08:05:05.974544Z WARN {farm_index=0}:{sector_index=199}: subspace_farmer::single_disk_farm::plotting: Failed to plot sector, retrying in 1s error=Low-level plotting error: Timed out without ping from plotter
2024-06-24T08:05:05.976655Z WARN {farm_index=0}:{sector_index=201}: subspace_farmer::single_disk_farm::plotting: Failed to plot sector, retrying in 1s error=Low-level plotting error: Timed out without ping from plotter
2024-06-24T08:05:07.687240Z WARN {farm_index=0}:{sector_index=202}: subspace_farmer::single_disk_farm::plotting: Failed to plot sector, retrying in 1s error=Low-level plotting error: Timed out without ping from plotter
2024-06-24T08:05:08.838602Z WARN {farm_index=0}:{sector_index=204}: subspace_farmer::single_disk_farm::plotting: Failed to plot sector, retrying in 1s error=Low-level plotting error: Timed out without ping from plotter
2024-06-24T08:05:45.546573Z WARN {farm_index=0}:{sector_index=192}: subspace_farmer::single_disk_farm::plotting: Failed to plot sector, retrying in 1s error=Low-level plotting error: Timed out without ping from plotter
2024-06-24T08:05:47.688083Z WARN {farm_index=0}:{sector_index=203}: subspace_farmer::single_disk_farm::plotting: Failed to plot sector, retrying in 1s error=Low-level plotting error: Timed out without ping from plotter