Bug report - node went out of sync when a remote farmer (with all finished plots) replotted

Support
1

One of my node went out of sync for no reason when a farmer with all finished plots went to replotting phase all at once. Below is the node’s log.

It seems the farmer was disconnected and connected all over again, causing the node to go out of sync.

Summary 
2024-04-15T01:40:43.375873Z  INFO Consensus: substrate: ✨ Imported #←[1;37m1084673←[0m (0xa26b…9c1b)
2024-04-15T01:40:47.434323Z  INFO Consensus: substrate: 💤 Idle (40 peers), best: #1084673 (0xa26b…9c1b), finalized #988889 (0x01de…b547), ⬇ 67.3kiB/s ⬆ 56.6kiB/s
2024-04-15T01:40:52.442369Z  INFO Consensus: substrate: 💤 Idle (40 peers), best: #1084673 (0xa26b…9c1b), finalized #988889 (0x01de…b547), ⬇ 28.0kiB/s ⬆ 31.8kiB/s
2024-04-15T01:40:57.447998Z  INFO Consensus: substrate: 💤 Idle (40 peers), best: #1084673 (0xa26b…9c1b), finalized #988889 (0x01de…b547), ⬇ 34.4kiB/s ⬆ 44.8kiB/s
2024-04-15T01:41:02.454358Z  INFO Consensus: substrate: 💤 Idle (40 peers), best: #1084673 (0xa26b…9c1b), finalized #988889 (0x01de…b547), ⬇ 34.1kiB/s ⬆ 45.7kiB/s
2024-04-15T01:41:07.459183Z  INFO Consensus: substrate: 💤 Idle (40 peers), best: #1084673 (0xa26b…9c1b), finalized #988889 (0x01de…b547), ⬇ 42.9kiB/s ⬆ 55.2kiB/s
2024-04-15T01:41:12.461320Z  INFO Consensus: substrate: 💤 Idle (40 peers), best: #1084673 (0xa26b…9c1b), finalized #988889 (0x01de…b547), ⬇ 54.9kiB/s ⬆ 70.4kiB/s
2024-04-15T01:41:17.463276Z  INFO Consensus: substrate: 💤 Idle (40 peers), best: #1084673 (0xa26b…9c1b), finalized #988889 (0x01de…b547), ⬇ 32.7kiB/s ⬆ 46.2kiB/s
2024-04-15T01:41:22.477341Z  INFO Consensus: substrate: 💤 Idle (40 peers), best: #1084673 (0xa26b…9c1b), finalized #988889 (0x01de…b547), ⬇ 20.1kiB/s ⬆ 24.0kiB/s
2024-04-15T01:41:27.484433Z  INFO Consensus: substrate: 💤 Idle (40 peers), best: #1084673 (0xa26b…9c1b), finalized #988889 (0x01de…b547), ⬇ 30.2kiB/s ⬆ 48.8kiB/s
2024-04-15T01:41:32.487396Z  INFO Consensus: substrate: 💤 Idle (40 peers), best: #1084673 (0xa26b…9c1b), finalized #988889 (0x01de…b547), ⬇ 33.3kiB/s ⬆ 27.1kiB/s
2024-04-15T01:41:37.496642Z  INFO Consensus: substrate: 💤 Idle (40 peers), best: #1084673 (0xa26b…9c1b), finalized #988889 (0x01de…b547), ⬇ 29.7kiB/s ⬆ 38.4kiB/s
2024-04-15T01:41:40.621071Z ERROR connection{remote_addr=192.168.2.200:49736 conn_id=390}: jsonrpsee_server::transport::ws: WS transport error: i/o error: An existing connection was forcibly closed by the remote host. (os error 10054); terminate connection: 390
2024-04-15T01:41:40.621340Z ERROR connection{remote_addr=192.168.2.200:49737 conn_id=392}: jsonrpsee_server::transport::ws: WS transport error: i/o error: An existing connection was forcibly closed by the remote host. (os error 10054); terminate connection: 392
2024-04-15T01:41:40.621491Z ERROR connection{remote_addr=192.168.2.200:49740 conn_id=395}: jsonrpsee_server::transport::ws: WS transport error: i/o error: An existing connection was forcibly closed by the remote host. (os error 10054); terminate connection: 395
2024-04-15T01:41:40.621498Z ERROR connection{remote_addr=192.168.2.200:49738 conn_id=391}: jsonrpsee_server::transport::ws: WS transport error: i/o error: An existing connection was forcibly closed by the remote host. (os error 10054); terminate connection: 391
2024-04-15T01:41:40.621611Z ERROR connection{remote_addr=192.168.2.200:49745 conn_id=399}: jsonrpsee_server::transport::ws: WS transport error: i/o error: An existing connection was forcibly closed by the remote host. (os error 10054); terminate connection: 399
2024-04-15T01:41:40.621543Z ERROR connection{remote_addr=192.168.2.200:49739 conn_id=393}: jsonrpsee_server::transport::ws: WS transport error: i/o error: An existing connection was forcibly closed by the remote host. (os error 10054); terminate connection: 393
2024-04-15T01:41:40.621709Z ERROR connection{remote_addr=192.168.2.200:49746 conn_id=400}: jsonrpsee_server::transport::ws: WS transport error: i/o error: An existing connection was forcibly closed by the remote host. (os error 10054); terminate connection: 400
2024-04-15T01:41:40.621624Z ERROR connection{remote_addr=192.168.2.200:49743 conn_id=397}: jsonrpsee_server::transport::ws: WS transport error: i/o error: An existing connection was forcibly closed by the remote host. (os error 10054); terminate connection: 397
2024-04-15T01:41:40.621627Z ERROR connection{remote_addr=192.168.2.200:49741 conn_id=394}: jsonrpsee_server::transport::ws: WS transport error: i/o error: An existing connection was forcibly closed by the remote host. (os error 10054); terminate connection: 394
2024-04-15T01:41:40.621671Z ERROR connection{remote_addr=192.168.2.200:49742 conn_id=396}: jsonrpsee_server::transport::ws: WS transport error: i/o error: An existing connection was forcibly closed by the remote host. (os error 10054); terminate connection: 396
2024-04-15T01:41:40.621697Z ERROR connection{remote_addr=192.168.2.200:49750 conn_id=404}: jsonrpsee_server::transport::ws: WS transport error: i/o error: An existing connection was forcibly closed by the remote host. (os error 10054); terminate connection: 404
2024-04-15T01:41:40.621498Z ERROR connection{remote_addr=192.168.2.200:49744 conn_id=398}: jsonrpsee_server::transport::ws: WS transport error: i/o error: An existing connection was forcibly closed by the remote host. (os error 10054); terminate connection: 398
2024-04-15T01:41:40.621713Z ERROR connection{remote_addr=192.168.2.200:49748 conn_id=402}: jsonrpsee_server::transport::ws: WS transport error: i/o error: An existing connection was forcibly closed by the remote host. (os error 10054); terminate connection: 402
2024-04-15T01:41:40.621745Z ERROR connection{remote_addr=192.168.2.200:49747 conn_id=401}: jsonrpsee_server::transport::ws: WS transport error: i/o error: An existing connection was forcibly closed by the remote host. (os error 10054); terminate connection: 401
2024-04-15T01:41:40.621764Z ERROR connection{remote_addr=192.168.2.200:49749 conn_id=403}: jsonrpsee_server::transport::ws: WS transport error: i/o error: An existing connection was forcibly closed by the remote host. (os error 10054); terminate connection: 403
2024-04-15T01:41:40.621834Z ERROR jsonrpsee_server::transport::ws: WS transport error: send failed: Networking or low-level protocol error: connection closed
2024-04-15T01:41:41.060030Z  INFO connection{remote_addr=192.168.2.200:59749 conn_id=435}: jsonrpsee_server::server: Accepting new connection 126/200
2024-04-15T01:41:41.494309Z  INFO connection{remote_addr=192.168.2.200:59752 conn_id=436}: jsonrpsee_server::server: Accepting new connection 127/200
2024-04-15T01:41:41.494520Z  INFO connection{remote_addr=192.168.2.200:59751 conn_id=437}: jsonrpsee_server::server: Accepting new connection 128/200
2024-04-15T01:41:41.494619Z  INFO connection{remote_addr=192.168.2.200:59753 conn_id=438}: jsonrpsee_server::server: Accepting new connection 129/200
2024-04-15T01:41:41.494756Z  INFO connection{remote_addr=192.168.2.200:59754 conn_id=439}: jsonrpsee_server::server: Accepting new connection 130/200
2024-04-15T01:41:41.494877Z  INFO connection{remote_addr=192.168.2.200:59756 conn_id=440}: jsonrpsee_server::server: Accepting new connection 131/200
2024-04-15T01:41:41.495009Z  INFO connection{remote_addr=192.168.2.200:59755 conn_id=441}: jsonrpsee_server::server: Accepting new connection 132/200
2024-04-15T01:41:41.495124Z  INFO connection{remote_addr=192.168.2.200:59757 conn_id=442}: jsonrpsee_server::server: Accepting new connection 133/200
2024-04-15T01:41:41.495246Z  INFO connection{remote_addr=192.168.2.200:59758 conn_id=443}: jsonrpsee_server::server: Accepting new connection 134/200
2024-04-15T01:41:41.495360Z  INFO connection{remote_addr=192.168.2.200:59760 conn_id=444}: jsonrpsee_server::server: Accepting new connection 135/200
2024-04-15T01:41:41.495477Z  INFO connection{remote_addr=192.168.2.200:59759 conn_id=445}: jsonrpsee_server::server: Accepting new connection 136/200
2024-04-15T01:41:41.495603Z  INFO connection{remote_addr=192.168.2.200:59762 conn_id=446}: jsonrpsee_server::server: Accepting new connection 137/200
2024-04-15T01:41:41.495699Z  INFO connection{remote_addr=192.168.2.200:59761 conn_id=447}: jsonrpsee_server::server: Accepting new connection 138/200
2024-04-15T01:41:41.495794Z  INFO connection{remote_addr=192.168.2.200:59763 conn_id=448}: jsonrpsee_server::server: Accepting new connection 139/200
2024-04-15T01:41:41.495914Z  INFO connection{remote_addr=192.168.2.200:59764 conn_id=449}: jsonrpsee_server::server: Accepting new connection 140/200
2024-04-15T01:41:42.500561Z  INFO Consensus: substrate: 💤 Idle (40 peers), best: #1084673 (0xa26b…9c1b), finalized #988889 (0x01de…b547), ⬇ 24.8kiB/s ⬆ 23.1kiB/s
2024-04-15T01:41:47.509312Z  INFO Consensus: substrate: 💤 Idle (40 peers), best: #1084673 (0xa26b…9c1b), finalized #988889 (0x01de…b547), ⬇ 32.9kiB/s ⬆ 19.7kiB/s
2024-04-15T01:41:52.511064Z  INFO Consensus: substrate: 💤 Idle (40 peers), best: #1084673 (0xa26b…9c1b), finalized #988889 (0x01de…b547), ⬇ 20.6kiB/s ⬆ 26.0kiB/s
2024-04-15T01:41:57.513111Z  INFO Consensus: substrate: 💤 Idle (40 peers), best: #1084673 (0xa26b…9c1b), finalized #988889 (0x01de…b547), ⬇ 31.4kiB/s ⬆ 25.9kiB/s
2024-04-15T01:42:02.515799Z  INFO Consensus: substrate: 💤 Idle (40 peers), best: #1084673 (0xa26b…9c1b), finalized #988889 (0x01de…b547), ⬇ 16.9kiB/s ⬆ 20.7kiB/s
2024-04-15T01:42:07.520533Z  INFO Consensus: substrate: 💤 Idle (40 peers), best: #1084673 (0xa26b…9c1b), finalized #988889 (0x01de…b547), ⬇ 23.0kiB/s ⬆ 28.3kiB/s
2024-04-15T01:42:12.531200Z  INFO Consensus: substrate: 💤 Idle (40 peers), best: #1084673 (0xa26b…9c1b), finalized #988889 (0x01de…b547), ⬇ 56.8kiB/s ⬆ 25.2kiB/s
2024-04-15T01:42:17.536182Z  INFO Consensus: substrate: 💤 Idle (40 peers), best: #1084673 (0xa26b…9c1b), finalized #988889 (0x01de…b547), ⬇ 41.9kiB/s ⬆ 19.0kiB/s
2024-04-15T01:42:22.546019Z  INFO Consensus: substrate: 💤 Idle (40 peers), best: #1084673 (0xa26b…9c1b), finalized #988889 (0x01de…b547), ⬇ 40.0kiB/s ⬆ 18.2kiB/s
2024-04-15T01:42:27.547695Z  INFO Consensus: substrate: 💤 Idle (40 peers), best: #1084673 (0xa26b…9c1b), finalized #988889 (0x01de…b547), ⬇ 26.4kiB/s ⬆ 18.5kiB/s
2024-04-15T01:42:32.553237Z  INFO Consensus: substrate: 💤 Idle (40 peers), best: #1084673 (0xa26b…9c1b), finalized #988889 (0x01de…b547), ⬇ 16.0kiB/s ⬆ 17.8kiB/s
2024-04-15T01:42:37.556479Z  INFO Consensus: substrate: 💤 Idle (40 peers), best: #1084673 (0xa26b…9c1b), finalized #988889 (0x01de…b547), ⬇ 18.8kiB/s ⬆ 15.2kiB/s
2024-04-15T01:42:42.558766Z  INFO Consensus: substrate: 💤 Idle (40 peers), best: #1084673 (0xa26b…9c1b), finalized #988889 (0x01de…b547), ⬇ 39.5kiB/s ⬆ 16.5kiB/s
2024-04-15T01:42:47.573354Z  INFO Consensus: substrate: 💤 Idle (40 peers), best: #1084673 (0xa26b…9c1b), finalized #988889 (0x01de…b547), ⬇ 21.3kiB/s ⬆ 20.4kiB/s
2024-04-15T01:42:52.584449Z  INFO Consensus: substrate: 💤 Idle (40 peers), best: #1084673 (0xa26b…9c1b), finalized #988889 (0x01de…b547), ⬇ 40.4kiB/s ⬆ 23.7kiB/s
2024-04-15T01:42:57.591928Z  INFO Consensus: substrate: 💤 Idle (40 peers), best: #1084673 (0xa26b…9c1b), finalized #988889 (0x01de…b547), ⬇ 40.0kiB/s ⬆ 14.4kiB/s
2024-04-15T01:43:02.595797Z  INFO Consensus: substrate: 💤 Idle (40 peers), best: #1084673 (0xa26b…9c1b), finalized #988889 (0x01de…b547), ⬇ 19.6kiB/s ⬆ 13.2kiB/s
2024-04-15T01:43:07.600962Z  INFO Consensus: substrate: 💤 Idle (40 peers), best: #1084673 (0xa26b…9c1b), finalized #988889 (0x01de…b547), ⬇ 35.4kiB/s ⬆ 17.2kiB/s
2024-04-15T01:43:12.609277Z  INFO Consensus: substrate: 💤 Idle (40 peers), best: #1084673 (0xa26b…9c1b), finalized #988889 (0x01de…b547), ⬇ 26.9kiB/s ⬆ 16.1kiB/s
2024-04-15T01:43:17.615859Z  INFO Consensus: substrate: 💤 Idle (40 peers), best: #1084673 (0xa26b…9c1b), finalized #988889 (0x01de…b547), ⬇ 57.3kiB/s ⬆ 17.9kiB/s
2024-04-15T01:43:22.624861Z  INFO Consensus: substrate: 💤 Idle (40 peers), best: #1084673 (0xa26b…9c1b), finalized #988889 (0x01de…b547), ⬇ 34.2kiB/s ⬆ 21.5kiB/s
2024-04-15T01:43:27.638718Z  INFO Consensus: substrate: 💤 Idle (40 peers), best: #1084673 (0xa26b…9c1b), finalized #988889 (0x01de…b547), ⬇ 16.8kiB/s ⬆ 16.5kiB/s
2024-04-15T01:43:32.643520Z  INFO Consensus: substrate: 💤 Idle (40 peers), best: #1084673 (0xa26b…9c1b), finalized #988889 (0x01de…b547), ⬇ 32.7kiB/s ⬆ 19.3kiB/s
2024-04-15T01:43:37.651483Z  INFO Consensus: substrate: 💤 Idle (40 peers), best: #1084673 (0xa26b…9c1b), finalized #988889 (0x01de…b547), ⬇ 22.5kiB/s ⬆ 12.7kiB/s
2024-04-15T01:43:42.660889Z  INFO Consensus: substrate: 💤 Idle (40 peers), best: #1084673 (0xa26b…9c1b), finalized #988889 (0x01de…b547), ⬇ 25.6kiB/s ⬆ 12.6kiB/s
2024-04-15T01:43:42.893857Z ERROR connection{remote_addr=192.168.2.200:59749 conn_id=435}: jsonrpsee_server::transport::ws: WS transport error: i/o error: An existing connection was forcibly closed by the remote host. (os error 10054); terminate connection: 435
2024-04-15T01:43:42.894138Z ERROR connection{remote_addr=192.168.2.200:59755 conn_id=441}: jsonrpsee_server::transport::ws: WS transport error: i/o error: An existing connection was forcibly closed by the remote host. (os error 10054); terminate connection: 441
2024-04-15T01:43:42.894157Z ERROR connection{remote_addr=192.168.2.200:59751 conn_id=437}: jsonrpsee_server::transport::ws: WS transport error: i/o error: An existing connection was forcibly closed by the remote host. (os error 10054); terminate connection: 437
2024-04-15T01:43:42.894271Z ERROR connection{remote_addr=192.168.2.200:59757 conn_id=442}: jsonrpsee_server::transport::ws: WS transport error: i/o error: An existing connection was forcibly closed by the remote host. (os error 10054); terminate connection: 442
2024-04-15T01:43:42.894236Z ERROR connection{remote_addr=192.168.2.200:59759 conn_id=445}: jsonrpsee_server::transport::ws: WS transport error: i/o error: An existing connection was forcibly closed by the remote host. (os error 10054); terminate connection: 445
2024-04-15T01:43:42.894279Z ERROR connection{remote_addr=192.168.2.200:59761 conn_id=447}: jsonrpsee_server::transport::ws: WS transport error: i/o error: An existing connection was forcibly closed by the remote host. (os error 10054); terminate connection: 447
2024-04-15T01:43:42.894197Z ERROR connection{remote_addr=192.168.2.200:59753 conn_id=438}: jsonrpsee_server::transport::ws: WS transport error: i/o error: An existing connection was forcibly closed by the remote host. (os error 10054); terminate connection: 438
2024-04-15T01:43:42.894329Z ERROR connection{remote_addr=192.168.2.200:59763 conn_id=448}: jsonrpsee_server::transport::ws: WS transport error: i/o error: An existing connection was forcibly closed by the remote host. (os error 10054); terminate connection: 448
2024-04-15T01:43:42.937790Z ERROR connection{remote_addr=192.168.2.200:59752 conn_id=436}: jsonrpsee_server::transport::ws: WS transport error: i/o error: An existing connection was forcibly closed by the remote host. (os error 10054); terminate connection: 436
2024-04-15T01:43:42.937863Z ERROR connection{remote_addr=192.168.2.200:59754 conn_id=439}: jsonrpsee_server::transport::ws: WS transport error: i/o error: An existing connection was forcibly closed by the remote host. (os error 10054); terminate connection: 439
2024-04-15T01:43:42.937862Z ERROR connection{remote_addr=192.168.2.200:59762 conn_id=446}: jsonrpsee_server::transport::ws: WS transport error: i/o error: An existing connection was forcibly closed by the remote host. (os error 10054); terminate connection: 446
2024-04-15T01:43:42.937863Z ERROR connection{remote_addr=192.168.2.200:59760 conn_id=444}: jsonrpsee_server::transport::ws: WS transport error: i/o error: An existing connection was forcibly closed by the remote host. (os error 10054); terminate connection: 444
2024-04-15T01:43:42.937868Z ERROR connection{remote_addr=192.168.2.200:59764 conn_id=449}: jsonrpsee_server::transport::ws: WS transport error: i/o error: An existing connection was forcibly closed by the remote host. (os error 10054); terminate connection: 449
2024-04-15T01:43:42.937874Z ERROR connection{remote_addr=192.168.2.200:59758 conn_id=443}: jsonrpsee_server::transport::ws: WS transport error: i/o error: An existing connection was forcibly closed by the remote host. (os error 10054); terminate connection: 443
2024-04-15T01:43:42.937880Z ERROR connection{remote_addr=192.168.2.200:59756 conn_id=440}: jsonrpsee_server::transport::ws: WS transport error: i/o error: An existing connection was forcibly closed by the remote host. (os error 10054); terminate connection: 440
2024-04-15T01:43:43.291281Z  INFO connection{remote_addr=192.168.2.200:60069 conn_id=450}: jsonrpsee_server::server: Accepting new connection 126/200
2024-04-15T01:43:43.691890Z  INFO connection{remote_addr=192.168.2.200:60071 conn_id=451}: jsonrpsee_server::server: Accepting new connection 127/200
2024-04-15T01:43:43.739506Z  INFO connection{remote_addr=192.168.2.200:60072 conn_id=452}: jsonrpsee_server::server: Accepting new connection 128/200
2024-04-15T01:43:43.741442Z  INFO connection{remote_addr=192.168.2.200:60073 conn_id=453}: jsonrpsee_server::server: Accepting new connection 129/200
2024-04-15T01:43:43.741571Z  INFO connection{remote_addr=192.168.2.200:60075 conn_id=454}: jsonrpsee_server::server: Accepting new connection 130/200
2024-04-15T01:43:43.741662Z  INFO connection{remote_addr=192.168.2.200:60077 conn_id=455}: jsonrpsee_server::server: Accepting new connection 131/200
2024-04-15T01:43:43.741750Z  INFO connection{remote_addr=192.168.2.200:60074 conn_id=456}: jsonrpsee_server::server: Accepting new connection 132/200
2024-04-15T01:43:43.741849Z  INFO connection{remote_addr=192.168.2.200:60076 conn_id=457}: jsonrpsee_server::server: Accepting new connection 133/200
2024-04-15T01:43:43.741925Z  INFO connection{remote_addr=192.168.2.200:60079 conn_id=458}: jsonrpsee_server::server: Accepting new connection 134/200
2024-04-15T01:43:43.742010Z  INFO connection{remote_addr=192.168.2.200:60081 conn_id=459}: jsonrpsee_server::server: Accepting new connection 135/200
2024-04-15T01:43:43.742108Z  INFO connection{remote_addr=192.168.2.200:60083 conn_id=460}: jsonrpsee_server::server: Accepting new connection 136/200
2024-04-15T01:43:43.879617Z  INFO connection{remote_addr=192.168.2.200:60078 conn_id=461}: jsonrpsee_server::server: Accepting new connection 137/200
2024-04-15T01:43:43.879772Z  INFO connection{remote_addr=192.168.2.200:60080 conn_id=462}: jsonrpsee_server::server: Accepting new connection 138/200
2024-04-15T01:43:43.879912Z  INFO connection{remote_addr=192.168.2.200:60082 conn_id=463}: jsonrpsee_server::server: Accepting new connection 139/200
2024-04-15T01:43:43.879984Z  INFO connection{remote_addr=192.168.2.200:60084 conn_id=464}: jsonrpsee_server::server: Accepting new connection 140/200
2024-04-15T01:43:47.666506Z  INFO Consensus: substrate: 💤 Idle (40 peers), best: #1084673 (0xa26b…9c1b), finalized #988889 (0x01de…b547), ⬇ 35.6kiB/s ⬆ 15.2kiB/s
2024-04-15T01:43:52.670659Z  INFO Consensus: substrate: 💤 Idle (40 peers), best: #1084673 (0xa26b…9c1b), finalized #988889 (0x01de…b547), ⬇ 28.9kiB/s ⬆ 15.4kiB/s
2024-04-15T01:43:57.224779Z  WARN Consensus: sync: 💔 Ignored block (#1084679 -- 0xc519…1a3f) announcement from 12D3KooWAX7SxrzUZvYuZcDG2XftetyCuofdBDeWLrZG78EvdRGW because all validation slots for this peer are occupied.
2024-04-15T01:43:57.682367Z  INFO Consensus: substrate: 💤 Idle (40 peers), best: #1084673 (0xa26b…9c1b), finalized #988889 (0x01de…b547), ⬇ 84.9kiB/s ⬆ 18.7kiB/s
2024-04-15T01:44:02.696536Z  INFO Consensus: substrate: ⚙️  Preparing  0.0 bps, target=#1084690 (40 peers), best: #1084673 (0xa26b…9c1b), finalized #988889 (0x01de…b547), ⬇ 30.5kiB/s ⬆ 7.2kiB/s
2024-04-15T01:44:07.701308Z  INFO Consensus: substrate: ⚙️  Preparing  0.0 bps, target=#1084691 (40 peers), best: #1084673 (0xa26b…9c1b), finalized #988889 (0x01de…b547), ⬇ 29.4kiB/s ⬆ 1.6kiB/s
2024-04-15T01:44:12.712875Z  INFO Consensus: substrate: ⚙️  Preparing  0.0 bps, target=#1084692 (40 peers), best: #1084673 (0xa26b…9c1b), finalized #988889 (0x01de…b547), ⬇ 28.9kiB/s ⬆ 1.1kiB/s
2024-04-15T01:44:17.714275Z  INFO Consensus: substrate: ⚙️  Preparing  0.0 bps, target=#1084694 (40 peers), best: #1084673 (0xa26b…9c1b), finalized #988889 (0x01de…b547), ⬇ 24.1kiB/s ⬆ 1.5kiB/s
2024-04-15T01:44:22.724846Z  INFO Consensus: substrate: ⚙️  Preparing  0.0 bps, target=#1084695 (40 peers), best: #1084673 (0xa26b…9c1b), finalized #988889 (0x01de…b547), ⬇ 28.1kiB/s ⬆ 1.4kiB/s
2024-04-15T01:44:27.729763Z  INFO Consensus: substrate: ⚙️  Preparing  0.0 bps, target=#1084695 (40 peers), best: #1084673 (0xa26b…9c1b), finalized #988889 (0x01de…b547), ⬇ 28.9kiB/s ⬆ 1.1kiB/s
2024-04-15T01:44:32.742241Z  INFO Consensus: substrate: ⚙️  Preparing  0.0 bps, target=#1084695 (40 peers), best: #1084673 (0xa26b…9c1b), finalized #988889 (0x01de…b547), ⬇ 27.5kiB/s ⬆ 2.1kiB/s
2024-04-15T01:44:37.747975Z  INFO Consensus: substrate: ⚙️  Preparing  0.0 bps, target=#1084696 (40 peers), best: #1084673 (0xa26b…9c1b), finalized #988889 (0x01de…b547), ⬇ 16.7kiB/s ⬆ 1.2kiB/s
2024-04-15T01:44:42.750803Z  INFO Consensus: substrate: 💤 Idle (40 peers), best: #1084673 (0xa26b…9c1b), finalized #988889 (0x01de…b547), ⬇ 44.0kiB/s ⬆ 1.3kiB/s
2024-04-15T01:44:47.758696Z  INFO Consensus: substrate: ⚙️  Preparing  0.0 bps, target=#1084697 (40 peers), best: #1084673 (0xa26b…9c1b), finalized #988889 (0x01de…b547), ⬇ 55.1kiB/s ⬆ 2.3kiB/s
2024-04-15T01:44:52.759872Z  INFO Consensus: substrate: ⚙️  Preparing  0.0 bps, target=#1084697 (40 peers), best: #1084673 (0xa26b…9c1b), finalized #988889 (0x01de…b547), ⬇ 20.0kiB/s ⬆ 1.7kiB/s
2024-04-15T01:44:57.761148Z  INFO Consensus: substrate: ⚙️  Preparing  0.0 bps, target=#1084698 (40 peers), best: #1084673 (0xa26b…9c1b), finalized #988889 (0x01de…b547), ⬇ 23.9kiB/s ⬆ 2.6kiB/s
2024-04-15T01:45:02.763736Z  INFO Consensus: substrate: 💤 Idle (40 peers), best: #1084673 (0xa26b…9c1b), finalized #988889 (0x01de…b547), ⬇ 67.6kiB/s ⬆ 2.6kiB/s
2024-04-15T01:45:07.765862Z  INFO Consensus: substrate: 💤 Idle (40 peers), best: #1084673 (0xa26b…9c1b), finalized #988889 (0x01de…b547), ⬇ 26.8kiB/s ⬆ 14.9kiB/s
2024-04-15T01:45:12.771653Z  INFO Consensus: substrate: ⚙️  Preparing  0.0 bps, target=#1084700 (40 peers), best: #1084673 (0xa26b…9c1b), finalized #988889 (0x01de…b547), ⬇ 46.3kiB/s ⬆ 3.3kiB/s
2024-04-15T01:45:17.779759Z  INFO Consensus: substrate: ⚙️  Preparing  0.0 bps, target=#1084700 (40 peers), best: #1084673 (0xa26b…9c1b), finalized #988889 (0x01de…b547), ⬇ 21.8kiB/s ⬆ 1.2kiB/s
2024-04-15T01:45:22.784662Z  INFO Consensus: substrate: ⚙️  Preparing  0.0 bps, target=#1084701 (40 peers), best: #1084673 (0xa26b…9c1b), finalized #988889 (0x01de…b547), ⬇ 28.4kiB/s ⬆ 1.2kiB/s
2024-04-15T01:45:27.791767Z  INFO Consensus: substrate: ⚙️  Preparing  0.0 bps, target=#1084701 (40 peers), best: #1084673 (0xa26b…9c1b), finalized #988889 (0x01de…b547), ⬇ 26.3kiB/s ⬆ 1.8kiB/s
2024-04-15T01:45:32.797844Z  INFO Consensus: substrate: ⚙️  Preparing  0.0 bps, target=#1084701 (40 peers), best: #1084673 (0xa26b…9c1b), finalized #988889 (0x01de…b547), ⬇ 41.8kiB/s ⬆ 3.0kiB/s
2024-04-15T01:45:34.786416Z ERROR connection{remote_addr=192.168.2.200:60069 conn_id=450}: jsonrpsee_server::transport::ws: WS transport error: i/o error: An existing connection was forcibly closed by the remote host. (os error 10054); terminate connection: 450
2024-04-15T01:45:34.786680Z ERROR connection{remote_addr=192.168.2.200:60076 conn_id=457}: jsonrpsee_server::transport::ws: WS transport error: i/o error: An existing connection was forcibly closed by the remote host. (os error 10054); terminate connection: 457
2024-04-15T01:45:34.786700Z ERROR connection{remote_addr=192.168.2.200:60073 conn_id=453}: jsonrpsee_server::transport::ws: WS transport error: i/o error: An existing connection was forcibly closed by the remote host. (os error 10054); terminate connection: 453
2024-04-15T01:45:34.786701Z ERROR connection{remote_addr=192.168.2.200:60072 conn_id=452}: jsonrpsee_server::transport::ws: WS transport error: i/o error: An existing connection was forcibly closed by the remote host. (os error 10054); terminate connection: 452
2024-04-15T01:45:34.786730Z ERROR connection{remote_addr=192.168.2.200:60075 conn_id=454}: jsonrpsee_server::transport::ws: WS transport error: i/o error: An existing connection was forcibly closed by the remote host. (os error 10054); terminate connection: 454
2024-04-15T01:45:34.786740Z ERROR connection{remote_addr=192.168.2.200:60074 conn_id=456}: jsonrpsee_server::transport::ws: WS transport error: i/o error: An existing connection was forcibly closed by the remote host. (os error 10054); terminate connection: 456
2024-04-15T01:45:34.786744Z ERROR connection{remote_addr=192.168.2.200:60071 conn_id=451}: jsonrpsee_server::transport::ws: WS transport error: i/o error: An existing connection was forcibly closed by the remote host. (os error 10054); terminate connection: 451
2024-04-15T01:45:34.786770Z ERROR connection{remote_addr=192.168.2.200:60081 conn_id=459}: jsonrpsee_server::transport::ws: WS transport error: i/o error: An existing connection was forcibly closed by the remote host. (os error 10054); terminate connection: 459
2024-04-15T01:45:34.786792Z ERROR connection{remote_addr=192.168.2.200:60078 conn_id=461}: jsonrpsee_server::transport::ws: WS transport error: i/o error: An existing connection was forcibly closed by the remote host. (os error 10054); terminate connection: 461
2024-04-15T01:45:34.786811Z ERROR connection{remote_addr=192.168.2.200:60079 conn_id=458}: jsonrpsee_server::transport::ws: WS transport error: i/o error: An existing connection was forcibly closed by the remote host. (os error 10054); terminate connection: 458
2024-04-15T01:45:34.786826Z ERROR connection{remote_addr=192.168.2.200:60077 conn_id=455}: jsonrpsee_server::transport::ws: WS transport error: i/o error: An existing connection was forcibly closed by the remote host. (os error 10054); terminate connection: 455
2024-04-15T01:45:34.786827Z ERROR connection{remote_addr=192.168.2.200:60080 conn_id=462}: jsonrpsee_server::transport::ws: WS transport error: i/o error: An existing connection was forcibly closed by the remote host. (os error 10054); terminate connection: 462
2024-04-15T01:45:34.786848Z ERROR connection{remote_addr=192.168.2.200:60082 conn_id=463}: jsonrpsee_server::transport::ws: WS transport error: i/o error: An existing connection was forcibly closed by the remote host. (os error 10054); terminate connection: 463
2024-04-15T01:45:34.786847Z ERROR connection{remote_addr=192.168.2.200:60083 conn_id=460}: jsonrpsee_server::transport::ws: WS transport error: i/o error: An existing connection was forcibly closed by the remote host. (os error 10054); terminate connection: 460
2024-04-15T01:45:34.786863Z ERROR connection{remote_addr=192.168.2.200:60084 conn_id=464}: jsonrpsee_server::transport::ws: WS transport error: i/o error: An existing connection was forcibly closed by the remote host. (os error 10054); terminate connection: 464
2024-04-15T01:45:35.216333Z  INFO connection{remote_addr=192.168.2.200:60235 conn_id=465}: jsonrpsee_server::server: Accepting new connection 126/200
2024-04-15T01:45:35.600747Z  INFO connection{remote_addr=192.168.2.200:60238 conn_id=466}: jsonrpsee_server::server: Accepting new connection 127/200
2024-04-15T01:45:35.600905Z  INFO connection{remote_addr=192.168.2.200:60237 conn_id=467}: jsonrpsee_server::server: Accepting new connection 128/200
2024-04-15T01:45:35.600976Z  INFO connection{remote_addr=192.168.2.200:60240 conn_id=468}: jsonrpsee_server::server: Accepting new connection 129/200
2024-04-15T01:45:35.601034Z  INFO connection{remote_addr=192.168.2.200:60239 conn_id=469}: jsonrpsee_server::server: Accepting new connection 130/200
2024-04-15T01:45:35.601105Z  INFO connection{remote_addr=192.168.2.200:60241 conn_id=470}: jsonrpsee_server::server: Accepting new connection 131/200
2024-04-15T01:45:35.601174Z  INFO connection{remote_addr=192.168.2.200:60242 conn_id=471}: jsonrpsee_server::server: Accepting new connection 132/200
2024-04-15T01:45:35.601246Z  INFO connection{remote_addr=192.168.2.200:60244 conn_id=472}: jsonrpsee_server::server: Accepting new connection 133/200
2024-04-15T01:45:35.601303Z  INFO connection{remote_addr=192.168.2.200:60243 conn_id=473}: jsonrpsee_server::server: Accepting new connection 134/200
2024-04-15T01:45:35.601386Z  INFO connection{remote_addr=192.168.2.200:60245 conn_id=474}: jsonrpsee_server::server: Accepting new connection 135/200
2024-04-15T01:45:35.601444Z  INFO connection{remote_addr=192.168.2.200:60246 conn_id=475}: jsonrpsee_server::server: Accepting new connection 136/200
2024-04-15T01:45:35.601501Z  INFO connection{remote_addr=192.168.2.200:60247 conn_id=476}: jsonrpsee_server::server: Accepting new connection 137/200
2024-04-15T01:45:35.601563Z  INFO connection{remote_addr=192.168.2.200:60248 conn_id=477}: jsonrpsee_server::server: Accepting new connection 138/200
2024-04-15T01:45:35.601623Z  INFO connection{remote_addr=192.168.2.200:60249 conn_id=478}: jsonrpsee_server::server: Accepting new connection 139/200
2024-04-15T01:45:35.601680Z  INFO connection{remote_addr=192.168.2.200:60250 conn_id=479}: jsonrpsee_server::server: Accepting new connection 140/200
2024-04-15T01:45:37.813184Z  INFO Consensus: substrate: ⚙️  Preparing  0.0 bps, target=#1084688 (40 peers), best: #1084673 (0xa26b…9c1b), finalized #988889 (0x01de…b547), ⬇ 39.6kiB/s ⬆ 3.6kiB/s
2024-04-15T01:45:42.815272Z  INFO Consensus: substrate: 💤 Idle (40 peers), best: #1084673 (0xa26b…9c1b), finalized #988889 (0x01de…b547), ⬇ 48.1kiB/s ⬆ 55.9kiB/s
2024-04-15T01:45:47.829837Z  INFO Consensus: substrate: 💤 Idle (40 peers), best: #1084673 (0xa26b…9c1b), finalized #988889 (0x01de…b547), ⬇ 50.8kiB/s ⬆ 28.9kiB/s
2024-04-15T01:45:52.842184Z  INFO Consensus: substrate: ⚙️  Preparing  0.0 bps, target=#1084703 (40 peers), best: #1084673 (0xa26b…9c1b), finalized #988889 (0x01de…b547), ⬇ 73.7kiB/s ⬆ 3.8kiB/s
2024-04-15T01:45:57.853556Z  INFO Consensus: substrate: ⚙️  Preparing  0.0 bps, target=#1084704 (40 peers), best: #1084673 (0xa26b…9c1b), finalized #988889 (0x01de…b547), ⬇ 47.6kiB/s ⬆ 2.6kiB/s
2024-04-15T01:46:02.855112Z  INFO Consensus: substrate: ⚙️  Preparing  0.0 bps, target=#1084706 (40 peers), best: #1084673 (0xa26b…9c1b), finalized #988889 (0x01de…b547), ⬇ 37.6kiB/s ⬆ 2.8kiB/s
2024-04-15T01:46:07.858419Z  INFO Consensus: substrate: ⚙️  Preparing  0.0 bps, target=#1084706 (40 peers), best: #1084673 (0xa26b…9c1b), finalized #988889 (0x01de…b547), ⬇ 32.4kiB/s ⬆ 2.2kiB/s
2024-04-15T01:46:12.872785Z  INFO Consensus: substrate: ⚙️  Preparing  0.0 bps, target=#1084706 (40 peers), best: #1084673 (0xa26b…9c1b), finalized #988889 (0x01de…b547), ⬇ 35.7kiB/s ⬆ 2.1kiB/s
2

This happens to both of my nodes every time that a replotting starts. 90% of the time the node will restart within 30 minutes. For that 10% I usually kill the node after about 45-60 minutes (unless it happened while I was sleeping) and restart it. Within 5-10 minutes of the node restart it is back in sync again.

I always have 40 peers on both nodes. It sucks that it goes out of sync, but why does it take so long to get back into sync when I have all of these peers?

3

Same with me. My node had 40 peers but it just went out of sync. I believe this is very critical bug that the team should look into ASAP.

4

I think we have the same problem

5

Yes probably. When I first saw your post, I thought it’s because you’re located in China. So I didn’t follow up.

My case, the signal is when a farmer with 100% plotted. So when replot is triggered, it triggered replot for all farms. I was able to capture the node’s log during its out of sync. The same remote farmer (local IP) was connected and disconnected again and again. During this connect/disconnect, the node was frozen: not able to sync up more block with all of its peers.

At least this is what I’ve observed. Nazar knows better since he codes and designs the protocol. This really needs a fix before mainnet.

6

It has nothing to do with the region; do not be misled by others. In China, many blockchain nodes can operate normally.

7

This seems to be related to large number of farmers connected to the same node. It wasn’t really designed for this and there are likely bottlenecks there that blow up when new segment is archived. I’m not planning to address this before we do Substrate upgrade (likely next release) and then farming cluster that should remove the need to have many connections to the node in the first place in addition to other improvements .

8

Noted. So we won’t fix it since we plan to build something new for big farm that has the design to resolve it. This makes sense.

But why with current node-farmer connection, this only happens when the replot is triggered? I’ve been farming perfectly for several days. And boom, I have that farmer with all plots finished, and in a moment, when replot was triggered for all the farms of that farmer, it caused the node out of sync.

9

Replotting is is not the cause, it is the consequence of new segment of archival history being created. In such case depending on case size farmers will start bombarding node with request for newly archived pieces and node is having a really hard time responding to large number of requests with (relatively speaking) large responses. As the result some requests can time out, connections drop, etc. And node is waiting for acknowledgement from farmers that they received and processed segments and that node can move on from there.

10

Thanks. However please note that from the node’s log I’ve shared. The disconnect/connect happened for only 1 farmer and this is the one that has 100% plot completed.

From your explanation above "farmers will start bombarding node with request for newly archived pieces", why the connect/disconnect didn’t happen to the all other farmers (I have 9 farmers connecting to this node). This is such a coincidence with Wezl’s case and he complained about that in Discord before (when he did, I didn’t believe until I have the 3 farmers finished plotting. Then, 1 farm got stuck in the disconnect/connect loop that brought down the node).

If this happens again, I’ll update.

11

Plotting status is irrelevant for cache update.

I’m afraid this is simply non-deterministic process, not sure there is a better explanation.

12

hmm, so is this the correspond code, i have a question for this code, but if no farm subscribe new segment , will this function block for acknowledged?

async fn send_archived_segment_notification(
    archived_segment_notification_sender: &SubspaceNotificationSender<ArchivedSegmentNotification>,
    archived_segment: NewArchivedSegment,
) {
    let segment_index = archived_segment.segment_header.segment_index();
    let (acknowledgement_sender, mut acknowledgement_receiver) =
        tracing_unbounded::<()>("subspace_acknowledgement", 100);
    // Keep `archived_segment` around until all acknowledgements are received since some receivers
    // might use weak references
    let archived_segment = Arc::new(archived_segment);
    let archived_segment_notification = ArchivedSegmentNotification {
        archived_segment: Arc::clone(&archived_segment),
        acknowledgement_sender,
    };

    archived_segment_notification_sender.notify(move || archived_segment_notification);

    while acknowledgement_receiver.next().await.is_some() {
        debug!(
            "Archived segment notification acknowledged: {}",
            segment_index
        );
    }
}
13

Of course not. It waits for acknowledgements on all subscription. No subscription - no acknowledgement needed.

14

i only find the rpc api subscribe this notify, and what happer if farmer subscribe but not response,?

15

You will see exactly what is described at the beginning of this thread. You’ll have to read the code for more details.

16

I think it makes the most sense to separate nodes and farms. This is because the farm always pushes CPU usage to 100%, which can significantly impact the performance of the nodes.