Hello, out of all the nodes i have one that isn’t able to sync. I have no clue why that is. It’s stuck at the moment at 13k. It originally was synced, then it stopped being able to sync. Eventually I was thinking the database might be corrupted so I deleted the node’s database. Now it’s still stuck at 13k height. I tried with fewer in/out peers and at the moment with a lot. This is 2x 32 core Epyc/Milan and Fedora 37. Not sure what else to tell you. I think there is something specific to this node.
1 Like
Blockquote
– firewall ports open
firewall-cmd --list-ports
3389/tcp 30200-30206/tcp 30900-30906/tcp
Blockquote
– listening on ports
lsof -i -P -n | grep LISTEN |grep -i sub
subspace- 248964 srv_subspace 292u IPv4 17774598 0t0 TCP *:30904 (LISTEN)
subspace- 248964 srv_subspace 297u IPv6 17774601 0t0 TCP *:30900 (LISTEN)
subspace- 248964 srv_subspace 299u IPv4 17774603 0t0 TCP *:30900 (LISTEN)
subspace- 248964 srv_subspace 466u IPv4 18735601 0t0 TCP 127.0.0.1:30903 (LISTEN)
subspace- 248964 srv_subspace 467u IPv4 18936226 0t0 TCP 127.0.0.1:30901 (LISTEN)
That’s how it’s looking for days… Restart of node or server doesn’t help. I have 2 totally identical machines, same HW and SW, no issues there. Where and what do I need to dig out find the reason for this issue
There was an issue with node sync from DSN in older releases, sep-13-2 fixes that. Also you have way too many peers, no need to customize node options, it may cause more harm than good for you. Defaults should work just fine. Might take some time before it starts importing blocks, but it shouldn’t take days or anything like that.
Thanks, 13-2 has made no difference so far and as I said, since it doesn’t sync i started to change defaults for example the peers, i tried with lesser and with more. None of which had a positive effect.
sep-13-2 takes time to start, give it maybe an hour, depending on what is the last block you have in archival history, but it should get un-stuck. If you restart the node you just reset the process to zero. You can observe more detailed logs with RUST_LOG=info,subspace_service=trace, it will print some useful details about sync process so you can see it is doing something.
Here is another node that went unsync… In the log it looks good until
Blockquote
2023-09-16 08:10:55 [Consensus]Imported #453531 (0xca39…b27b)
2023-09-16 08:10:55 [Consensus]
2023-09-16 08:10:57 [Consensus]
2023-09-16 08:10:58 [Consensus]Idle (17 peers), best: #453532 (0xf412…1fea), finalized #232206 (0x9143…9e6f),
61.8kiB/s
27.0kiB/s
2023-09-16 08:10:58 [Consensus]Claimed vote at slot 1694866258
2023-09-16 08:10:59 [Consensus] Received notification to sync from DSN reason=WentOnlineSubspace
2023-09-16 08:11:03 [Consensus]
2023-09-16 08:11:08 [Consensus]
2023-09-16 08:11:13 [Consensus]
2023-09-16 08:11:18 [Consensus]
2023-09-16 08:11:23 [Consensus]
2023-09-16 08:11:28 [Consensus]
2023-09-16 08:11:33 [Consensus]Syncing 0.0 bps, target=#453539 (25 peers), best: #453532 (0xf412…1fea), finalized #232206 (0x9143…9e6f),
2023-09-16 08:11:38 [Consensus]
2023-09-16 08:11:43 [Consensus]
…it continues this way for 4-5 hours until there is this WS transport error: i/o and then it start’s to sync again. this is currently sep-11
Blockquote
2023-09-16 12:24:54 [Consensus]
2023-09-16 12:24:59 [Consensus]
2023-09-16 12:25:04 [Consensus]
2023-09-16 12:25:09 [Consensus]
2023-09-16 12:25:12 WS transport error: i/o error: Connection reset by peer (os error 104); terminate connection: 64
2023-09-16 12:25:12 WS transport error: i/o error: Connection reset by peer (os error 104); terminate connection: 58
2023-09-16 12:25:12 WS transport error: i/o error: Connection reset by peer (os error 104); terminate connection: 52
2023-09-16 12:25:12 WS transport error: i/o error: Connection reset by peer (os error 104); terminate connection: 55
2023-09-16 12:25:12 WS transport error: i/o error: Connection reset by peer (os error 104); terminate connection: 54
2023-09-16 12:25:12 WS transport error: i/o error: Connection reset by peer (os error 104); terminate connection: 53
2023-09-16 12:25:12 WS transport error: i/o error: Connection reset by peer (os error 104); terminate connection: 56
2023-09-16 12:25:12 WS transport error: i/o error: Connection reset by peer (os error 104); terminate connection: 59
2023-09-16 12:25:12 WS transport error: i/o error: Connection reset by peer (os error 104); terminate connection: 57
2023-09-16 12:25:12 WS transport error: i/o error: Connection reset by peer (os error 104); terminate connection: 60
2023-09-16 12:25:12 WS transport error: i/o error: Connection reset by peer (os error 104); terminate connection: 34
2023-09-16 12:25:12 WS transport error: i/o error: Connection reset by peer (os error 104); terminate connection: 63
2023-09-16 12:25:12 WS transport error: i/o error: Connection reset by peer (os error 104); terminate connection: 43
2023-09-16 12:25:12 WS transport error: i/o error: Connection reset by peer (os error 104); terminate connection: 37
2023-09-16 12:25:12 WS transport error: i/o error: Connection reset by peer (os error 104); terminate connection: 36
2023-09-16 12:25:12 WS transport error: i/o error: Connection reset by peer (os error 104); terminate connection: 62
2023-09-16 12:25:12 WS transport error: i/o error: Connection reset by peer (os error 104); terminate connection: 46
2023-09-16 12:25:12 WS transport error: i/o error: Connection reset by peer (os error 104); terminate connection: 47
2023-09-16 12:25:12 WS transport error: i/o error: Connection reset by peer (os error 104); terminate connection: 42
2023-09-16 12:25:12 WS transport error: i/o error: Connection reset by peer (os error 104); terminate connection: 51
2023-09-16 12:25:12 WS transport error: i/o error: Connection reset by peer (os error 104); terminate connection: 35
2023-09-16 12:25:12 WS transport error: i/o error: Connection reset by peer (os error 104); terminate connection: 45
2023-09-16 12:25:12 WS transport error: i/o error: Connection reset by peer (os error 104); terminate connection: 61
2023-09-16 12:25:12 WS transport error: i/o error: Connection reset by peer (os error 104); terminate connection: 41
2023-09-16 12:25:12 WS transport error: i/o error: Connection reset by peer (os error 104); terminate connection: 50
2023-09-16 12:25:12 WS transport error: i/o error: Connection reset by peer (os error 104); terminate connection: 48
2023-09-16 12:25:12 WS transport error: i/o error: Connection reset by peer (os error 104); terminate connection: 40
2023-09-16 12:25:12 WS transport error: i/o error: Connection reset by peer (os error 104); terminate connection: 44
2023-09-16 12:25:12 WS transport error: i/o error: Connection reset by peer (os error 104); terminate connection: 49
2023-09-16 12:25:12 WS transport error: i/o error: Connection reset by peer (os error 104); terminate connection: 39
2023-09-16 12:25:12 WS transport error: i/o error: Connection reset by peer (os error 104); terminate connection: 38
2023-09-16 12:25:12 WS transport error: i/o error: Connection reset by peer (os error 104); terminate connection: 33
2023-09-16 12:25:13 WS transport error: i/o error: Connection reset by peer (os error 104); terminate connection: 0
2023-09-16 12:25:14 [Consensus]
2023-09-16 12:25:19 [Consensus]
2023-09-16 12:25:24 [Consensus]
2023-09-16 12:25:29 [Consensus]
2023-09-16 12:25:34 [Consensus]
2023-09-16 12:25:39 [Consensus]
2023-09-16 12:25:44 [Consensus]
2023-09-16 12:25:49 [Consensus]
2023-09-16 12:25:54 [Consensus]
2023-09-16 12:25:59 [Consensus]
2023-09-16 12:26:03 [Consensus]Error importing block 0xc0db2b2e28fab26a0ea9d16ffd9b08c0292cf5b9567855da96f42d3f9b231623: block has an unknown parent
2023-09-16 12:26:03 [Consensus]
2023-09-16 12:26:04 [Consensus]
2023-09-16 12:26:09 [Consensus]
2023-09-16 12:26:10 [Consensus]
2023-09-16 12:26:14 [Consensus]
2023-09-16 12:26:19 [Consensus]
2023-09-16 12:26:24 [Consensus]
2023-09-16 12:26:29 [Consensus]
2023-09-16 12:26:34 [Consensus]
2023-09-16 12:26:39 [Consensus]
2023-09-16 12:26:44 [Consensus]
As long as it is making progress there is nothing inherently wrong.
If it gets stuck, it is likely due to a bug in Substrate. But also your node is not using DSN sync because you’re not on latest release.
If you’re not on latest release even if you sync successfully you’ll likely get banned (and ban yourself) a bunch of peers in the process.
Please upgrade to new releases as they come in, especially if you see issues.
Can you expand on what would cause the ban on users running the old versions? Is this a Farmer ban as weve discussed in the past or a different type of ban?
Bans can be two-way here.
One can be caused by local node requesting blocks from other nodes, but other nodes don’t have them (pruned already), thus reputation will be decreased for those peers until it reaches the floor, at which point peer will be banned.
The other happens in the opposite direction when local node requests the same blocks over and over again without getting desired response. Asking for the same blocks many times will result in reputation decrease down to being banned by other peers.
So one way or another node will either sync or get banned and probability of sync decreases over time as more nodes are pruned.
2 Likes