Gemini-3g nodes not syncing on Linux

Support
1

Nodes on gemini-3g don’t sync, not matter if run in plain vanilla mode or run on specfific ports. Nodes start without error, display the correct ports and parameters, connect to 40 peers and display the Received notification to sync from DSN reason=WentOnlineSubstrate message.

However, node stays at 0.0 bps and never starts to sync (hours).
Occasional Ignored block (#24908 -- 0xa09c…c070) announcement from 12D3KooWJaUVqS3AHUa6X9ki659DVgXXM9BrVFXPY5GZA1qPVVP9 because all validation slots for this peer are occupied. occurs

  • No other subspace version running on network.
  • Tested on Ubunbtu server 22.04 and 20.04
  • All ports (TCP/UDP) forwarded to the appropriate server
  • Nodes appear on telemetry - as grayed out non-synced nodes.
  • Nodes seem to switch address occassionaly between public and private, i.e. Public address status changed. old=Private new=Public("/ip4/IP_4/udp/30433/quic-v1/p2p/node_ID") and Public address status changed. old=Public("/ip4/IP_4/udp/30433/quic-v1/p2p/node_ID") new=Private

Gemini-3f nodes in the same environment start to sync within 5 minutes

Startup script (neither syncing):

./subspace-node-ubuntu-x86_64-v2-gemini-3g-2023-oct-31 \
  --chain gemini-3g \
  --base-path /home/USER/.subspace \
  --blocks-pruning 256 \
  --state-pruning archive-canonical  \
  --rpc-methods unsafe \
  --unsafe-rpc-external \
  --rpc-external \
  --rpc-cors all \
  --no-private-ipv4 \
  --validator \
  --name NAME

Running on specific ports:

./subspace-node-ubuntu-x86_64-skylake-gemini-3g-2023-oct-31 \
  --chain gemini-3g \
  --base-path /home/USER/.subspace \
  --blocks-pruning 256 \
  --state-pruning archive-canonical  \
  --rpc-methods unsafe \
  --unsafe-rpc-external \
  --rpc-external \
  --rpc-cors all \
  --no-private-ipv4 \
  --dsn-listen-on /ip4/0.0.0.0/tcp/31434 \
  --dsn-listen-on /ip4/0.0.0.0/udp/31434/quic-v1 \
  --port 31334 \
  --validator \
  --name NAME

A plain vanilla node without local network access synced once. I could not get it to sync again with local network access.

Any hints welcome.

2

I get quite a bit of up/down data on the previously synced node, but it never starts to sync again.

image
image1352×265 10.5 KB

3

Please attach node logs all the way since it was started. Always post logs as text instead of screenshots, screenshots are not searchable.

4

Most recent log. I restarted server and router.
I can’t find a way to attach a file.

5

pastebin is the right way to do it :+1:

Can you start the node with RUST_LOG=info,subspace_service=trace and post logs after 30 minutes if it didn’t start syncing?

6

Restarted already. Will post in 30+min.

7

Damn it! That is syncing with the debug logging.

2023-11-02T02:42:49.427572Z TRACE tokio-runtime-worker subspace_service::sync_from_dsn::import_blocks: [Consensus] Number of importing blocks reached queue limit, waiting before retrying block_num
ber=3297 best_block_number=2797
2023-11-02T02:42:50.428491Z TRACE tokio-runtime-worker subspace_service::sync_from_dsn::import_blocks: [Consensus] Number of importing blocks reached queue limit, waiting before retrying block_num
ber=3316 best_block_number=2816
2023-11-02T02:42:51.429527Z TRACE tokio-runtime-worker subspace_service::sync_from_dsn::import_blocks: [Consensus] Number of importing blocks reached queue limit, waiting before retrying block_num
ber=3364 best_block_number=2864
2023-11-02T02:42:52.430518Z TRACE tokio-runtime-worker subspace_service::sync_from_dsn::import_blocks: [Consensus] Number of importing blocks reached queue limit, waiting before retrying block_num
ber=3381 best_block_number=2881
2023-11-02T02:42:53.432361Z TRACE tokio-runtime-worker subspace_service::sync_from_dsn::import_blocks: [Consensus] Number of importing blocks reached queue limit, waiting before retrying block_num
ber=3435 best_block_number=2935
2023-11-02T02:42:53.699429Z INFO tokio-runtime-worker substrate: [Consensus] ⚙ Syncing 36.8 bps, target=#29550 (40 peers), best: #2946 (0x472a…3df9), finalized #0 (0x4180…180b), ⬇ 50.8kiB/s ⬆ 1.
7kiB/s
2023-11-02T02:42:54.433851Z TRACE tokio-runtime-worker subspace_service::sync_from_dsn::import_blocks: [Consensus] Number of importing blocks reached queue limit, waiting before retrying block_num
ber=3459 best_block_number=2959
2023-11-02T02:42:55.434652Z TRACE tokio-runtime-worker subspace_service::sync_from_dsn::import_blocks: [Consensus] Number of importing blocks reached queue limit, waiting before retrying block_num
ber=3491 best_block_number=2991
2023-11-02T02:42:56.435860Z TRACE tokio-runtime-worker subspace_service::sync_from_dsn::import_blocks: [Consensus] Number of importing blocks reached queue limit, waiting before retrying block_num
ber

First time today. :roll_eyes:

8

me too!!! it works now !

9

me too , it is working now , syncing ,amazing

10

Try to restart without those logs, it should still work just fine.

11

Restarted without debug log and it keeps syncing.
30 hours of additional gray hair!

Attempt 15 worked:

node_20231101_1138_1974553.log
node_20231101_1212_1975200.log
node_20231101_1227_1975644.log
node_20231101_1329_1976552.log
node_20231101_1704_2135349.log
node_20231101_1723_2137043.log
node_20231101_1833_2138809.log
node_20231101_2011_2879.log
node_20231101_2035_3256.log
node_20231101_2156_3864.log
node_20231101_2214_4208.log
node_20231102_0112_114560.log
node_20231102_0216_115992.log
node_20231102_0217_116009.log
node_20231102_0237_116950.log
node_20231102_0245_117368.log

Thanks a LOT for the help!

Any idea what fixed it?
(My node with the specified ports also started to sync after restart now.)

12

Glad I’m not the only one. Discord made it really look like I’m the odd one out.

13

Not sure yet, would be helpful to get logs from someone who is still experiencing it.
I’m sure we’ll figure it out, just hoping sooner rather than later.

14

When I opened the farmer, the node stopped synchronizing again…

15

I plan to wait until the synchronization is complete before attempting to open the farmer

16

Full debug log when it started to sync. Not sure it helps, since it started to sync very fast.

17

I also have a 5min debug log from yesterday when it didn’t sync.
Log level debug - not the specialized instructions from today.
It is 41MB, too big for pastebin. Also not as clinically run, i.e. some ports may not be forwarded etc.
Let me know if you’d like to get the whole file or if I should look for something specific.

Those are the last few seconds before I killed the process: 2023-11-01T03:55:09.901683Z DEBUG tokio-runtime-worker netlink_proto::connection - Pastebin.com

18

Farmer started without issues for me.

19

The whole file might be helpful indeed :pray:

20

https://drive.google.com/drive/folders/14_ld_FRpGbLaWj9fItDRja5LJlTE4zup?usp=sharing

Please let me know when you got it. Thanks!

1 Like