# Gemini-3g nodes not syncing on Linux **URL:** \ **Category:** Support\ **Created:** [November 1, 2023, 8:46pm UTC](https://forum.autonomys.xyz/t/gemini-3g-nodes-not-syncing-on-linux/2090 "2023-11-01T20:46:29Z")\ **Posts on this page:** 20\ **Page:** 1
**Author:** ![erdnapa](https://avatars.discourse-cdn.com/v4/letter/e/ebca7d/32.png) [@erdnapa](https://forum.autonomys.xyz/u/erdnapa)\ **Post date:** [November 1, 2023, 8:46pm UTC](https://forum.autonomys.xyz/t/gemini-3g-nodes-not-syncing-on-linux/2090/1 "2023-11-01T20:46:29Z")
Nodes on gemini-3g don’t sync, not matter if run in plain vanilla mode or run on specfific ports. Nodes start without error, display the correct ports and parameters, connect to 40 peers and display the `Received notification to sync from DSN reason=WentOnlineSubstrate` message. However, node stays at 0.0 bps and never starts to sync (hours). Occasional `Ignored block (#24908 -- 0xa09c…c070) announcement from 12D3KooWJaUVqS3AHUa6X9ki659DVgXXM9BrVFXPY5GZA1qPVVP9 because all validation slots for this peer are occupied.` occurs - No other subspace version running on network. - Tested on Ubunbtu server 22.04 and 20.04 - All ports (TCP/UDP) forwarded to the appropriate server - Nodes appear on telemetry - as grayed out non-synced nodes. - Nodes seem to switch address occassionaly between public and private, i.e. `Public address status changed. old=Private new=Public("/ip4/IP_4/udp/30433/quic-v1/p2p/node_ID")` and `Public address status changed. old=Public("/ip4/IP_4/udp/30433/quic-v1/p2p/node_ID") new=Private` Gemini-3f nodes in the same environment start to sync within 5 minutes Startup script (neither syncing): ```auto ./subspace-node-ubuntu-x86_64-v2-gemini-3g-2023-oct-31 \ --chain gemini-3g \ --base-path /home/USER/.subspace \ --blocks-pruning 256 \ --state-pruning archive-canonical \ --rpc-methods unsafe \ --unsafe-rpc-external \ --rpc-external \ --rpc-cors all \ --no-private-ipv4 \ --validator \ --name NAME ``` Running on specific ports: ```auto ./subspace-node-ubuntu-x86_64-skylake-gemini-3g-2023-oct-31 \ --chain gemini-3g \ --base-path /home/USER/.subspace \ --blocks-pruning 256 \ --state-pruning archive-canonical \ --rpc-methods unsafe \ --unsafe-rpc-external \ --rpc-external \ --rpc-cors all \ --no-private-ipv4 \ --dsn-listen-on /ip4/0.0.0.0/tcp/31434 \ --dsn-listen-on /ip4/0.0.0.0/udp/31434/quic-v1 \ --port 31334 \ --validator \ --name NAME ``` A plain vanilla node without local network access synced once. I could not get it to sync again with local network access. Any hints welcome. ---
**Author:** ![erdnapa](https://avatars.discourse-cdn.com/v4/letter/e/ebca7d/32.png) [@erdnapa](https://forum.autonomys.xyz/u/erdnapa)\ **Post date:** [November 1, 2023, 8:55pm UTC](https://forum.autonomys.xyz/t/gemini-3g-nodes-not-syncing-on-linux/2090/2 "2023-11-01T20:55:08Z")
I get quite a bit of up/down data on the previously synced node, but it never starts to sync again. ![image](https://canada1.discourse-cdn.com/flex011/uploads/subspace/original/2X/a/a93fefba413f1f528976fca4cd3f3eb2ba44eb81.png) ---
**Author:** ![nazar-pc](https://yyz1.discourse-cdn.com/flex011/user_avatar/forum.autonomys.xyz/nazar-pc/32/6_2.png) [@nazar-pc](https://forum.autonomys.xyz/u/nazar-pc)\ **Post date:** [November 1, 2023, 10:37pm UTC](https://forum.autonomys.xyz/t/gemini-3g-nodes-not-syncing-on-linux/2090/3 "2023-11-01T22:37:18Z")
Please attach node logs all the way since it was started. Always post logs as text instead of screenshots, screenshots are not searchable. ---
**Author:** ![erdnapa](https://avatars.discourse-cdn.com/v4/letter/e/ebca7d/32.png) [@erdnapa](https://forum.autonomys.xyz/u/erdnapa)\ **Post date:** [November 1, 2023, 10:55pm UTC](https://forum.autonomys.xyz/t/gemini-3g-nodes-not-syncing-on-linux/2090/4 "2023-11-01T22:55:48Z")
Most recent log. I restarted server and router. I can’t find a way to attach a file. > **[2023-11-01T22:14:36.803405Z Subspace 2023-11-01T22:14:36.803562Z ✌ version...](https://pastebin.com/aRWNTK2M)** > > Pastebin.com is the number one paste tool since 2002. Pastebin is a website where you can store text online for a set period of time. ---
**Author:** ![nazar-pc](https://yyz1.discourse-cdn.com/flex011/user_avatar/forum.autonomys.xyz/nazar-pc/32/6_2.png) [@nazar-pc](https://forum.autonomys.xyz/u/nazar-pc)\ **Post date:** [November 2, 2023, 2:34am UTC](https://forum.autonomys.xyz/t/gemini-3g-nodes-not-syncing-on-linux/2090/5 "2023-11-02T02:34:00Z")
pastebin is the right way to do it 👍 Can you start the node with `RUST_LOG=info,subspace_service=trace` and post logs after 30 minutes if it didn’t start syncing? ---
**Author:** ![erdnapa](https://avatars.discourse-cdn.com/v4/letter/e/ebca7d/32.png) [@erdnapa](https://forum.autonomys.xyz/u/erdnapa)\ **Post date:** [November 2, 2023, 2:38am UTC](https://forum.autonomys.xyz/t/gemini-3g-nodes-not-syncing-on-linux/2090/6 "2023-11-02T02:38:24Z")
Restarted already. Will post in 30+min. ---
**Author:** ![erdnapa](https://avatars.discourse-cdn.com/v4/letter/e/ebca7d/32.png) [@erdnapa](https://forum.autonomys.xyz/u/erdnapa)\ **Post date:** [November 2, 2023, 2:42am UTC](https://forum.autonomys.xyz/t/gemini-3g-nodes-not-syncing-on-linux/2090/7 "2023-11-02T02:42:56Z")
Damn it! That is syncing with the debug logging. ```auto 2023-11-02T02:42:49.427572Z TRACE tokio-runtime-worker subspace_service::sync_from_dsn::import_blocks: [Consensus] Number of importing blocks reached queue limit, waiting before retrying block_num ber=3297 best_block_number=2797 2023-11-02T02:42:50.428491Z TRACE tokio-runtime-worker subspace_service::sync_from_dsn::import_blocks: [Consensus] Number of importing blocks reached queue limit, waiting before retrying block_num ber=3316 best_block_number=2816 2023-11-02T02:42:51.429527Z TRACE tokio-runtime-worker subspace_service::sync_from_dsn::import_blocks: [Consensus] Number of importing blocks reached queue limit, waiting before retrying block_num ber=3364 best_block_number=2864 2023-11-02T02:42:52.430518Z TRACE tokio-runtime-worker subspace_service::sync_from_dsn::import_blocks: [Consensus] Number of importing blocks reached queue limit, waiting before retrying block_num ber=3381 best_block_number=2881 2023-11-02T02:42:53.432361Z TRACE tokio-runtime-worker subspace_service::sync_from_dsn::import_blocks: [Consensus] Number of importing blocks reached queue limit, waiting before retrying block_num ber=3435 best_block_number=2935 2023-11-02T02:42:53.699429Z INFO tokio-runtime-worker substrate: [Consensus] ⚙ Syncing 36.8 bps, target=#29550 (40 peers), best: #2946 (0x472a…3df9), finalized #0 (0x4180…180b), ⬇ 50.8kiB/s ⬆ 1. 7kiB/s 2023-11-02T02:42:54.433851Z TRACE tokio-runtime-worker subspace_service::sync_from_dsn::import_blocks: [Consensus] Number of importing blocks reached queue limit, waiting before retrying block_num ber=3459 best_block_number=2959 2023-11-02T02:42:55.434652Z TRACE tokio-runtime-worker subspace_service::sync_from_dsn::import_blocks: [Consensus] Number of importing blocks reached queue limit, waiting before retrying block_num ber=3491 best_block_number=2991 2023-11-02T02:42:56.435860Z TRACE tokio-runtime-worker subspace_service::sync_from_dsn::import_blocks: [Consensus] Number of importing blocks reached queue limit, waiting before retrying block_num ber ``` First time today. 🙄 ---
**Author:** ![tao\_sun](https://yyz1.discourse-cdn.com/flex011/user_avatar/forum.autonomys.xyz/tao_sun/32/825_2.png) [@tao\_sun](https://forum.autonomys.xyz/u/tao_sun)\ **Post date:** [November 2, 2023, 2:45am UTC](https://forum.autonomys.xyz/t/gemini-3g-nodes-not-syncing-on-linux/2090/8 "2023-11-02T02:45:48Z")
me too!!! it works now ! ---
**Author:** ![tao\_sun](https://yyz1.discourse-cdn.com/flex011/user_avatar/forum.autonomys.xyz/tao_sun/32/825_2.png) [@tao\_sun](https://forum.autonomys.xyz/u/tao_sun)\ **Post date:** [November 2, 2023, 2:46am UTC](https://forum.autonomys.xyz/t/gemini-3g-nodes-not-syncing-on-linux/2090/9 "2023-11-02T02:46:52Z")
me too , it is working now , syncing ,amazing ---
**Author:** ![nazar-pc](https://yyz1.discourse-cdn.com/flex011/user_avatar/forum.autonomys.xyz/nazar-pc/32/6_2.png) [@nazar-pc](https://forum.autonomys.xyz/u/nazar-pc)\ **Post date:** [November 2, 2023, 2:52am UTC](https://forum.autonomys.xyz/t/gemini-3g-nodes-not-syncing-on-linux/2090/10 "2023-11-02T02:52:57Z")
Try to restart without those logs, it should still work just fine. ---
**Author:** ![erdnapa](https://avatars.discourse-cdn.com/v4/letter/e/ebca7d/32.png) [@erdnapa](https://forum.autonomys.xyz/u/erdnapa)\ **Post date:** [November 2, 2023, 2:56am UTC](https://forum.autonomys.xyz/t/gemini-3g-nodes-not-syncing-on-linux/2090/11 "2023-11-02T02:56:41Z")
Restarted without debug log and it keeps syncing. 30 hours of additional gray hair! Attempt 15 worked: ```auto node_20231101_1138_1974553.log node_20231101_1212_1975200.log node_20231101_1227_1975644.log node_20231101_1329_1976552.log node_20231101_1704_2135349.log node_20231101_1723_2137043.log node_20231101_1833_2138809.log node_20231101_2011_2879.log node_20231101_2035_3256.log node_20231101_2156_3864.log node_20231101_2214_4208.log node_20231102_0112_114560.log node_20231102_0216_115992.log node_20231102_0217_116009.log node_20231102_0237_116950.log node_20231102_0245_117368.log ``` Thanks a LOT for the help! Any idea what fixed it? (My node with the specified ports also started to sync after restart now.) ---
**Author:** ![erdnapa](https://avatars.discourse-cdn.com/v4/letter/e/ebca7d/32.png) [@erdnapa](https://forum.autonomys.xyz/u/erdnapa)\ **Post date:** [November 2, 2023, 2:58am UTC](https://forum.autonomys.xyz/t/gemini-3g-nodes-not-syncing-on-linux/2090/12 "2023-11-02T02:58:01Z")
Glad I’m not the only one. Discord made it really look like I’m the odd one out. ---
**Author:** ![nazar-pc](https://yyz1.discourse-cdn.com/flex011/user_avatar/forum.autonomys.xyz/nazar-pc/32/6_2.png) [@nazar-pc](https://forum.autonomys.xyz/u/nazar-pc)\ **Post date:** [November 2, 2023, 3:02am UTC](https://forum.autonomys.xyz/t/gemini-3g-nodes-not-syncing-on-linux/2090/13 "2023-11-02T03:02:06Z")
Not sure yet, would be helpful to get logs from someone who is still experiencing it. I’m sure we’ll figure it out, just hoping sooner rather than later. ---
**Author:** ![tao\_sun](https://yyz1.discourse-cdn.com/flex011/user_avatar/forum.autonomys.xyz/tao_sun/32/825_2.png) [@tao\_sun](https://forum.autonomys.xyz/u/tao_sun)\ **Post date:** [November 2, 2023, 3:02am UTC](https://forum.autonomys.xyz/t/gemini-3g-nodes-not-syncing-on-linux/2090/14 "2023-11-02T03:02:39Z")
When I opened the farmer, the node stopped synchronizing again… ---
**Author:** ![tao\_sun](https://yyz1.discourse-cdn.com/flex011/user_avatar/forum.autonomys.xyz/tao_sun/32/825_2.png) [@tao\_sun](https://forum.autonomys.xyz/u/tao_sun)\ **Post date:** [November 2, 2023, 3:07am UTC](https://forum.autonomys.xyz/t/gemini-3g-nodes-not-syncing-on-linux/2090/15 "2023-11-02T03:07:34Z")
I plan to wait until the synchronization is complete before attempting to open the farmer ---
**Author:** ![erdnapa](https://avatars.discourse-cdn.com/v4/letter/e/ebca7d/32.png) [@erdnapa](https://forum.autonomys.xyz/u/erdnapa)\ **Post date:** [November 2, 2023, 3:08am UTC](https://forum.autonomys.xyz/t/gemini-3g-nodes-not-syncing-on-linux/2090/16 "2023-11-02T03:08:05Z")
Full debug log when it started to sync. Not sure it helps, since it started to sync very fast. > **[2023-11-02T02:37:09.122314Z INFO main sc\_cli::runner: Subspace 2023-11-02T...](https://pastebin.com/DC2EbPhC)** > > Pastebin.com is the number one paste tool since 2002. Pastebin is a website where you can store text online for a set period of time. ---
**Author:** ![erdnapa](https://avatars.discourse-cdn.com/v4/letter/e/ebca7d/32.png) [@erdnapa](https://forum.autonomys.xyz/u/erdnapa)\ **Post date:** [November 2, 2023, 3:18am UTC](https://forum.autonomys.xyz/t/gemini-3g-nodes-not-syncing-on-linux/2090/17 "2023-11-02T03:18:26Z")
I also have a 5min debug log from yesterday when it didn’t sync. Log level debug - not the specialized instructions from today. It is 41MB, too big for pastebin. Also not as clinically run, i.e. some ports may not be forwarded etc. Let me know if you’d like to get the whole file or if I should look for something specific. Those are the last few seconds before I killed the process: [2023-11-01T03:55:09.901683Z DEBUG tokio-runtime-worker netlink\_proto::connection - Pastebin.com](https://pastebin.com/i31XhjaJ) ---
**Author:** ![erdnapa](https://avatars.discourse-cdn.com/v4/letter/e/ebca7d/32.png) [@erdnapa](https://forum.autonomys.xyz/u/erdnapa)\ **Post date:** [November 2, 2023, 3:33am UTC](https://forum.autonomys.xyz/t/gemini-3g-nodes-not-syncing-on-linux/2090/18 "2023-11-02T03:33:08Z")
Farmer started without issues for me. ---
**Author:** ![nazar-pc](https://yyz1.discourse-cdn.com/flex011/user_avatar/forum.autonomys.xyz/nazar-pc/32/6_2.png) [@nazar-pc](https://forum.autonomys.xyz/u/nazar-pc)\ **Post date:** [November 2, 2023, 3:43am UTC](https://forum.autonomys.xyz/t/gemini-3g-nodes-not-syncing-on-linux/2090/19 "2023-11-02T03:43:13Z")
The whole file might be helpful indeed 🙏 ---
**Author:** ![erdnapa](https://avatars.discourse-cdn.com/v4/letter/e/ebca7d/32.png) [@erdnapa](https://forum.autonomys.xyz/u/erdnapa)\ **Post date:** [November 2, 2023, 3:52am UTC](https://forum.autonomys.xyz/t/gemini-3g-nodes-not-syncing-on-linux/2090/20 "2023-11-02T03:52:31Z")
[https://drive.google.com/drive/folders/14\_ld\_FRpGbLaWj9fItDRja5LJlTE4zup?usp=sharing](https://drive.google.com/drive/folders/14_ld_FRpGbLaWj9fItDRja5LJlTE4zup?usp=sharing) Please let me know when you got it. Thanks! [Next page](https://forum.autonomys.xyz/t/gemini-3g-nodes-not-syncing-on-linux/2090.md?page=2)