# Delay before starting to sync on build sep-13-2 **URL:** \ **Category:** Support\ **Created:** [September 14, 2023, 12:56pm UTC](https://forum.autonomys.xyz/t/delay-before-starting-to-sync-on-build-sep-13-2/1878 "2023-09-14T12:56:57Z")\ **Posts on this page:** 18\ **Page:** 1
**Author:** ![Jim-Autonomys](https://yyz1.discourse-cdn.com/flex011/user_avatar/forum.autonomys.xyz/jim-autonomys/32/2922_2.png) [@Jim-Autonomys](https://forum.autonomys.xyz/u/Jim-Autonomys)\ **Post date:** [September 14, 2023, 12:56pm UTC](https://forum.autonomys.xyz/t/delay-before-starting-to-sync-on-build-sep-13-2/1878/1 "2023-09-14T12:56:57Z")
A few community members are reporting they are unable to sync on the latest version `sep-13-2` and are switching back to `sep-11` . I have tested it out and there _is_ a while spent at 0.0bps but I’ve seen this delay starting to sync since the start of Gemini 3. Are there any concerns about [these logs](https://gist.github.com/jim-counter/c9575bc2a1a391cb4f6ef7403b1ca040) (10 minutes to start) or [these ones](https://gist.github.com/jim-counter/f4a072ac45ab484da4748bf5a910c06c) (25 minutes to start)? ---
**Author:** ![nazar-pc](https://yyz1.discourse-cdn.com/flex011/user_avatar/forum.autonomys.xyz/nazar-pc/32/6_2.png) [@nazar-pc](https://forum.autonomys.xyz/u/nazar-pc)\ **Post date:** [September 14, 2023, 12:58pm UTC](https://forum.autonomys.xyz/t/delay-before-starting-to-sync-on-build-sep-13-2/1878/2 "2023-09-14T12:58:47Z")
It should eventually finish sync (it is reporting 0.0bps because it doesn’t import any blocks, but that doesn’t mean it doesn’t do anything). There are a few Substratisms we’re fighting here that prevent us from making the process faster, for example [Pruning doesn't have expected behavior · Issue #14758 · paritytech/substrate · GitHub](https://github.com/paritytech/substrate/issues/14758) describes why we can’t finalize blocks that we want to finalize, which sync from DSN depends on. We’ll get there, but currently it is somewhat expected. But it does start eventually. The only reason sep-11 “works better” is because it fails to sync from DSN faster and sep-13-2 actually succeeds, which takes more time. But sep-11 and Substrate sync generally suffer from unresolved (for now) [Node can't sync with the network · Issue #493 · paritytech/polkadot-sdk · GitHub](https://github.com/paritytech/polkadot-sdk/issues/493) that DSN sync is able to bypass eventually, so there is no perfect solution here. As more nodes become pruned in Subspace network, sep-11 will eventually fail to start from genesis, while sep-13-2 will work successfully. ---
**Author:** ![xorinox](https://avatars.discourse-cdn.com/v4/letter/x/e68b1a/32.png) [@xorinox](https://forum.autonomys.xyz/u/xorinox)\ **Post date:** [September 16, 2023, 5:17pm UTC](https://forum.autonomys.xyz/t/delay-before-starting-to-sync-on-build-sep-13-2/1878/3 "2023-09-16T17:17:59Z")
I don’t know what to say about this, I had 5 nodes updated to 13-2 none of them was able to sync over night (they were synced before), they stayed unsync and my farmers didn’t gain any rewards for a window of 6-8 hours. I rolled them back to sep-11 and each node went sync in under 1 hour and farmers went back online. ---
**Author:** ![nazar-pc](https://yyz1.discourse-cdn.com/flex011/user_avatar/forum.autonomys.xyz/nazar-pc/32/6_2.png) [@nazar-pc](https://forum.autonomys.xyz/u/nazar-pc)\ **Post date:** [September 16, 2023, 5:29pm UTC](https://forum.autonomys.xyz/t/delay-before-starting-to-sync-on-build-sep-13-2/1878/4 "2023-09-16T17:29:57Z")
Was there anything interesting in logs except 0.0 bps speed, was node using any CPU at all? ---
**Author:** ![xorinox](https://avatars.discourse-cdn.com/v4/letter/x/e68b1a/32.png) [@xorinox](https://forum.autonomys.xyz/u/xorinox)\ **Post date:** [September 16, 2023, 5:32pm UTC](https://forum.autonomys.xyz/t/delay-before-starting-to-sync-on-build-sep-13-2/1878/5 "2023-09-16T17:32:36Z")
Nothing obvious to me other than 0.0 bps speed (like I usually do a grep -v INFO to find the unwanted stuff). I have upgraded one of these 5 nodes again to 13-2 since you insist this is the version to go. I can post results in a few hours. I didn’t look at the CPU load of the node specifically. ---
**Author:** ![nazar-pc](https://yyz1.discourse-cdn.com/flex011/user_avatar/forum.autonomys.xyz/nazar-pc/32/6_2.png) [@nazar-pc](https://forum.autonomys.xyz/u/nazar-pc)\ **Post date:** [September 16, 2023, 5:46pm UTC](https://forum.autonomys.xyz/t/delay-before-starting-to-sync-on-build-sep-13-2/1878/6 "2023-09-16T17:46:36Z")
If CPU load was there then it was busy trying to catch up from last finalized block, the reason for that might be very low last finalized block as explained above with some Substratisms we’re still fighting. There are things we can and will do about this both upstream and downstream. ---
**Author:** ![xorinox](https://avatars.discourse-cdn.com/v4/letter/x/e68b1a/32.png) [@xorinox](https://forum.autonomys.xyz/u/xorinox)\ **Post date:** [September 16, 2023, 6:00pm UTC](https://forum.autonomys.xyz/t/delay-before-starting-to-sync-on-build-sep-13-2/1878/7 "2023-09-16T18:00:41Z")
So far didn’t pick up a new block > Blockquote > 2023-09-16 13:32:32 [Consensus] Last archived block 40345 > 2023-09-16 13:32:32 [Consensus] Archiving already produced blocks 40346…=54821 > 2023-09-16 13:38:23 [Consensus] 🧑\<200d\>:ear\_of\_rice: Starting Subspace Authorship worker > 2023-09-16 13:38:23 [Consensus] 💻 Operating system: linux > 2023-09-16 13:38:23 [Consensus] 💻 CPU architecture: x86\_64 > 2023-09-16 13:38:23 [Consensus] 💻 Target environment: gnu > 2023-09-16 13:38:23 [Consensus] 💻 CPU: AMD EPYC 7513 32-Core Processor > 2023-09-16 13:38:23 [Consensus] 💻 CPU cores: 64 > 2023-09-16 13:38:23 [Consensus] 💻 Memory: 257748MB > 2023-09-16 13:38:23 [Consensus] 💻 Kernel: 6.4.13-100.fc37.x86\_64 > 2023-09-16 13:38:23 [Consensus] 💻 Linux distribution: Fedora Linux 37 (Server Edition) > 2023-09-16 13:38:23 [Consensus] 💻 Virtual machine: no > 2023-09-16 13:38:23 [Consensus] 📦 Highest known block at #54921 > … > … > … > 2023-09-16 13:38:24 [Consensus] Received notification to sync from DSN reason=WentOnlineSubstrate > 2023-09-16 13:38:28 [Consensus] ⚙ Syncing, target=#456954 (20 peers), best: #54921 (0xc776…6620), finalized #0 (0x92e9…5095), ⬇ 34.8kiB/s ⬆ 20.0kiB/s > 2023-09-16 13:38:33 [Consensus] ⚙ Syncing 0.0 bps, target=#456954 (40 peers), best: #54921 (0xc776…6620), finalized #0 (0x92e9…5095), ⬇ 61.2kiB/s ⬆ 28.3kiB/s > 2023-09-16 13:38:38 [Consensus] ⚙ Syncing 0.0 bps, target=#456954 (40 peers), best: #54921 (0xc776…6620), finalized #0 (0x92e9…5095), ⬇ 53.6kiB/s ⬆ 37.0kiB/s > 2023-09-16 13:38:43 [Consensus] ⚙ Syncing 0.0 bps, target=#456954 (40 peers), best: #54921 (0xc776…6620), finalized #0 (0x92e9…5095), ⬇ 51.7kiB/s ⬆ 40.5kiB/s > 2023-09-16 13:38:43 [Consensus] ❌ Error while dialing /dns/telemetry.subspace.network/tcp/443/x-parity-wss/%2Fsubmit%2F: Custom { kind: Other, error: Timeout } > 2023-09-16 13:38:48 [Consensus] ⚙ Syncing 0.0 bps, target=#456954 (40 peers), best: #54921 (0xc776…6620), finalized #0 (0x92e9…5095), ⬇ 65.2kiB/s ⬆ 50.6kiB/s > 2023-09-16 13:38:53 [Consensus] ⚙ Syncing 0.0 bps, target=#456954 (40 peers), best: #54921 (0xc776…6620), finalized #0 (0x92e9…5095), ⬇ 64.2kiB/s ⬆ 57.9kiB/s > 2023-09-16 13:38:58 [Consensus] ⚙ Syncing 0.0 bps, target=#456958 (33 peers), best: #54921 (0xc776…6620), finalized #0 (0x92e9…5095), ⬇ 68.6kiB/s ⬆ 61.5kiB/s > 2023-09-16 13:39:03 [Consensus] ⚙ Syncing 0.0 bps, target=#456960 (40 peers), best: #54921 (0xc776…6620), finalized #0 (0x92e9…5095), ⬇ 59.1kiB/s ⬆ 48.8kiB/s > … > … > … > 2023-09-16 13:59:48 [Consensus] ⚙ Syncing 0.0 bps, target=#457181 (28 peers), best: #54921 (0xc776…6620), finalized #0 (0x92e9…5095), ⬇ 38.6kiB/s ⬆ 0.8kiB/s > 2023-09-16 13:59:53 [Consensus] ⚙ Syncing 0.0 bps, target=#457181 (28 peers), best: #54921 (0xc776…6620), finalized #0 (0x92e9…5095), ⬇ 28.6kiB/s ⬆ 116.9kiB/s > 2023-09-16 13:59:58 [Consensus] ⚙ Syncing 0.0 bps, target=#457183 (27 peers), best: #54921 (0xc776…6620), finalized #0 (0x92e9…5095), ⬇ 33.2kiB/s ⬆ 87.3kiB/s > 2023-09-16 14:00:03 [Consensus] ⚙ Syncing 0.0 bps, target=#457183 (27 peers), best: #54921 (0xc776…6620), finalized #0 (0x92e9…5095), ⬇ 22.7kiB/s ⬆ 35.5kiB/s > 2023-09-16 14:00:08 [Consensus] ⚙ Syncing 0.0 bps, target=#457183 (28 peers), best: #54921 (0xc776…6620), finalized #0 (0x92e9…5095), ⬇ 26.4kiB/s ⬆ 1.6kiB/s 8% CPU load ![008](https://canada1.discourse-cdn.com/flex011/uploads/subspace/original/2X/0/0891a95ebc2b4aa1dee539bcf4acd4b78320bb5d.jpeg) ---
**Author:** ![xorinox](https://avatars.discourse-cdn.com/v4/letter/x/e68b1a/32.png) [@xorinox](https://forum.autonomys.xyz/u/xorinox)\ **Post date:** [September 17, 2023, 2:38am UTC](https://forum.autonomys.xyz/t/delay-before-starting-to-sync-on-build-sep-13-2/1878/8 "2023-09-17T02:38:38Z")
I looked at the entire log since I posted earlier. The only thing that changes are the target. It’s not syncing and the CPU is not busy (~5% subspace-node process). The only ERROR since earlier this afternoon is posted below. > Blockquote > 2023-09-16 18:31:54 [Consensus] ⚙ Syncing 0.0 bps, target=#459972 (40 peers), best: #54921 (0xc776…6620), finalized #0 (0x92e9…5095), ⬇ 49.6kiB/s ⬆ 2.0kiB/s > 2023-09-16 18:31:59 [Consensus] ⚙ Syncing 0.0 bps, target=#459973 (40 peers), best: #54921 (0xc776…6620), finalized #0 (0x92e9…5095), ⬇ 54.2kiB/s ⬆ 2.2kiB/s > 2023-09-16 18:32:04 [Consensus] ⚙ Syncing 0.0 bps, target=#459973 (40 peers), best: #54921 (0xc776…6620), finalized #0 (0x92e9…5095), ⬇ 45.4kiB/s ⬆ 1.2kiB/s > 2023-09-16 18:32:09 [Consensus] ⚙ Syncing 0.0 bps, target=#459974 (40 peers), best: #54921 (0xc776…6620), finalized #0 (0x92e9…5095), ⬇ 63.1kiB/s ⬆ 1.3kiB/s > 2023-09-16 18:32:14 [Consensus] ⚙ Syncing 0.0 bps, target=#459975 (40 peers), best: #54921 (0xc776…6620), finalized #0 (0x92e9…5095), ⬇ 67.5kiB/s ⬆ 2.2kiB/s > 2023-09-16 18:32:18 [Consensus] Error when syncing blocks from DSN error=Other: Error during data shards reconstruction: Impossible to recover, too many shards are missing > 2023-09-16 18:32:18 [Consensus] Received notification to sync from DSN reason=WentOnlineSubspace > 2023-09-16 18:32:19 [Consensus] ⚙ Syncing 0.0 bps, target=#459976 (40 peers), best: #54921 (0xc776…6620), finalized #0 (0x92e9…5095), ⬇ 50.0kiB/s ⬆ 1.5kiB/s > 2023-09-16 18:32:24 [Consensus] ⚙ Syncing 0.0 bps, target=#459977 (40 peers), best: #54921 (0xc776…6620), finalized #0 (0x92e9…5095), ⬇ 68.3kiB/s ⬆ 1.3kiB/s > 2023-09-16 18:32:29 [Consensus] ⚙ Syncing 0.0 bps, target=#459978 (40 peers), best: #54921 (0xc776…6620), finalized #0 (0x92e9…5095), ⬇ 51.3kiB/s ⬆ 1.8kiB/s > 2023-09-16 18:32:34 [Consensus] ⚙ Syncing 0.0 bps, target=#459978 (40 peers), best: #54921 (0xc776…6620), finalized #0 (0x92e9…5095), ⬇ 61.2kiB/s ⬆ 0.9kiB/s > 2023-09-16 18:32:39 [Consensus] ⚙ Syncing 0.0 bps, target=#459978 (40 peers), best: #54921 (0xc776…6620), finalized #0 (0x92e9…5095), ⬇ 46.0kiB/s ⬆ 0.6kiB/s ---
**Author:** ![nazar-pc](https://yyz1.discourse-cdn.com/flex011/user_avatar/forum.autonomys.xyz/nazar-pc/32/6_2.png) [@nazar-pc](https://forum.autonomys.xyz/u/nazar-pc)\ **Post date:** [September 17, 2023, 3:15am UTC](https://forum.autonomys.xyz/t/delay-before-starting-to-sync-on-build-sep-13-2/1878/9 "2023-09-17T03:15:16Z")
> [@xorinox](#): > > 2023-09-16 18:32:18 [Consensus] Error when syncing blocks from DSN error=Other: Error during data shards reconstruction: Impossible to recover, too many shards are missing This is a problem. Is it possible that you have bad connectivity to the rest of the network? `RUST_LOG=info,subspace_service=trace` will tell you more about what node is doing. ---
**Author:** ![xorinox](https://avatars.discourse-cdn.com/v4/letter/x/e68b1a/32.png) [@xorinox](https://forum.autonomys.xyz/u/xorinox)\ **Post date:** [September 17, 2023, 3:22am UTC](https://forum.autonomys.xyz/t/delay-before-starting-to-sync-on-build-sep-13-2/1878/10 "2023-09-17T03:22:14Z")
That network of this machine is stable, it’s located in a high end data center, I won’t exclude issues, but the server rack has 5 nodes, 4 are in sync with sep-11 and the 5th with 13-2 doesn’t sync. I can restart the node with RUST\_LOG on ---
**Author:** ![xorinox](https://avatars.discourse-cdn.com/v4/letter/x/e68b1a/32.png) [@xorinox](https://forum.autonomys.xyz/u/xorinox)\ **Post date:** [September 17, 2023, 1:53pm UTC](https://forum.autonomys.xyz/t/delay-before-starting-to-sync-on-build-sep-13-2/1878/11 "2023-09-17T13:53:27Z")
Here is the trace of when the error was happening again (Error during data shards reconstruction: Impossible to recover, too many shards are missing) The node is not synced at all, has not made a single add block of progress. > Blockquote > 2023-09-17 05:04:06.334 INFO tokio-runtime-worker substrate: [Consensus] ⚙ Syncing 0.0 bps, target=#466156 (40 peers), best: #54921 (0xc776…6620), finalized #0 (0x92e9…5095), ⬇ 46.3kiB/s ⬆ 2.8kiB/s > 2023-09-17 05:04:11.334 INFO tokio-runtime-worker substrate: [Consensus] ⚙ Syncing 0.0 bps, target=#466156 (40 peers), best: #54921 (0xc776…6620), finalized #0 (0x92e9…5095), ⬇ 38.1kiB/s ⬆ 2.0kiB/s > 2023-09-17 05:04:16.334 INFO tokio-runtime-worker substrate: [Consensus] ⚙ Syncing 0.0 bps, target=#466156 (40 peers), best: #54921 (0xc776…6620), finalized #0 (0x92e9…5095), ⬇ 37.7kiB/s ⬆ 1.3kiB/s > 2023-09-17 05:04:21.335 INFO tokio-runtime-worker substrate: [Consensus] ⚙ Syncing 0.0 bps, target=#466157 (40 peers), best: #54921 (0xc776…6620), finalized #0 (0x92e9…5095), ⬇ 81.9kiB/s ⬆ 0.9kiB/s > 2023-09-17 05:04:26.335 INFO tokio-runtime-worker substrate: [Consensus] ⚙ Syncing 0.0 bps, target=#466159 (40 peers), best: #54921 (0xc776…6620), finalized #0 (0x92e9…5095), ⬇ 50.4kiB/s ⬆ 1.8kiB/s > 2023-09-17 05:04:31.335 INFO tokio-runtime-worker substrate: [Consensus] ⚙ Syncing 0.0 bps, target=#466160 (40 peers), best: #54921 (0xc776…6620), finalized #0 (0x92e9…5095), ⬇ 63.6kiB/s ⬆ 1.1kiB/s > 2023-09-17 05:04:36.336 INFO tokio-runtime-worker substrate: [Consensus] ⚙ Syncing 0.0 bps, target=#466160 (40 peers), best: #54921 (0xc776…6620), finalized #0 (0x92e9…5095), ⬇ 60.2kiB/s ⬆ 1.7kiB/s > 2023-09-17 05:04:39.231 TRACE tokio-runtime-worker subspace\_service::sync\_from\_dsn::import\_blocks: [Consensus] Piece request succeeded piece\_index=PieceIndex(511) piece\_found=false > 2023-09-17 05:04:41.336 INFO tokio-runtime-worker substrate: [Consensus] ⚙ Syncing 0.0 bps, target=#466162 (40 peers), best: #54921 (0xc776…6620), finalized #0 (0x92e9…5095), ⬇ 39.4kiB/s ⬆ 1.5kiB/s > 2023-09-17 05:04:46.336 INFO tokio-runtime-worker substrate: [Consensus] ⚙ Syncing 0.0 bps, target=#466163 (40 peers), best: #54921 (0xc776…6620), finalized #0 (0x92e9…5095), ⬇ 67.7kiB/s ⬆ 0.7kiB/s > 2023-09-17 05:04:51.337 INFO tokio-runtime-worker substrate: [Consensus] ⚙ Syncing 0.0 bps, target=#466164 (40 peers), best: #54921 (0xc776…6620), finalized #0 (0x92e9…5095), ⬇ 52.4kiB/s ⬆ 0.6kiB/s > 2023-09-17 05:04:56.337 INFO tokio-runtime-worker substrate: [Consensus] ⚙ Syncing 0.0 bps, target=#466165 (40 peers), best: #54921 (0xc776…6620), finalized #0 (0x92e9…5095), ⬇ 42.5kiB/s ⬆ 0.6kiB/s > 2023-09-17 05:05:01.338 INFO tokio-runtime-worker substrate: [Consensus] ⚙ Syncing 0.0 bps, target=#466166 (40 peers), best: #54921 (0xc776…6620), finalized #0 (0x92e9…5095), ⬇ 45.6kiB/s ⬆ 1.0kiB/s > 2023-09-17 05:05:06.338 INFO tokio-runtime-worker substrate: [Consensus] ⚙ Syncing 0.0 bps, target=#466167 (40 peers), best: #54921 (0xc776…6620), finalized #0 (0x92e9…5095), ⬇ 59.2kiB/s ⬆ 1.1kiB/s > 2023-09-17 05:05:11.338 INFO tokio-runtime-worker substrate: [Consensus] ⚙ Syncing 0.0 bps, target=#466168 (40 peers), best: #54921 (0xc776…6620), finalized #0 (0x92e9…5095), ⬇ 54.5kiB/s ⬆ 1.2kiB/s > 2023-09-17 05:05:14.859 TRACE tokio-runtime-worker subspace\_service::sync\_from\_dsn::import\_blocks: [Consensus] Piece request succeeded piece\_index=PieceIndex(507) piece\_found=false > 2023-09-17 05:05:14.861 WARN tokio-runtime-worker subspace\_service::sync\_from\_dsn: [Consensus] Error when syncing blocks from DSN error=Other: Error during data shards reconstruction: Impossible to recover, too many shards are missing > 2023-09-17 05:05:14.861 INFO tokio-runtime-worker subspace\_service::sync\_from\_dsn: [Consensus] Received notification to sync from DSN reason=WentOnlineSubspace > 2023-09-17 05:05:14.861 TRACE tokio-runtime-worker subspace\_service::sync\_from\_dsn::segment\_header\_downloader: [Consensus] Searching for latest segment header last\_known\_segment\_index=30 > 2023-09-17 05:05:14.861 TRACE tokio-runtime-worker subspace\_service::sync\_from\_dsn::segment\_header\_downloader: [Consensus] Downloading last segment headers retry\_attempt=1 > 2023-09-17 05:05:16.339 INFO tokio-runtime-worker substrate: [Consensus] ⚙ Syncing 0.0 bps, target=#466170 (40 peers), best: #54921 (0xc776…6620), finalized #0 (0x92e9…5095), ⬇ 47.5kiB/s ⬆ 0.5kiB/s > 2023-09-17 05:05:21.339 INFO tokio-runtime-worker substrate: [Consensus] ⚙ Syncing 0.0 bps, target=#466171 (40 peers), best: #54921 (0xc776…6620), finalized #0 (0x92e9…5095), ⬇ 45.0kiB/s ⬆ 0.8kiB/s > 2023-09-17 05:05:26.339 INFO tokio-runtime-worker substrate: [Consensus] ⚙ Syncing 0.0 bps, target=#466172 (40 peers), best: #54921 (0xc776…6620), finalized #0 (0x92e9…5095), ⬇ 58.4kiB/s ⬆ 0.7kiB/s > 2023-09-17 05:05:30.611 TRACE tokio-runtime-worker subspace\_service::sync\_from\_dsn::segment\_header\_downloader: [Consensus] Last segment headers request succeeded peer\_id=12D3KooWFXNp8WnDbaJuwZFXAzP2hdD4WdaZEw5T5483C4AJmKm4 segment\_headers\_number=2 > 2023-09-17 05:05:30.945 TRACE tokio-runtime-worker subspace\_service::sync\_from\_dsn::segment\_header\_downloader: [Consensus] Last segment headers request succeeded peer\_id=12D3KooWLJXmLXFA7GHkZqJwXAFAjWHYv95W3hgQku3eeTjo3Aer segment\_headers\_number=2 > 2023-09-17 05:05:31.180 TRACE tokio-runtime-worker subspace\_service::sync\_from\_dsn::segment\_header\_downloader: [Consensus] Last segment headers request succeeded peer\_id=12D3KooW9pzn6TS737XoaB9qEN7XqnJc4N2hzv353Wieda1AwifN segment\_headers\_number=2 > 2023-09-17 05:05:31.340 INFO tokio-runtime-worker substrate: [Consensus] ⚙ Syncing 0.0 bps, target=#466172 (40 peers), best: #54921 (0xc776…6620), finalized #0 (0x92e9…5095), ⬇ 45.2kiB/s ⬆ 0.9kiB/s > 2023-09-17 05:05:31.762 TRACE tokio-runtime-worker subspace\_service::sync\_from\_dsn::segment\_header\_downloader: [Consensus] Last segment headers request succeeded peer\_id=12D3KooWHLsojXfP5M2ugsoffR5emqbvMhivAYkgZVudzP6xSYcU segment\_headers\_number=2 > 2023-09-17 05:05:32.673 TRACE tokio-runtime-worker subspace\_service::sync\_from\_dsn::segment\_header\_downloader: [Consensus] Last segment headers request succeeded peer\_id=12D3KooWKFqqcpjn2o3pARMFsFWDyFhzMux7p1aAeJzoJhaBdcyc segment\_headers\_number=2 > 2023-09-17 05:05:33.327 TRACE tokio-runtime-worker subspace\_service::sync\_from\_dsn::segment\_header\_downloader: [Consensus] Last segment headers request succeeded peer\_id=12D3KooWJYTJwA1zsuHSdke6YjZLXprijmws4qvAEL6F1SzeUdkJ segment\_headers\_number=2 > 2023-09-17 05:05:34.315 TRACE tokio-runtime-worker subspace\_service::sync\_from\_dsn::segment\_header\_downloader: [Consensus] Last segment headers request succeeded peer\_id=12D3KooWSGSEvSb1CpbpknxMTPisCK8giQmeC3Web9RhRusJb7qN segment\_headers\_number=2 > 2023-09-17 05:05:34.708 TRACE tokio-runtime-worker subspace\_service::sync\_from\_dsn::segment\_header\_downloader: [Consensus] Last segment headers request succeeded peer\_id=12D3KooWQfpyBYW7GirTL7gnpvCJgJZQzFKEL6vFQR7b1ha98ZBG segment\_headers\_number=2 > 2023-09-17 05:05:35.221 TRACE tokio-runtime-worker subspace\_service::sync\_from\_dsn::segment\_header\_downloader: [Consensus] Last segment headers request succeeded peer\_id=12D3KooWLSrTDT5rL1W9GDpB1KcE2u545j5Jc1MiEG3XxP3hzHV5 segment\_headers\_number=2 > 2023-09-17 05:05:35.446 TRACE tokio-runtime-worker subspace\_service::sync\_from\_dsn::segment\_header\_downloader: [Consensus] Last segment headers request succeeded peer\_id=12D3KooWSDXR57C9z2T1EbcwSJfwCJPPQDrG7Nex3UjT8cpHN9a9 segment\_headers\_number=2 > 2023-09-17 05:05:35.740 TRACE tokio-runtime-worker subspace\_service::sync\_from\_dsn::segment\_header\_downloader: [Consensus] Last segment headers request succeeded peer\_id=12D3KooWHtcxFAceo8RABudUYkMdjfTx5ntT9aBGjM5CpJ86apVr segment\_headers\_number=2 > 2023-09-17 05:05:36.137 TRACE tokio-runtime-worker subspace\_service::sync\_from\_dsn::segment\_header\_downloader: [Consensus] Last segment headers request succeeded peer\_id=12D3KooWFR1U1GuTLMq7NTocJBkKD7YXEfDxBnaAfQXfv3K8kUbg segment\_headers\_number=2 > 2023-09-17 05:05:36.340 INFO tokio-runtime-worker substrate: [Consensus] ⚙ Syncing 0.0 bps, target=#466173 (40 peers), best: #54921 (0xc776…6620), finalized #0 (0x92e9…5095), ⬇ 36.1kiB/s ⬆ 1.0kiB/s > 2023-09-17 05:05:36.357 TRACE tokio-runtime-worker subspace\_service::sync\_from\_dsn::segment\_header\_downloader: [Consensus] Last segment headers request succeeded peer\_id=12D3KooWKGD72BshrijyJFryGFyQBpi7BHEV97fxeb8tmp2ifCKu segment\_headers\_number=2 > 2023-09-17 05:05:36.690 TRACE tokio-runtime-worker subspace\_service::sync\_from\_dsn::segment\_header\_downloader: [Consensus] Last segment headers request succeeded peer\_id=12D3KooWE7JMP3JsNmgHZ5KnoBkYiJ3wTLHwH5WJMyzfvgExQ5y7 segment\_headers\_number=2 ---
**Author:** ![nazar-pc](https://yyz1.discourse-cdn.com/flex011/user_avatar/forum.autonomys.xyz/nazar-pc/32/6_2.png) [@nazar-pc](https://forum.autonomys.xyz/u/nazar-pc)\ **Post date:** [September 17, 2023, 9:12pm UTC](https://forum.autonomys.xyz/t/delay-before-starting-to-sync-on-build-sep-13-2/1878/12 "2023-09-17T21:12:25Z")
> [@xorinox](#): > > 2023-09-17 05:05:14.861 WARN tokio-runtime-worker subspace\_service::sync\_from\_dsn: [Consensus] Error when syncing blocks from DSN error=Other: Error during data shards reconstruction: Impossible to recover, too many shards are missing This is an indication of a problem. It didn’t find even half out of 256 pieces it was looking for. Moreover, judging from this: > [@xorinox](#): > > 2023-09-17 05:05:14.861 INFO tokio-runtime-worker subspace\_service::sync\_from\_dsn: [Consensus] Received notification to sync from DSN reason=WentOnlineSubspace There was zero Subspace networking connections for some reason, which is not good at all. I’m not sure what caused it, but essentially your node manages to establish a few connections, make some requests, then gets essentially “offline” and when it gets back up the process continues in a loop 😕 Longer logs would help to clarify information some more, but so far something is definitely working suboptimally or effectively not working at all. ---
**Author:** ![xorinox](https://avatars.discourse-cdn.com/v4/letter/x/e68b1a/32.png) [@xorinox](https://forum.autonomys.xyz/u/xorinox)\ **Post date:** [September 18, 2023, 2:33am UTC](https://forum.autonomys.xyz/t/delay-before-starting-to-sync-on-build-sep-13-2/1878/13 "2023-09-18T02:33:53Z")
I get the entire file to you, sometime Monday. ---
**Author:** ![xorinox](https://avatars.discourse-cdn.com/v4/letter/x/e68b1a/32.png) [@xorinox](https://forum.autonomys.xyz/u/xorinox)\ **Post date:** [September 18, 2023, 12:54pm UTC](https://forum.autonomys.xyz/t/delay-before-starting-to-sync-on-build-sep-13-2/1878/14 "2023-09-18T12:54:04Z")
I sent you @nazar-pc the file via discord ---
**Author:** ![nazar-pc](https://yyz1.discourse-cdn.com/flex011/user_avatar/forum.autonomys.xyz/nazar-pc/32/6_2.png) [@nazar-pc](https://forum.autonomys.xyz/u/nazar-pc)\ **Post date:** [September 18, 2023, 9:28pm UTC](https://forum.autonomys.xyz/t/delay-before-starting-to-sync-on-build-sep-13-2/1878/15 "2023-09-18T21:28:39Z")
Something is bad in there, piece retrieval is extremely slow and full of `piece_found=false`. I do not see such issue on my machine, all pieces my node tries to download are retrieved successfully. On your node though most of them fail. Another suspicious thing is this: ```auto 2023-09-16 23:38:25.412 DEBUG tokio-runtime-worker subspace_service::sync_from_dsn::segment_header_downloader: [Consensus] Last segment header matches last known segment header, nothing to download last_known_segment_index=30 2023-09-16 23:38:25.412 DEBUG tokio-runtime-worker subspace_service::sync_from_dsn::import_blocks: [Consensus] Found 0 new segment headers 2023-09-16 23:38:25.412 DEBUG tokio-runtime-worker subspace_service::sync_from_dsn::import_blocks: [Consensus] Processing segment segment_index=1 2023-09-16 23:38:25.412 TRACE tokio-runtime-worker subspace_service::sync_from_dsn::import_blocks: [Consensus] Checking segment header segment_index=1 last_archived_block_number=14493 last_archived_block_progress=Partial(384) ``` Your node thinks `segment_index==1` is the latest, which is very far from truth. And most of the nodes you’re connected to seem to agree, which makes little sense actually, here is what my node sees (just a few days later than your logs): ```auto 2023-09-19 00:21:43.495 DEBUG tokio-runtime-worker subspace_service::sync_from_dsn::segment_header_downloader: [Consensus] Downloading segment headers last_known_segment_index=0 last_segment_index=32 2023-09-19 00:21:43.495 TRACE tokio-runtime-worker subspace_service::sync_from_dsn::segment_header_downloader: [Consensus] Getting segment header batch.. segment_indexes=[SegmentIndex(31), SegmentIndex(30), SegmentIndex(29), SegmentIndex(28), SegmentIndex(27), SegmentIndex(26), SegmentIndex(25), SegmentIndex(24), SegmentIndex(23), SegmentIndex(22), SegmentIndex(21), SegmentIndex(20), SegmentIndex(19), SegmentIndex(18), SegmentIndex(17), SegmentIndex(16), SegmentIndex(15), SegmentIndex(14), SegmentIndex(13), SegmentIndex(12), SegmentIndex(11), SegmentIndex(10), SegmentIndex(9), SegmentIndex(8), SegmentIndex(7), SegmentIndex(6), SegmentIndex(5), SegmentIndex(4), SegmentIndex(3), SegmentIndex(2), SegmentIndex(1), SegmentIndex(0)] 2023-09-19 00:21:43.495 TRACE tokio-runtime-worker subspace_service::sync_from_dsn::segment_header_downloader: [Consensus] get_closest_peers returned an item peer_id=12D3KooWBzZsQTYUwx4XxsFe5oWcSUNADey5mmCSipW2jQuYagxy 2023-09-19 00:21:43.578 TRACE tokio-runtime-worker subspace_service::sync_from_dsn::segment_header_downloader: [Consensus] Segment header request succeeded peer_id=12D3KooWBzZsQTYUwx4XxsFe5oWcSUNADey5mmCSipW2jQuYagxy segment_indexes=[SegmentIndex(31), SegmentIndex(30), SegmentIndex(29), SegmentIndex(28), SegmentIndex(27), SegmentIndex(26), SegmentIndex(25), SegmentIndex(24), SegmentIndex(23), SegmentIndex(22), SegmentIndex(21), SegmentIndex(20), SegmentIndex(19), SegmentIndex(18), SegmentIndex(17), SegmentIndex(16), SegmentIndex(15), SegmentIndex(14), SegmentIndex(13), SegmentIndex(12), SegmentIndex(11), SegmentIndex(10), SegmentIndex(9), SegmentIndex(8), SegmentIndex(7), SegmentIndex(6), SegmentIndex(5), SegmentIndex(4), SegmentIndex(3), SegmentIndex(2), SegmentIndex(1), SegmentIndex(0)] 2023-09-19 00:21:43.578 DEBUG tokio-runtime-worker subspace_service::sync_from_dsn::import_blocks: [Consensus] Found 33 new segment headers ``` It found segment indices up to 31! What are the CLI arguments of the node in your case? There is either something off there or you’re incredibly unlucky to be stuck in these conditions. ---
**Author:** ![xorinox](https://avatars.discourse-cdn.com/v4/letter/x/e68b1a/32.png) [@xorinox](https://forum.autonomys.xyz/u/xorinox)\ **Post date:** [September 19, 2023, 1:43am UTC](https://forum.autonomys.xyz/t/delay-before-starting-to-sync-on-build-sep-13-2/1878/16 "2023-09-19T01:43:53Z")
Is there a chance it’s related to the region of where the node is located like North America? How I am starting this node see below. > Blockquote > name=“foo-02” > base\_path=“/sub/node/” > mkdir -p “$base\_path” > export RUST\_LOG=info,subspace\_service=trace > cmd="./target/x86\_64-unknown-linux-gnu/production/subspace-node > –name $name > –base-path $base\_path > –port 33930 > –rpc-port 33931 > –prometheus-port 33933 > –dsn-listen-on /ip4/0.0.0.0/tcp/33934 > –chain gemini-3f > –execution wasm > –blocks-pruning 256 > –state-pruning archive > –no-private-ipv4 > –validator > " > $cmd ---
**Author:** ![nazar-pc](https://yyz1.discourse-cdn.com/flex011/user_avatar/forum.autonomys.xyz/nazar-pc/32/6_2.png) [@nazar-pc](https://forum.autonomys.xyz/u/nazar-pc)\ **Post date:** [September 19, 2023, 1:56am UTC](https://forum.autonomys.xyz/t/delay-before-starting-to-sync-on-build-sep-13-2/1878/17 "2023-09-19T01:56:37Z")
Anything is possible of course, but we have a lot of nodes on NA according to telemetry. Try to stop the node and delete/rename `known_addresses.bin` file in node’s directory and see if it helps at all. ---
**Author:** ![xorinox](https://avatars.discourse-cdn.com/v4/letter/x/e68b1a/32.png) [@xorinox](https://forum.autonomys.xyz/u/xorinox)\ **Post date:** [September 19, 2023, 3:24am UTC](https://forum.autonomys.xyz/t/delay-before-starting-to-sync-on-build-sep-13-2/1878/18 "2023-09-19T03:24:55Z")
Ok, I try that, will see tomorrow, if that helped at all.