Can't start node error DB

Support
1

Issue Report

ERROR Domain: domain_client_operator::domain_worker: Failed to process consensus block error=UnknownBlock("Header was not found in the database:

Environment

ubuntu 22.04

  • Operating System: r9 5950x, 64gb
  • Space Acres/Advanced CLI/Docker: advanced CLI

Problem

my parameters to start

run --base-path=“/home/subspacetest” --chain=“taurus” --farmer --name=“xxx” blocks-pruning=“archive-canonical” --state-pruning=“archive-canonical” --sync full --listen-on=“/ip4/0.0.0.0/tcp/20333” --dsn-listen-on=“/ip4/0.0.0.0/tcp/20433” --rpc-listen-on 0.0.0.0:9955 --rpc-cors all --rpc-methods unsafe – --domain-id 0 --operator-id 679 --blocks-pruning archive-canonical --state-pruning 28800

2025-06-03T19:01:37.198055Z  INFO Consensus: substrate: ⚙️  Syncing  0.0 bps, target=#2700068 (26 peers), best: #2514062 (0x4266…5a5c), finalized #2461689 (0xa3fd…5227), ⬇ 15.2kiB/s ⬆ 4.9kiB/s
2025-06-03T19:01:37.301652Z  INFO Domain: substrate: ⚙️  Syncing  0.0 bps, target=#1759796 (11 peers), best: #1603660 (0x0551…2e35), finalized #1589176 (0xf6ac…e52b), ⬇ 39 B/s ⬆ 0.4kiB/s
2025-06-03T19:01:42.199896Z  INFO Consensus: substrate: ⚙️  Syncing  0.0 bps, target=#2700068 (26 peers), best: #2514062 (0x4266…5a5c), finalized #2461689 (0xa3fd…5227), ⬇ 15.5kiB/s ⬆ 7.3kiB/s
2025-06-03T19:01:42.302315Z  INFO Domain: substrate: ⚙️  Syncing  0.0 bps, target=#1759796 (11 peers), best: #1603660 (0x0551…2e35), finalized #1589176 (0xf6ac…e52b), ⬇ 64 B/s ⬆ 64 B/s
2025-06-03T19:01:44.904819Z ERROR Domain: domain_client_operator::domain_worker: Failed to process consensus block error=UnknownBlock("Header was not found in the database: 0x203948f3f24ca42b1e6766d183dcfcc5265c7ae52df5e9629a61e7c27b0ea913")
2025-06-03T19:01:44.904902Z ERROR Domain: sc_service::task_manager: Essential task `domain-operator-worker` failed. Shutting down service.
2025-06-03T19:01:44.905111Z ERROR Domain: subspace_node::commands::run: Domain starter exited with an error error=Other: Essential task failed.
2025-06-03T19:01:44.905145Z ERROR sc_service::task_manager: Essential task `domain` failed. Shutting down service.
2025-06-03T19:01:44.986754Z  WARN Consensus: consensus_sync: Error when syncing blocks from DSN error=Other: Failed to download segment pieces: Not enough (0/128) pieces for segment 696
2025-06-03T19:01:52.746837Z ERROR Consensus: sc_service::client::client: Unable to send AnnouncePin worker message for import: sending into a closed channel
2

From the log, it seems your node’s best domain block #1603660 (0x0551…2e35) is in a stale fork that derived from a missing consensus block 0x203948f3f24ca42b1e6766d183dcfcc5265c7ae52df5e9629a61e7c27b0ea913 which is also in a stale fork. And given your node’s best consensus block is #2514062 (0x4266…5a5c) which supposed to derive domain block #1,605,333, I guess the cause of this error is:

  1. The node is first running with domain enabled (e.g. subspace-node run ... -- --domain-id 0 ..)
  2. Then the node is restarted and runs without domain (e.g. subspace-node run ..)
  3. Finally, the node is again restarted and running with domain enabled (e.g. subspace-node run ... -- --domain-id 0 ..) then encountered this error.

If this is the case, unfortunately, you will have to resync from scratch and make sure the domain is always enabled.

2 Likes
3

thank u for your support, after remove old db and domains. I resync again but my finalized of consensus is 0, is it normal?

2025-06-06T20:53:13.448479Z  INFO Domain: substrate: ⚙️  Syncing  1.0 bps, target=#1791213 (14 peers), best: #1167350 (0x7597…71cc), finalized #1152882 (0x4b04…0aa3), ⬇ 15.9kiB/s ⬆ 2.1kiB/s
2025-06-06T20:53:14.086368Z  INFO Consensus: substrate: ⚙️  Syncing  1.6 bps, target=#2738146 (40 peers), best: #1901654 (0x7e6f…2d7b), finalized #0 (0x295a…6c5e), ⬇ 58.8kiB/s ⬆ 12.9kiB/s
2025-06-06T20:53:14.455926Z  WARN Domain: domain_client_operator::bundle_processor: Slow domain block execution, took 1916ms consensus_block_info=(0xb9f5edaa71b88f323742cf83883791c64db7f0fe9f1914cfb2636d0bc4d21faf, 1895442) built_block_info=(0x495dec8af175ebb98a20740f34029634a2cf0c3e1d2695288e3a84f65cc04372, 1167351) reference_block_execution_duration_ms=8
2025-06-06T20:53:15.577281Z  WARN Domain: domain_client_operator::bundle_processor: Slow domain block execution, took 848ms consensus_block_info=(0x8829b0a66381df610561d7de191e67808f2d2d033f0ac38e84fde652acbed1ba, 1895452) built_block_info=(0x5782b6402a31c429ad854a74d11e4c5c01dae5f45fa4f7b2ac48d219b6ffedbd, 1167356) reference_block_execution_duration_ms=4
2025-06-06T20:53:18.273364Z  WARN Domain: domain_client_operator::bundle_processor: Slow domain block execution, took 2370ms consensus_block_info=(0x10bf74882e05eab7d51ee75572d92ddac81fedaef69a348a24ab962dae133059, 1895457) built_block_info=(0x3e481b36cf75ab70dc8ba227dcf9f71d02ce16b9af10135fad00d342f446fd1e, 1167360) reference_block_execution_duration_ms=10
2025-06-06T20:53:18.448686Z  INFO Domain: substrate: ⚙️  Syncing  2.2 bps, target=#1791214 (14 peers), best: #1167361 (0x78bf…9c1f), finalized #1152894 (0xdb4b…f87b), ⬇ 0.9kiB/s ⬆ 71 B/s
2025-06-06T20:53:19.015962Z  WARN Domain: domain_client_operator::bundle_processor: Slow domain block execution, took 712ms consensus_block_info=(0xdb7736f09c9b273f3e666f501ffb10ab64399c36d05d717ccd598c640c540766, 1895460) built_block_info=(0x25e8e7ff6b75cf088bbef0a29b0b081b971194039a44807b72edfaf3f08ed95b, 1167362) reference_block_execution_duration_ms=4
2025-06-06T20:53:19.086493Z  INFO Consensus: substrate: ⚙️  Syncing  1.6 bps, target=#2738148 (40 peers), best: #1901662 (0x6b30…0ebc), finalized #0 (0x295a…6c5e), ⬇ 25.1kiB/s ⬆ 11.8kiB/s
2025-06-06T20:53:19.438590Z  WARN Domain: domain_client_operator::bundle_processor: Slow domain block execution, took 391ms consensus_block_info=(0x9e8c727399d65f8ea0f6faaa838de79d710baaf90b3ef570d693b5ecb94cece4, 1895462) built_block_info=(0x3e38158230662f2b5dabd531010ecf28d2567eb6e9d18686aae56cb1abe57e32, 1167364) reference_block_execution_duration_ms=2
2025-06-06T20:53:19.791382Z  WARN Domain: domain_client_operator::bundle_processor: Slow domain block execution, took 266ms consensus_block_info=(0xdf83bc297a9f81ceb99f69d383236c3b0333aa3e233f7a502bb7996bc73747f8, 1895466) built_block_info=(0x92e1bca91699fefaf8dd511edc1f6b708afe4e847eff46901bff5f57e21f30cd, 1167366) reference_block_execution_duration_ms=2
2025-06-06T20:53:20.349222Z  WARN Domain: domain_client_operator::bundle_processor: Slow domain block execution, took 521ms consensus_block_info=(0x4f2d3d2b6eca9b5b6b7100cba90271322d99c09475e4dc0b17553ccf1a86ac36, 1895467) built_block_info=(0xfe5cf2b90d70d30c992b52eec365a31c1f191557f1a0b7536b47d324770da310, 1167367) reference_block_execution_duration_ms=3
2025-06-06T20:53:23.448853Z  INFO Domain: substrate: ⚙️  Syncing  2.2 bps, target=#1791214 (14 peers), best: #1167372 (0x314a…af72), finalized #1152904 (0xc8b0…54a5), ⬇ 2.0kiB/s ⬆ 0.3kiB/s
2025-06-06T20:53:23.678066Z  WARN Domain: domain_client_operator::bundle_processor: Slow domain block execution, took 2923ms consensus_block_info=(0xfba272a9ff9e73e5ddf9beecdc81bc75bcd26f6a2ce150d470296f195bef8c04, 1895475) built_block_info=(0x669b147e3a0cb8243b24cf459fdf1314bf67b10a3f5375fcaa13546203c0c70b, 1167373) reference_block_execution_duration_ms=12
2025-06-06T20:53:24.086684Z  INFO Consensus: substrate: ⚙️  Syncing  2.6 bps, target=#2738148 (40 peers), best: #1901675 (0x7aa0…e16e), finalized #0 (0x295a…6c5e), ⬇ 18.1kiB/s ⬆ 9.0kiB/s
2025-06-06T20:53:26.160233Z  WARN Domain: domain_client_operator::bundle_processor: Slow domain block execution, took 2224ms consensus_block_info=(0x4da854cea803fee93408097713bb406251a63338db1b4ab47ae9ca53f088feeb, 1895478) built_block_info=(0xcd611819042edc456daea24d514d6397e3774e15dd783dae3a9aa2138565ce2c, 1167374) reference_block_execution_duration_ms=9
2025-06-06T20:53:27.455512Z  WARN Domain: domain_client_operator::bundle_processor: Slow domain block execution, took 1224ms consensus_block_info=(0x627992dd79884664569a8c94529720f9287ac05bf30ceed6f1323dae25a7b1ca, 1895480) built_block_info=(0xbcb0c7b3573b9f9950c6940ae3939120bbdaa1db7544b2e524d55351df5d4636, 1167375) reference_block_execution_duration_ms=6
4

This is nothing to worry about. It will update at the next finalized block.

Once best catches up to target, you will be fully synced.