Checking inherents failed: IncorrectSegmentHeadersList

Support
1

Issue Report

Environment

  • Operating System: Windows
  • Pulsar/Advanced CLI/Docker: Advanced CLI

Problem

Till yesterday my node worked just fine, but all of a sudden it doesn’t work at all. Here are the logs:

←[2m2023-10-20 16:30:21←[0m [Consensus] 💔 Error importing block 0x86d90982c6736beca6ee05cd4d531eb6e56417e926a7909fc98145431841858f: consensus error: Import failed: Checking inherents failed: IncorrectSegmentHeadersList { expected: [], actual: [V0 { segment_index: SegmentIndex(61), segment_commitment: Commitment(FsG1(blst_p1 { x: blst_fp { l: [1915009962429024792, 17419528363848502193, 14332235145882194266, 1003262627574642561, 449433953399824120, 1787444782105925510] }, y: blst_fp { l: [12446756501216069180, 12929733915699018679, 9363656477027728172, 9300234500525194046, 7560972507692318494, 785832040305047296] }, z: blst_fp { l: [8505329371266088957, 17002214543764226050, 6865905132761471162, 8632934651105793861, 6631298214892334189, 1582556514881692819] } })), prev_segment_header_hash: [144, 161, 134, 89, 157, 40, 243, 73, 242, 31, 144, 102, 248, 98, 22, 51, 164, 150, 9, 109, 112, 42, 14, 80, 103, 190, 85, 203, 100, 243, 179, 139], last_archived_block: LastArchivedBlock { number: 944834, archived_progress: Partial(3524) } }] }
←[2m2023-10-20 16:30:22←[0m [Consensus] ⚙️  Syncing  0.0 bps, target=#947400 (1 peers), best: #944934 (0x7250…3438), finalized #845572 (0x4aeb…fb62), ⬇ 755.3kiB/s ⬆ 1.9kiB/s
←[2m2023-10-20 16:30:23←[0m [Consensus] 💔 Error importing block 0x86d90982c6736beca6ee05cd4d531eb6e56417e926a7909fc98145431841858f: consensus error: Import failed: Checking inherents failed: IncorrectSegmentHeadersList { expected: [], actual: [V0 { segment_index: SegmentIndex(61), segment_commitment: Commitment(FsG1(blst_p1 { x: blst_fp { l: [1915009962429024792, 17419528363848502193, 14332235145882194266, 1003262627574642561, 449433953399824120, 1787444782105925510] }, y: blst_fp { l: [12446756501216069180, 12929733915699018679, 9363656477027728172, 9300234500525194046, 7560972507692318494, 785832040305047296] }, z: blst_fp { l: [8505329371266088957, 17002214543764226050, 6865905132761471162, 8632934651105793861, 6631298214892334189, 1582556514881692819] } })), prev_segment_header_hash: [144, 161, 134, 89, 157, 40, 243, 73, 242, 31, 144, 102, 248, 98, 22, 51, 164, 150, 9, 109, 112, 42, 14, 80, 103, 190, 85, 203, 100, 243, 179, 139], last_archived_block: LastArchivedBlock { number: 944834, archived_progress: Partial(3524) } }] }
←[2m2023-10-20 16:30:24←[0m [Consensus] 💔 Error importing block 0x86d90982c6736beca6ee05cd4d531eb6e56417e926a7909fc98145431841858f: consensus error: Import failed: Checking inherents failed: IncorrectSegmentHeadersList { expected: [], actual: [V0 { segment_index: SegmentIndex(61), segment_commitment: Commitment(FsG1(blst_p1 { x: blst_fp { l: [1915009962429024792, 17419528363848502193, 14332235145882194266, 1003262627574642561, 449433953399824120, 1787444782105925510] }, y: blst_fp { l: [12446756501216069180, 12929733915699018679, 9363656477027728172, 9300234500525194046, 7560972507692318494, 785832040305047296] }, z: blst_fp { l: [8505329371266088957, 17002214543764226050, 6865905132761471162, 8632934651105793861, 6631298214892334189, 1582556514881692819] } })), prev_segment_header_hash: [144, 161, 134, 89, 157, 40, 243, 73, 242, 31, 144, 102, 248, 98, 22, 51, 164, 150, 9, 109, 112, 42, 14, 80, 103, 190, 85, 203, 100, 243, 179, 139], last_archived_block: LastArchivedBlock { number: 944834, archived_progress: Partial(3524) } }] }
←[2m2023-10-20 16:30:25←[0m [Consensus] 💔 Error importing block 0x86d90982c6736beca6ee05cd4d531eb6e56417e926a7909fc98145431841858f: consensus error: Import failed: Checking inherents failed: IncorrectSegmentHeadersList { expected: [], actual: [V0 { segment_index: SegmentIndex(61), segment_commitment: Commitment(FsG1(blst_p1 { x: blst_fp { l: [1915009962429024792, 17419528363848502193, 14332235145882194266, 1003262627574642561, 449433953399824120, 1787444782105925510] }, y: blst_fp { l: [12446756501216069180, 12929733915699018679, 9363656477027728172, 9300234500525194046, 7560972507692318494, 785832040305047296] }, z: blst_fp { l: [8505329371266088957, 17002214543764226050, 6865905132761471162, 8632934651105793861, 6631298214892334189, 1582556514881692819] } })), prev_segment_header_hash: [144, 161, 134, 89, 157, 40, 243, 73, 242, 31, 144, 102, 248, 98, 22, 51, 164, 150, 9, 109, 112, 42, 14, 80, 103, 190, 85, 203, 100, 243, 179, 139], last_archived_block: LastArchivedBlock { number: 944834, archived_progress: Partial(3524) } }] }
←[2m2023-10-20 16:30:26←[0m [Consensus] 💔 Error importing block 0x86d90982c6736beca6ee05cd4d531eb6e56417e926a7909fc98145431841858f: consensus error: Import failed: Checking inherents failed: IncorrectSegmentHeadersList { expected: [], actual: [V0 { segment_index: SegmentIndex(61), segment_commitment: Commitment(FsG1(blst_p1 { x: blst_fp { l: [1915009962429024792, 17419528363848502193, 14332235145882194266, 1003262627574642561, 449433953399824120, 1787444782105925510] }, y: blst_fp { l: [12446756501216069180, 12929733915699018679, 9363656477027728172, 9300234500525194046, 7560972507692318494, 785832040305047296] }, z: blst_fp { l: [8505329371266088957, 17002214543764226050, 6865905132761471162, 8632934651105793861, 6631298214892334189, 1582556514881692819] } })), prev_segment_header_hash: [144, 161, 134, 89, 157, 40, 243, 73, 242, 31, 144, 102, 248, 98, 22, 51, 164, 150, 9, 109, 112, 42, 14, 80, 103, 190, 85, 203, 100, 243, 179, 139], last_archived_block: LastArchivedBlock { number: 944834, archived_progress: Partial(3524) } }] }
←[2m2023-10-20 16:30:27←[0m [Consensus] 💔 Error importing block 0x86d90982c6736beca6ee05cd4d531eb6e56417e926a7909fc98145431841858f: consensus error: Import failed: Checking inherents failed: IncorrectSegmentHeadersList { expected: [], actual: [V0 { segment_index: SegmentIndex(61), segment_commitment: Commitment(FsG1(blst_p1 { x: blst_fp { l: [1915009962429024792, 17419528363848502193, 14332235145882194266, 1003262627574642561, 449433953399824120, 1787444782105925510] }, y: blst_fp { l: [12446756501216069180, 12929733915699018679, 9363656477027728172, 9300234500525194046, 7560972507692318494, 785832040305047296] }, z: blst_fp { l: [8505329371266088957, 17002214543764226050, 6865905132761471162, 8632934651105793861, 6631298214892334189, 1582556514881692819] } })), prev_segment_header_hash: [144, 161, 134, 89, 157, 40, 243, 73, 242, 31, 144, 102, 248, 98, 22, 51, 164, 150, 9, 109, 112, 42, 14, 80, 103, 190, 85, 203, 100, 243, 179, 139], last_archived_block: LastArchivedBlock { number: 944834, archived_progress: Partial(3524) } }] }
←[2m2023-10-20 16:30:27←[0m [Consensus] ⚙️  Syncing  0.0 bps, target=#947401 (3 peers), best: #944934 (0x7250…3438), finalized #845572 (0x4aeb…fb62), ⬇ 968.7kiB/s ⬆ 2.2kiB/s
←[2m2023-10-20 16:30:32←[0m [Consensus] 💔 Error importing block 0x86d90982c6736beca6ee05cd4d531eb6e56417e926a7909fc98145431841858f: consensus error: Import failed: Checking inherents failed: IncorrectSegmentHeadersList { expected: [], actual: [V0 { segment_index: SegmentIndex(61), segment_commitment: Commitment(FsG1(blst_p1 { x: blst_fp { l: [1915009962429024792, 17419528363848502193, 14332235145882194266, 1003262627574642561, 449433953399824120, 1787444782105925510] }, y: blst_fp { l: [12446756501216069180, 12929733915699018679, 9363656477027728172, 9300234500525194046, 7560972507692318494, 785832040305047296] }, z: blst_fp { l: [8505329371266088957, 17002214543764226050, 6865905132761471162, 8632934651105793861, 6631298214892334189, 1582556514881692819] } })), prev_segment_header_hash: [144, 161, 134, 89, 157, 40, 243, 73, 242, 31, 144, 102, 248, 98, 22, 51, 164, 150, 9, 109, 112, 42, 14, 80, 103, 190, 85, 203, 100, 243, 179, 139], last_archived_block: LastArchivedBlock { number: 944834, archived_progress: Partial(3524) } }] }
←[2m2023-10-20 16:30:32←[0m [Consensus] 💔 Error importing block 0x1d66c72b548b573e753f5f9f0a5803280b924a400c41efae9bf717f7716389a0: block has an unknown parent
←[2m2023-10-20 16:30:32←[0m [Consensus] 💤 Idle (0 peers), best: #944934 (0x7250…3438), finalized #845572 (0x4aeb…fb62), ⬇ 374.4kiB/s ⬆ 1.8kiB/s
←[2m2023-10-20 16:30:34←[0m [Consensus] 💔 Error importing block 0x86d90982c6736beca6ee05cd4d531eb6e56417e926a7909fc98145431841858f: consensus error: Import failed: Checking inherents failed: IncorrectSegmentHeadersList { expected: [], actual: [V0 { segment_index: SegmentIndex(61), segment_commitment: Commitment(FsG1(blst_p1 { x: blst_fp { l: [1915009962429024792, 17419528363848502193, 14332235145882194266, 1003262627574642561, 449433953399824120, 1787444782105925510] }, y: blst_fp { l: [12446756501216069180, 12929733915699018679, 9363656477027728172, 9300234500525194046, 7560972507692318494, 785832040305047296] }, z: blst_fp { l: [8505329371266088957, 17002214543764226050, 6865905132761471162, 8632934651105793861, 6631298214892334189, 1582556514881692819] } })), prev_segment_header_hash: [144, 161, 134, 89, 157, 40, 243, 73, 242, 31, 144, 102, 248, 98, 22, 51, 164, 150, 9, 109, 112, 42, 14, 80, 103, 190, 85, 203, 100, 243, 179, 139], last_archived_block: LastArchivedBlock { number: 944834, archived_progress: Partial(3524) } }] }
←[2m2023-10-20 16:30:35←[0m [Consensus] 💔 Error importing block 0x86d90982c6736beca6ee05cd4d531eb6e56417e926a7909fc98145431841858f: consensus error: Import failed: Checking inherents failed: IncorrectSegmentHeadersList { expected: [], actual: [V0 { segment_index: SegmentIndex(61), segment_commitment: Commitment(FsG1(blst_p1 { x: blst_fp { l: [1915009962429024792, 17419528363848502193, 14332235145882194266, 1003262627574642561, 449433953399824120, 1787444782105925510] }, y: blst_fp { l: [12446756501216069180, 12929733915699018679, 9363656477027728172, 9300234500525194046, 7560972507692318494, 785832040305047296] }, z: blst_fp { l: [8505329371266088957, 17002214543764226050, 6865905132761471162, 8632934651105793861, 6631298214892334189, 1582556514881692819] } })), prev_segment_header_hash: [144, 161, 134, 89, 157, 40, 243, 73, 242, 31, 144, 102, 248, 98, 22, 51, 164, 150, 9, 109, 112, 42, 14, 80, 103, 190, 85, 203, 100, 243, 179, 139], last_archived_block: LastArchivedBlock { number: 944834, archived_progress: Partial(3524) } }] }
←[2m2023-10-20 16:30:37←[0m [Consensus] 💤 Idle (0 peers), best: #944934 (0x7250…3438), finalized #845572 (0x4aeb…fb62), ⬇ 380.9kiB/s ⬆ 1.2kiB/s
←[2m2023-10-20 16:30:41←[0m [Consensus] 💔 Error importing block 0x86d90982c6736beca6ee05cd4d531eb6e56417e926a7909fc98145431841858f: consensus error: Import failed: Checking inherents failed: IncorrectSegmentHeadersList { expected: [], actual: [V0 { segment_index: SegmentIndex(61), segment_commitment: Commitment(FsG1(blst_p1 { x: blst_fp { l: [1915009962429024792, 17419528363848502193, 14332235145882194266, 1003262627574642561, 449433953399824120, 1787444782105925510] }, y: blst_fp { l: [12446756501216069180, 12929733915699018679, 9363656477027728172, 9300234500525194046, 7560972507692318494, 785832040305047296] }, z: blst_fp { l: [8505329371266088957, 17002214543764226050, 6865905132761471162, 8632934651105793861, 6631298214892334189, 1582556514881692819] } })), prev_segment_header_hash: [144, 161, 134, 89, 157, 40, 243, 73, 242, 31, 144, 102, 248, 98, 22, 51, 164, 150, 9, 109, 112, 42, 14, 80, 103, 190, 85, 203, 100, 243, 179, 139], last_archived_block: LastArchivedBlock { number: 944834, archived_progress: Partial(3524) } }] }
←[2m2023-10-20 16:30:41←[0m [Consensus] 💔 Error importing block 0x20ee5ee0c43f4e23a7529a85b857d1203f1851cef0889eb69aa6278f42f80d20: block has an unknown parent
←[2m2023-10-20 16:30:42←[0m [Consensus] ⚙️  Syncing  0.0 bps, target=#947402 (1 peers), best: #944934 (0x7250…3438), finalized #845572 (0x4aeb…fb62), ⬇ 591.3kiB/s ⬆ 1.7kiB/s
←[2m2023-10-20 16:30:46←[0m [Consensus] 💔 Error importing block 0x86d90982c6736beca6ee05cd4d531eb6e56417e926a7909fc98145431841858f: consensus error: Import failed: Checking inherents failed: IncorrectSegmentHeadersList { expected: [], actual: [V0 { segment_index: SegmentIndex(61), segment_commitment: Commitment(FsG1(blst_p1 { x: blst_fp { l: [1915009962429024792, 17419528363848502193, 14332235145882194266, 1003262627574642561, 449433953399824120, 1787444782105925510] }, y: blst_fp { l: [12446756501216069180, 12929733915699018679, 9363656477027728172, 9300234500525194046, 7560972507692318494, 785832040305047296] }, z: blst_fp { l: [8505329371266088957, 17002214543764226050, 6865905132761471162, 8632934651105793861, 6631298214892334189, 1582556514881692819] } })), prev_segment_header_hash: [144, 161, 134, 89, 157, 40, 243, 73, 242, 31, 144, 102, 248, 98, 22, 51, 164, 150, 9, 109, 112, 42, 14, 80, 103, 190, 85, 203, 100, 243, 179, 139], last_archived_block: LastArchivedBlock { number: 944834, archived_progress: Partial(3524) } }] }
←[2m2023-10-20 16:30:46←[0m [Consensus] 💔 Error importing block 0x6eef8630484e588e3e8927bf079130cb6837e80ab616bab6889c20b5e7750aca: block has an unknown parent
←[2m2023-10-20 16:30:47←[0m [Consensus] ⚙️  Syncing  0.0 bps, target=#947404 (2 peers), best: #944934 (0x7250…3438), finalized #845572 (0x4aeb…fb62), ⬇ 474.8kiB/s ⬆ 2.0kiB/s
←[2m2023-10-20 16:30:52←[0m [Consensus] ⚙️  Syncing  0.0 bps, target=#947404 (2 peers), best: #944934 (0x7250…3438), finalized #845572 (0x4aeb…fb62), ⬇ 78.6kiB/s ⬆ 1.2kiB/s
←[2m2023-10-20 16:30:55←[0m [Consensus] 💔 Error importing block 0x86d90982c6736beca6ee05cd4d531eb6e56417e926a7909fc98145431841858f: consensus error: Import failed: Checking inherents failed: IncorrectSegmentHeadersList { expected: [], actual: [V0 { segment_index: SegmentIndex(61), segment_commitment: Commitment(FsG1(blst_p1 { x: blst_fp { l: [1915009962429024792, 17419528363848502193, 14332235145882194266, 1003262627574642561, 449433953399824120, 1787444782105925510] }, y: blst_fp { l: [12446756501216069180, 12929733915699018679, 9363656477027728172, 9300234500525194046, 7560972507692318494, 785832040305047296] }, z: blst_fp { l: [8505329371266088957, 17002214543764226050, 6865905132761471162, 8632934651105793861, 6631298214892334189, 1582556514881692819] } })), prev_segment_header_hash: [144, 161, 134, 89, 157, 40, 243, 73, 242, 31, 144, 102, 248, 98, 22, 51, 164, 150, 9, 109, 112, 42, 14, 80, 103, 190, 85, 203, 100, 243, 179, 139], last_archived_block: LastArchivedBlock { number: 944834, archived_progress: Partial(3524) } }] }
←[2m2023-10-20 16:30:55←[0m [Consensus] 💔 Error importing block 0xbdfdbad9374d83d0ce2b207f3d26c32fc9a8ec49849da94f0434c4e115f4152b: block has an unknown parent
←[2m2023-10-20 16:30:57←[0m [Consensus] 💤 Idle (0 peers), best: #944934 (0x7250…3438), finalized #845572 (0x4aeb…fb62), ⬇ 1.5MiB/s ⬆ 3.2kiB/s
←[2m2023-10-20 16:30:59←[0m [Consensus] 💔 Error importing block 0x86d90982c6736beca6ee05cd4d531eb6e56417e926a7909fc98145431841858f: consensus error: Import failed: Checking inherents failed: IncorrectSegmentHeadersList { expected: [], actual: [V0 { segment_index: SegmentIndex(61), segment_commitment: Commitment(FsG1(blst_p1 { x: blst_fp { l: [1915009962429024792, 17419528363848502193, 14332235145882194266, 1003262627574642561, 449433953399824120, 1787444782105925510] }, y: blst_fp { l: [12446756501216069180, 12929733915699018679, 9363656477027728172, 9300234500525194046, 7560972507692318494, 785832040305047296] }, z: blst_fp { l: [8505329371266088957, 17002214543764226050, 6865905132761471162, 8632934651105793861, 6631298214892334189, 1582556514881692819] } })), prev_segment_header_hash: [144, 161, 134, 89, 157, 40, 243, 73, 242, 31, 144, 102, 248, 98, 22, 51, 164, 150, 9, 109, 112, 42, 14, 80, 103, 190, 85, 203, 100, 243, 179, 139], last_archived_block: LastArchivedBlock { number: 944834, archived_progress: Partial(3524) } }] }
←[2m2023-10-20 16:31:02←[0m [Consensus] ⚙️  Syncing  0.0 bps, target=#947406 (1 peers), best: #944934 (0x7250…3438), finalized #845572 (0x4aeb…fb62), ⬇ 398.2kiB/s ⬆ 2.3kiB/s
←[2m2023-10-20 16:31:02←[0m [Consensus] 💔 Error importing block 0x86d90982c6736beca6ee05cd4d531eb6e56417e926a7909fc98145431841858f: consensus error: Import failed: Checking inherents failed: IncorrectSegmentHeadersList { expected: [], actual: [V0 { segment_index: SegmentIndex(61), segment_commitment: Commitment(FsG1(blst_p1 { x: blst_fp { l: [1915009962429024792, 17419528363848502193, 14332235145882194266, 1003262627574642561, 449433953399824120, 1787444782105925510] }, y: blst_fp { l: [12446756501216069180, 12929733915699018679, 9363656477027728172, 9300234500525194046, 7560972507692318494, 785832040305047296] }, z: blst_fp { l: [8505329371266088957, 17002214543764226050, 6865905132761471162, 8632934651105793861, 6631298214892334189, 1582556514881692819] } })), prev_segment_header_hash: [144, 161, 134, 89, 157, 40, 243, 73, 242, 31, 144, 102, 248, 98, 22, 51, 164, 150, 9, 109, 112, 42, 14, 80, 103, 190, 85, 203, 100, 243, 179, 139], last_archived_block: LastArchivedBlock { number: 944834, archived_progress: Partial(3524) } }] }
←[2m2023-10-20 16:31:07←[0m [Consensus] 💤 Idle (0 peers), best: #944934 (0x7250…3438), finalized #845572 (0x4aeb…fb62), ⬇ 111.1kiB/s ⬆ 2.1kiB/s
←[2m2023-10-20 16:31:12←[0m [Consensus] ⚙️  Syncing  0.0 bps, target=#947407 (1 peers), best: #944934 (0x7250…3438), finalized #845572 (0x4aeb…fb62), ⬇ 124.9kiB/s ⬆ 1.9kiB/s
2

Note this is the second report of this issue in 24 hours.

3

This is very odd, this issue was supposed to be long fixed and we have not seen reports of it for a long time now (Import failed: Checking inherents failed: MissingSegmentHeadersList · Issue #871 · subspace/subspace · GitHub). Try restarting your node and see if it helps. It would also be helpful to see logs right before this happened.