Issue Report
Environment
- Operating System: Linux distribution: Ubuntu 20.04.5 LTS
- CPU Architecture: x86_64 CPU: Intel(R) Xeon(R) CPU E5-2695 v4 @ 2.10GHz CPU cores: 30
- RAM: 117919MB
- Storage: 3 TB
- Plot Size: 800 GB
- Subspace Deployment Method: Nodes Guru automated deploy script (Subspace CLI under systemd service with --verbose)
Problem
Error messages appearing in the logs related to database issues and then failing to get header for hash.
Note the plot size of 800G.
Note that the user was following a third party guide.
Error messages encountered:
sc_service::client::client: Block import error: Databaseslots: Unable to author block in slot. No best block header: Chain lookup failed: Failed to get header for hashsync: đź’” Error importing block 0xcc3f4910b76799b870e372f8293c05ce066eb41b001d448dca078ee1bcda1f0e: consensus error: Import failed: Database- `DB saving error
ERROR subspace_networking::behavior::provider_storage::providers: DB saving error. key=[248, 185, 206, 5, 32, 204, 156, 36, 161, 42, 60, 149, 134, 148, 89, 250, 20, 176, 49, 134, 2, 7, 172, 173, 202, 153, 191, 87, 116, 222, 11, 69, 111, 51, 182, 9, 72] err=Background(Io(Os { code: 22, kind: InvalidInput, message: "Invalid argument" }))
Steps to reproduce
- Start farming using Nodes Guru guide.
- (Possibly) start an 800G plot.
Expected result
- Syncing completes and plotting starts.
What happens instead
- Errors in the logs as per below.
2023-03-18T13:49:01.249829Z INFO substrate: ⚙️ Syncing 0.0 bps, target=#494075 (79 peers), best: #62787 (0xe687…b32a), finalized #62686 (0xbe11…7e44), ⬇ 1.2MiB/s ⬆ 6.6kiB/s
2023-03-18T13:49:02.001219Z WARN slots: Unable to author block in slot. No best block header: Chain lookup failed: Failed to get header for hash 0xcc3f…1f0e
2023-03-18T13:49:02.903390Z WARN sc_service::client::client: Block import error: Database
2023-03-18T13:49:03.001210Z WARN slots: Unable to author block in slot. No best block header: Chain lookup failed: Failed to get header for hash 0xcc3f…1f0e
2023-03-18T13:49:03.050278Z WARN sync: đź’” Error importing block 0xcc3f4910b76799b870e372f8293c05ce066eb41b001d448dca078ee1bcda1f0e: consensus error: Import failed: Database
2023-03-18T13:49:04.000919Z WARN slots: Unable to author block in slot. No best block header: Chain lookup failed: Failed to get header for hash 0xcc3f…1f0e
2023-03-18T13:49:05.000813Z WARN slots: Unable to author block in slot. No best block header: Chain lookup failed: Failed to get header for hash 0xcc3f…1f0e
2023-03-18T13:49:06.001138Z WARN slots: Unable to author block in slot. No best block header: Chain lookup failed: Failed to get header for hash 0xcc3f…1f0e
2023-03-18T13:49:06.250781Z INFO substrate: ⚙️ Syncing 0.0 bps, target=#494075 (76 peers), best: #62787 (0xe687…b32a), finalized #62686 (0xbe11…7e44), ⬇ 4.8MiB/s ⬆ 8.6kiB/s
2023-03-18T13:49:07.000669Z WARN slots: Unable to author block in slot. No best block header: Chain lookup failed: Failed to get header for hash 0xcc3f…1f0e
2023-03-18T13:49:08.001360Z WARN slots: Unable to author block in slot. No best block header: Chain lookup failed: Failed to get header for hash 0xcc3f…1f0e
2023-03-18T13:49:09.000702Z WARN slots: Unable to author block in slot. No best block header: Chain lookup failed: Failed to get header for hash 0xcc3f…1f0e
2023-03-18T13:49:10.001026Z WARN slots: Unable to author block in slot. No best block header: Chain lookup failed: Failed to get header for hash 0xcc3f…1f0e
2023-03-18T13:49:11.000547Z WARN slots: Unable to author block in slot. No best block header: Chain lookup failed: Failed to get header for hash 0xcc3f…1f0e
2023-03-18T13:49:11.251426Z INFO substrate: ⚙️ Syncing 0.0 bps, target=#494075 (76 peers), best: #62787 (0xe687…b32a), finalized #62686 (0xbe11…7e44), ⬇ 7.9MiB/s ⬆ 7.1kiB/s
2023-03-18T13:49:12.001253Z WARN slots: Unable to author block in slot. No best block header: Chain lookup failed: Failed to get header for hash 0xcc3f…1f0e
2023-03-18T13:49:13.001083Z WARN slots: Unable to author block in slot. No best block header: Chain lookup failed: Failed to get header for hash 0xcc3f…1f0e
2023-03-18T13:49:14.000802Z WARN slots: Unable to author block in slot. No best block header: Chain lookup failed: Failed to get header for hash 0xcc3f…1f0e
2023-03-18T13:49:15.001274Z WARN slots: Unable to author block in slot. No best block header: Chain lookup failed: Failed to get header for hash 0xcc3f…1f0e
2023-03-18T13:49:16.000653Z WARN slots: Unable to author block in slot. No best block header: Chain lookup failed: Failed to get header for hash 0xcc3f…1f0e
2023-03-18T13:49:16.252277Z INFO substrate: ⚙️ Syncing 0.0 bps, target=#494076 (76 peers), best: #62787 (0xe687…b32a), finalized #62686 (0xbe11…7e44), ⬇ 7.5MiB/s ⬆ 7.8kiB/s
2023-03-18T13:49:17.001103Z WARN slots: Unable to author block in slot. No best block header: Chain lookup failed: Failed to get header for hash 0xcc3f…1f0e
2023-03-18T13:49:18.000639Z WARN slots: Unable to author block in slot. No best block header: Chain lookup failed: Failed to get header for hash 0xcc3f…1f0e
2023-03-18T13:49:19.001033Z WARN slots: Unable to author block in slot. No best block header: Chain lookup failed: Failed to get header for hash 0xcc3f…1f0e
2023-03-18T13:49:20.000407Z WARN slots: Unable to author block in slot. No best block header: Chain lookup failed: Failed to get header for hash 0xcc3f…1f0e
2023-03-18T13:49:21.001072Z WARN slots: Unable to author block in slot. No best block header: Chain lookup failed: Failed to get header for hash 0xcc3f…1f0e
2023-03-18T13:49:21.252763Z INFO substrate: ⚙️ Syncing 0.0 bps, target=#494077 (76 peers), best: #62787 (0xe687…b32a), finalized #62686 (0xbe11…7e44), ⬇ 6.9MiB/s ⬆ 7.3kiB/s
2023-03-18T13:49:22.000513Z WARN slots: Unable to author block in slot. No best block header: Chain lookup failed: Failed to get header for hash 0xcc3f…1f0e
2023-03-18T13:49:23.000997Z WARN slots: Unable to author block in slot. No best block header: Chain lookup failed: Failed to get header for hash 0xcc3f…1f0e
2023-03-18T13:49:24.000431Z WARN slots: Unable to author block in slot. No best block header: Chain lookup failed: Failed to get header for hash 0xcc3f…1f0e
2023-03-18T13:49:25.001249Z WARN slots: Unable to author block in slot. No best block header: Chain lookup failed: Failed to get header for hash 0xcc3f…1f0e
2023-03-18T13:49:26.000803Z WARN slots: Unable to author block in slot. No best block header: Chain lookup failed: Failed to get header for hash 0xcc3f…1f0e
2023-03-18T13:49:26.253796Z INFO substrate: ⚙️ Syncing 0.0 bps, target=#494080 (76 peers), best: #62787 (0xe687…b32a), finalized #62686 (0xbe11…7e44), ⬇ 8.0MiB/s ⬆ 7.9kiB/s
2023-03-18T13:49:27.001705Z WARN slots: Unable to author block in slot. No best block header: Chain lookup failed: Failed to get header for hash 0xcc3f…1f0e
2023-03-18T13:49:28.001259Z WARN slots: Unable to author block in slot. No best block header: Chain lookup failed: Failed to get header for hash 0xcc3f…1f0e
2023-03-18T13:49:29.000720Z WARN slots: Unable to author block in slot. No best block header: Chain lookup failed: Failed to get header for hash 0xcc3f…1f0e
2023-03-18T13:49:30.001104Z WARN slots: Unable to author block in slot. No best block header: Chain lookup failed: Failed to get header for hash 0xcc3f…1f0e
2023-03-18T13:49:31.000757Z WARN slots: Unable to author block in slot. No best block header: Chain lookup failed: Failed to get header for hash 0xcc3f…1f0e
2023-03-18T13:49:31.255037Z INFO substrate: ⚙️ Syncing 0.0 bps, target=#494082 (76 peers), best: #62787 (0xe687…b32a), finalized #62686 (0xbe11…7e44), ⬇ 7.7MiB/s ⬆ 7.8kiB/s
2023-03-18T13:49:32.001467Z WARN slots: Unable to author block in slot. No best block header: Chain lookup failed: Failed to get header for hash 0xcc3f…1f0e
2023-03-18T13:49:33.000865Z WARN slots: Unable to author block in slot. No best block header: Chain lookup failed: Failed to get header for hash 0xcc3f…1f0e
2023-03-18T13:49:34.001797Z WARN slots: Unable to author block in slot. No best block header: Chain lookup failed: Failed to get header for hash 0xcc3f…1f0e
2023-03-18T13:49:35.001467Z WARN slots: Unable to author block in slot. No best block header: Chain lookup failed: Failed to get header for hash 0xcc3f…1f0e
2023-03-18T13:49:36.000971Z WARN slots: Unable to author block in slot. No best block header: Chain lookup failed: Failed to get header for hash 0xcc3f…1f0e
2023-03-18T13:49:36.255748Z INFO substrate: ⚙️ Syncing 0.0 bps, target=#494083 (76 peers), best: #62787 (0xe687…b32a), finalized #62686 (0xbe11…7e44), ⬇ 7.6MiB/s ⬆ 7.7kiB/s
2023-03-18T13:49:37.000447Z WARN slots: Unable to author block in slot. No best block header: Chain lookup failed: Failed to get header for hash 0xcc3f…1f0e
2023-03-18T13:49:38.000840Z WARN slots: Unable to author block in slot. No best block header: Chain lookup failed: Failed to get header for hash 0xcc3f…1f0e
2023-03-18T13:49:39.001239Z WARN slots: Unable to author block in slot. No best block header: Chain lookup failed: Failed to get header for hash 0xcc3f…1f0e
2023-03-18T13:49:40.000899Z WARN slots: Unable to author block in slot. No best block header: Chain lookup failed: Failed to get header for hash 0xcc3f…1f0e
2023-03-18T13:49:41.001240Z WARN slots: Unable to author block in slot. No best block header: Chain lookup failed: Failed to get header for hash 0xcc3f…1f0e
2023-03-18T13:49:41.256625Z INFO substrate: ⚙️ Syncing 0.0 bps, target=#494084 (76 peers), best: #62787 (0xe687…b32a), finalized #62686 (0xbe11…7e44), ⬇ 6.9MiB/s ⬆ 8.0kiB/s
2023-03-18T13:49:42.000624Z WARN slots: Unable to author block in slot. No best block header: Chain lookup failed: Failed to get header for hash 0xcc3f…1f0e
2023-03-18T13:49:43.000989Z WARN slots: Unable to author block in slot. No best block header: Chain lookup failed: Failed to get header for hash 0xcc3f…1f0e
2023-03-18T13:49:44.000333Z WARN slots: Unable to author block in slot. No best block header: Chain lookup failed: Failed to get header for hash 0xcc3f…1f0e
2023-03-18T13:49:45.000689Z WARN slots: Unable to author block in slot. No best block header: Chain lookup failed: Failed to get header for hash 0xcc3f…1f0e
2023-03-18T13:49:46.001283Z WARN slots: Unable to author block in slot. No best block header: Chain lookup failed: Failed to get header for hash 0xcc3f…1f0e
2023-03-18T13:49:46.257318Z INFO substrate: ⚙️ Syncing 0.0 bps, target=#494084 (76 peers), best: #62787 (0xe687…b32a), finalized #62686 (0xbe11…7e44), ⬇ 7.8MiB/s ⬆ 7.4kiB/s
2023-03-18T13:49:47.001226Z WARN slots: Unable to author block in slot. No best block header: Chain lookup failed: Failed to get header for hash 0xcc3f…1f0e
2023-03-18T13:49:48.000598Z WARN slots: Unable to author block in slot. No best block header: Chain lookup failed: Failed to get header for hash 0xcc3f…1f0e
2023-03-18T13:49:49.001287Z WARN slots: Unable to author block in slot. No best block header: Chain lookup failed: Failed to get header for hash 0xcc3f…1f0e
2023-03-18T13:49:50.000813Z WARN slots: Unable to author block in slot. No best block header: Chain lookup failed: Failed to get header for hash 0xcc3f…1f0e
2023-03-18T13:49:51.001459Z WARN slots: Unable to author block in slot. No best block header: Chain lookup failed: Failed to get header for hash 0xcc3f…1f0e
2023-03-18T13:49:51.257880Z INFO substrate: ⚙️ Syncing 0.0 bps, target=#494085 (76 peers), best: #62787 (0xe687…b32a), finalized #62686 (0xbe11…7e44), ⬇ 7.8MiB/s ⬆ 7.8kiB/s
2023-03-18T13:49:52.000861Z WARN slots: Unable to author block in slot. No best block header: Chain lookup failed: Failed to get header for hash 0xcc3f…1f0e
2023-03-18T13:49:53.001707Z WARN slots: Unable to author block in slot. No best block header: Chain lookup failed: Failed to get header for hash 0xcc3f…1f0e
2023-03-18T13:49:54.001975Z WARN slots: Unable to author block in slot. No best block header: Chain lookup failed: Failed to get header for hash 0xcc3f…1f0e
2023-03-18T13:49:55.000391Z WARN slots: Unable to author block in slot. No best block header: Chain lookup failed: Failed to get header for hash 0xcc3f…1f0e
2023-03-18T13:49:56.000953Z WARN slots: Unable to author block in slot. No best block header: Chain lookup failed: Failed to get header for hash 0xcc3f…1f0e
2023-03-18T13:49:56.258606Z INFO substrate: ⚙️ Syncing 0.0 bps, target=#494085 (76 peers), best: #62787 (0xe687…b32a), finalized #62686 (0xbe11…7e44), ⬇ 7.8MiB/s ⬆ 7.9kiB/s
2023-03-18T13:49:57.000482Z WARN slots: Unable to author block in slot. No best block header: Chain lookup failed: Failed to get header for hash 0xcc3f…1f0e
2023-03-18T13:49:58.001174Z WARN slots: Unable to author block in slot. No best block header: Chain lookup failed: Failed to get header for hash 0xcc3f…1f0e
2023-03-18T13:49:59.000680Z WARN slots: Unable to author block in slot. No best block header: Chain lookup failed: Failed to get header for hash 0xcc3f…1f0e
2023-03-18T13:50:00.001310Z WARN slots: Unable to author block in slot. No best block header: Chain lookup failed: Failed to get header for hash 0xcc3f…1f0e
2023-03-18T13:50:01.000895Z WARN slots: Unable to author block in slot. No best block header: Chain lookup failed: Failed to get header for hash 0xcc3f…1f0e
2023-03-18T13:50:01.259157Z INFO substrate: ⚙️ Syncing 0.0 bps, target=#494085 (76 peers), best: #62787 (0xe687…b32a), finalized #62686 (0xbe11…7e44), ⬇ 7.5MiB/s ⬆ 7.9kiB/s
2023-03-18T13:50:02.001367Z WARN slots: Unable to author block in slot. No best block header: Chain lookup failed: Failed to get header for hash 0xcc3f…1f0e
2023-03-18T13:50:03.000677Z WARN slots: Unable to author block in slot. No best block header: Chain lookup failed: Failed to get header for hash 0xcc3f…1f0e
2023-03-18T13:50:04.001281Z WARN slots: Unable to author block in slot. No best block header: Chain lookup failed: Failed to get header for hash 0xcc3f…1f0e
2023-03-18T13:50:05.000878Z WARN slots: Unable to author block in slot. No best block header: Chain lookup failed: Failed to get header for hash 0xcc3f…1f0e
2023-03-18T13:50:06.001151Z WARN slots: Unable to author block in slot. No best block header: Chain lookup failed: Failed to get header for hash 0xcc3f…1f0e
2023-03-18T13:50:06.260559Z INFO substrate: ⚙️ Syncing 0.0 bps, target=#494085 (76 peers), best: #62787 (0xe687…b32a), finalized #62686 (0xbe11…7e44), ⬇ 6.6MiB/s ⬆ 7.2kiB/s
2023-03-18T13:50:07.000666Z WARN slots: Unable to author block in slot. No best block header: Chain lookup failed: Failed to get header for hash 0xcc3f…1f0e
2023-03-18T13:50:08.001699Z WARN slots: Unable to author block in slot. No best block header: Chain lookup failed: Failed to get header for hash 0xcc3f…1f0e
2023-03-18T13:50:09.001961Z WARN slots: Unable to author block in slot. No best block header: Chain lookup failed: Failed to get header for hash 0xcc3f…1f0e
2023-03-18T13:50:10.000705Z WARN slots: Unable to author block in slot. No best block header: Chain lookup failed: Failed to get header for hash 0xcc3f…1f0e
2023-03-18T13:50:11.001572Z WARN slots: Unable to author block in slot. No best block header: Chain lookup failed: Failed to get header for hash 0xcc3f…1f0e
2023-03-18T13:50:11.273920Z INFO substrate: ⚙️ Syncing 0.0 bps, target=#494085 (76 peers), best: #62787 (0xe687…b32a), finalized #62686 (0xbe11…7e44), ⬇ 7.6MiB/s ⬆ 7.7kiB/s
2023-03-18T13:50:12.001142Z WARN slots: Unable to author block in slot. No best block header: Chain lookup failed: Failed to get header for hash 0xcc3f…1f0e
2023-03-18T13:50:13.000529Z WARN slots: Unable to author block in slot. No best block header: Chain lookup failed: Failed to get header for hash 0xcc3f…1f0e
2023-03-18T13:50:14.001032Z WARN slots: Unable to author block in slot. No best block header: Chain lookup failed: Failed to get header for hash 0xcc3f…1f0e
2023-03-18T13:50:15.001197Z WARN slots: Unable to author block in slot. No best block header: Chain lookup failed: Failed to get header for hash 0xcc3f…1f0e
2023-03-18T13:50:16.000644Z WARN slots: Unable to author block in slot. No best block header: Chain lookup failed: Failed to get header for hash 0xcc3f…1f0e
2023-03-18T13:50:16.275278Z INFO substrate: ⚙️ Syncing 0.0 bps, target=#494085 (76 peers), best: #62787 (0xe687…b32a), finalized #62686 (0xbe11…7e44), ⬇ 6.5MiB/s ⬆ 7.6kiB/s
2023-03-18T13:50:17.001069Z WARN slots: Unable to author block in slot. No best block header: Chain lookup failed: Failed to get header for hash 0xcc3f…1f0e
2023-03-18T13:50:18.000368Z WARN slots: Unable to author block in slot. No best block header: Chain lookup failed: Failed to get header for hash 0xcc3f…1f0e
2023-03-18T13:50:19.001014Z WARN slots: Unable to author block in slot. No best block header: Chain lookup failed: Failed to get header for hash 0xcc3f…1f0e
2023-03-18T13:50:20.000775Z WARN slots: Unable to author block in slot. No best block header: Chain lookup failed: Failed to get header for hash 0xcc3f…1f0e
2023-03-18T13:50:21.001302Z WARN slots: Unable to author block in slot. No best block header: Chain lookup failed: Failed to get header for hash 0xcc3f…1f0e
2023-03-18T13:50:21.276067Z INFO substrate: ⚙️ Syncing 0.0 bps, target=#494087 (78 peers), best: #62787 (0xe687…b32a), finalized #62686 (0xbe11…7e44), ⬇ 6.9MiB/s ⬆ 20.6kiB/s
2023-03-18T13:50:22.000777Z WARN slots: Unable to author block in slot. No best block header: Chain lookup failed: Failed to get header for hash 0xcc3f…1f0e
Another user of the same script got some different database errors:
