My node suddenly stucked at block #356030 with strange error for past few days

Support Incentivized Testnet
,
1

Strange error happens for past few days prevented my node to continue syncing normaly and being stucked at block #356030. Lastest logs below
From node:

←[2m2022-10-03 20:15:35←[0m [PrimaryChain] Block import error: Backend error: Can't canonicalize missing block number #352035 when importing 0x65748c46501f8cdb1e50f4246bb33295af23bac44d15286013a3e564fefeb0aa (#356131)
←[2m2022-10-03 20:15:35←[0m [PrimaryChain] 💔 Error importing block 0x65748c46501f8cdb1e50f4246bb33295af23bac44d15286013a3e564fefeb0aa: consensus error: Import failed: Backend error: Can't canonicalize missing block number #352035 when importing 0x65748c46501f8cdb1e50f4246bb33295af23bac44d15286013a3e564fefeb0aa (#356131)
←[2m2022-10-03 20:15:37←[0m [PrimaryChain] ⚙️  Syncing  0.0 bps, target=#386282 (43 peers), best: #356130 (0xa2ba…2abe), finalized #356030 (0x4f21…0685), ⬇ 19.2MiB/s ⬆ 9.5kiB/s

Farmer:

←[2m2022-10-03T13:08:07.070884Z←[0m ←[32m INFO←[0m ←[1msingle_plot_farm←[0m←[1m{←[0m←[3msingle_plot_farm_id←[0m←[2m=←[0m01GCXHJACT5FEW7DWW3ABB0KEH←[1m}←[0m←[2m:←[0m ←[2msubspace_farmer::farming←[0m←[2m:←[0m Subscribing to slot info notifications
←[2m2022-10-03T13:08:07.071135Z←[0m ←[32m INFO←[0m ←[1msingle_plot_farm←[0m←[1m{←[0m←[3msingle_plot_farm_id←[0m←[2m=←[0m01GCXHJACTZ8DJXGANMNVZVTT6←[1m}←[0m←[2m:←[0m ←[2msubspace_farmer::farming←[0m←[2m:←[0m Subscribing to slot info notifications
←[2m2022-10-03T13:08:07.071383Z←[0m ←[32m INFO←[0m ←[1msingle_plot_farm←[0m←[1m{←[0m←[3msingle_plot_farm_id←[0m←[2m=←[0m01GCXHJACTF377V7GGSZQ337SR←[1m}←[0m←[2m:←[0m ←[2msubspace_farmer::farming←[0m←[2m:←[0m Subscribing to slot info notifications

My node were syncing good before this, no idea how this happening. Restart the node or computer doesn’t help.

Device’s Specifications:
Processor: 11th Gen Intel(R) Core™ i7-11700
Ram: 32GB
OS: Windows 11 Pro Version 22H2 x64
Storage: 2 NVMe 512GB SSD
Storage allocated for node: 100GB
CLI: Powershell

2

May you give some information such what is your OS, which way did you use to install subspace node, total storage etc?

3

I suspect your machine ran out of free space and crashed, after which database was corrupted. How much free space you have there?

4

Free space left at that time is about 200GB so I don’t think it’s about spaces. I tried many ways to fix it last week but no hope so I did a reset of the node and started again from beginning and now it’s working good again

5

That is an interesting bug I have never seen before. Glad to know reset helped, we’ll keep monitor if similar issues happen in the future.