Segmentation Fault Error - CLI Linux

The user reported after stopping node and farmer restarting them (CLI), they would get the segmentation error.

Additional info shared below.
The server has 16 Gb RAM



After that I restart software by

systemctl restart subspace-node.service subspace-farmer.service

So far everything looks good, let’s wait for a full synchronization and I’ll send the result


OS - Ubuntu 20.04, all the characteristics are exactly the sam as those of other servers that work correctly.

[501279.671842] tokio-runtime-w[653759]: segfault at 0 ip 0000555727c851af sp 00007f7e59c3ea58 error 4 in subspace-farmer[5557271ef000+b07000]
[501279.671856] Code: 6e ef 66 48 0f 6e f3 48 83 ec 40 44 8b 07 44 8b 4f 04 44 8b 57 08 44 8b 5f 0c 44 8b 67 10 44 8b 6f 14 44 8b 77 18 44 8b 7f 1c <8b> 1e 0f cb 89 1c 24 44 89 e0 c1 c8 0e 44 31 e0 c1 c8 05 44 31 e0
[501388.140144] tokio-runtime-w[653907]: segfault at 0 ip 0000555e555991af sp 00007f64f536dbf8 error 4 in subspace-farmer[555e54b03000+b07000]
[501388.140176] Code: 6e ef 66 48 0f 6e f3 48 83 ec 40 44 8b 07 44 8b 4f 04 44 8b 57 08 44 8b 5f 0c 44 8b 67 10 44 8b 6f 14 44 8b 77 18 44 8b 7f 1c <8b> 1e 0f cb 89 1c 24 44 89 e0 c1 c8 0e 44 31 e0 c1 c8 05 44 31 e0
[502085.055799] sh (654242): drop_caches: 3
[505684.285657] sh (655550): drop_caches: 3
[509283.864710] sh (656559): drop_caches: 3
[512883.410954] sh (657638): drop_caches: 3
[516483.845197] sh (658999): drop_caches: 3
[520083.080467] sh (660358): drop_caches: 3
[523683.754795] sh (660889): drop_caches: 3
[525516.481237] tokio-runtime-w[661424]: segfault at 0 ip 0000560cff9701af sp 00007f81573f4b98 error 4 in subspace-farmer[560cfeeda000+b07000]
[525516.481373] Code: 6e ef 66 48 0f 6e f3 48 83 ec 40 44 8b 07 44 8b 4f 04 44 8b 57 08 44 8b 5f 0c 44 8b 67 10 44 8b 6f 14 44 8b 77 18 44 8b 7f 1c <8b> 1e 0f cb 89 1c 24 44 89 e0 c1 c8 0e 44 31 e0 c1 c8 05 44 31 e0
[525550.129089] tokio-runtime-w[661756]: segfault at a28528df ip 000055d48a4ee1af sp 00007f08377f7638 error 4 in subspace-farmer[55d489a58000+b07000]
[525550.129105] Code: 6e ef 66 48 0f 6e f3 48 83 ec 40 44 8b 07 44 8b 4f 04 44 8b 57 08 44 8b 5f 0c 44 8b 67 10 44 8b 6f 14 44 8b 77 18 44 8b 7f 1c <8b> 1e 0f cb 89 1c 24 44 89 e0 c1 c8 0e 44 31 e0 c1 c8 05 44 31 e0

Rebooting the server did not help, error is

2022-06-14T18:27:59.427373Z  INFO subspace_farmer::commands::farm: WS RPC server listening on 127.0.0.1:9955
2022-06-14T18:28:00.002464Z  INFO subspace_farmer::farming: Salt updated, recommitting in background new_salt=79f25142c329a0d0
2022-06-14T18:28:00.002694Z  INFO subspace_farmer::farming: Salt updated, recommitting in background new_salt=79f25142c329a0d0
2022-06-14T18:28:11.732664Z ERROR subspace_farmer::plotting: Failed to write encoded pieces error=Corruption: Compaction sees out-of-order keys.
2022-06-14T18:28:12.182849Z  INFO subspace_farmer::archiving: Plotted segment segment_index=108478
subspace-farmer.service: Main process exited, code=killed, status=11/SEGV
subspace-farmer.service: Failed with result 'signal'.
subspace-farmer.service: Scheduled restart job, restart counter is at 19.
Stopped Subspace Farmer.
Started Subspace Farmer.

2022-06-14T18:29:03.340039Z ERROR subspace_farmer::plotting: Failed to write encoded pieces error=Corruption: Compaction sees out-of-order keys.

Duplicate of this post Farmer: Main process exited, code=killed, status=11/SEGV

Currently being escalated to a developer