Thread 'tokio-runtime-worker' panicked at 'Not empty, checked above; qed' (gemini-3f-2023-sep-13)

Issue Report

Environment

  • Operating System: Ubuntu 22.04.3 LTS
  • Advanced CLI: gemini-3f-2023-sep-13 (Latest)

Problem

Thread ‘tokio-runtime-worker’ panicked at ‘Not empty, checked above; qed’

Sep 14 00:25:59 ironfish subspace-node[263358]: 2023-09-14 00:25:59 [Consensus] ⚙️  Syncing  0.0 bps, target=#413049 (20 peers), best: #411981 (0x29b1…42e4), finalized #232206 (0x9143…9e6f), ⬇ 50.5kiB/s ⬆ 34.8kiB/s
Sep 14 00:26:04 ironfish subspace-node[263358]: 2023-09-14 00:26:04 [Consensus] ⚙️  Syncing  0.0 bps, target=#413051 (20 peers), best: #411981 (0x29b1…42e4), finalized #232206 (0x9143…9e6f), ⬇ 58.7kiB/s ⬆ 33.1kiB/s
Sep 14 00:26:09 ironfish subspace-node[263358]: 2023-09-14 00:26:09 [Consensus] ⚙️  Syncing  0.0 bps, target=#413051 (20 peers), best: #411981 (0x29b1…42e4), finalized #232206 (0x9143…9e6f), ⬇ 45.2kiB/s ⬆ 31.4kiB/s
Sep 14 00:26:14 ironfish subspace-node[263358]: 2023-09-14 00:26:14 [Consensus] ⚙️  Syncing  0.0 bps, target=#413053 (20 peers), best: #411981 (0x29b1…42e4), finalized #232206 (0x9143…9e6f), ⬇ 57.4kiB/s ⬆ 33.7kiB/s
Sep 14 00:26:15 ironfish subspace-node[263358]: ====================
Sep 14 00:26:15 ironfish subspace-node[263358]: Version: 0.1.0-5c2433599f4
Sep 14 00:26:15 ironfish subspace-node[263358]:    0: sp_panic_handler::set::{{closure}}
Sep 14 00:26:15 ironfish subspace-node[263358]:    1: <alloc::boxed::Box<F,A> as core::ops::function::Fn<Args>>::call
Sep 14 00:26:15 ironfish subspace-node[263358]:              at rustc/399b068235ceea440540539b3bfd1aeb82214a28/library/alloc/src/boxed.rs:2021:9
Sep 14 00:26:15 ironfish subspace-node[263358]:       std::panicking::rust_panic_with_hook
Sep 14 00:26:15 ironfish subspace-node[263358]:              at rustc/399b068235ceea440540539b3bfd1aeb82214a28/library/std/src/panicking.rs:733:13
Sep 14 00:26:15 ironfish subspace-node[263358]:    2: std::panicking::begin_panic_handler::{{closure}}
Sep 14 00:26:15 ironfish subspace-node[263358]:              at rustc/399b068235ceea440540539b3bfd1aeb82214a28/library/std/src/panicking.rs:621:13
Sep 14 00:26:15 ironfish subspace-node[263358]:    3: std::sys_common::backtrace::__rust_end_short_backtrace
Sep 14 00:26:15 ironfish subspace-node[263358]:              at rustc/399b068235ceea440540539b3bfd1aeb82214a28/library/std/src/sys_common/backtrace.rs:151:18
Sep 14 00:26:15 ironfish subspace-node[263358]:    4: rust_begin_unwind
Sep 14 00:26:15 ironfish subspace-node[263358]:              at rustc/399b068235ceea440540539b3bfd1aeb82214a28/library/std/src/panicking.rs:617:5
Sep 14 00:26:15 ironfish subspace-node[263358]:    5: core::panicking::panic_fmt
Sep 14 00:26:15 ironfish subspace-node[263358]:              at rustc/399b068235ceea440540539b3bfd1aeb82214a28/library/core/src/panicking.rs:67:14
Sep 14 00:26:15 ironfish subspace-node[263358]:    6: core::panicking::panic_display
Sep 14 00:26:15 ironfish subspace-node[263358]:              at rustc/399b068235ceea440540539b3bfd1aeb82214a28/library/core/src/panicking.rs:150:5
Sep 14 00:26:15 ironfish subspace-node[263358]:       core::panicking::panic_str
Sep 14 00:26:15 ironfish subspace-node[263358]:              at rustc/399b068235ceea440540539b3bfd1aeb82214a28/library/core/src/panicking.rs:134:5
Sep 14 00:26:15 ironfish subspace-node[263358]:       core::option::expect_failed
Sep 14 00:26:15 ironfish subspace-node[263358]:              at rustc/399b068235ceea440540539b3bfd1aeb82214a28/library/core/src/option.rs:1952:5
Sep 14 00:26:15 ironfish subspace-node[263358]:    7: subspace_service::sync_from_dsn::import_blocks::import_blocks_from_dsn::{{closure}}
Sep 14 00:26:15 ironfish subspace-node[263358]:    8: subspace_service::new_full::{{closure}}::{{closure}}
Sep 14 00:26:15 ironfish subspace-node[263358]:    9: <futures_util::future::future::Map<Fut,F> as core::future::future::Future>::poll
Sep 14 00:26:15 ironfish subspace-node[263358]:   10: <tracing_futures::Instrumented<T> as core::future::future::Future>::poll
Sep 14 00:26:15 ironfish subspace-node[263358]:   11: tokio::runtime::task::raw::poll
Sep 14 00:26:15 ironfish subspace-node[263358]:   12: std::sys_common::backtrace::__rust_begin_short_backtrace
Sep 14 00:26:15 ironfish subspace-node[263358]:   13: core::ops::function::FnOnce::call_once{{vtable.shim}}
Sep 14 00:26:15 ironfish subspace-node[263358]:   14: <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
Sep 14 00:26:15 ironfish subspace-node[263358]:              at rustc/399b068235ceea440540539b3bfd1aeb82214a28/library/alloc/src/boxed.rs:2007:9
Sep 14 00:26:15 ironfish subspace-node[263358]:       <alloc::boxed::Box<F,A> as core::ops::function::FnOnce<Args>>::call_once
Sep 14 00:26:15 ironfish subspace-node[263358]:              at rustc/399b068235ceea440540539b3bfd1aeb82214a28/library/alloc/src/boxed.rs:2007:9
Sep 14 00:26:15 ironfish subspace-node[263358]:       std::sys::unix::thread::Thread::new::thread_start
Sep 14 00:26:15 ironfish subspace-node[263358]:              at rustc/399b068235ceea440540539b3bfd1aeb82214a28/library/std/src/sys/unix/thread.rs:108:17
Sep 14 00:26:15 ironfish subspace-node[263358]:   15: <unknown>
Sep 14 00:26:15 ironfish subspace-node[263358]:   16: <unknown>
Sep 14 00:26:15 ironfish subspace-node[263358]: Thread 'tokio-runtime-worker' panicked at 'Not empty, checked above; qed', /home/rabinovitch/subspace/crates/subspace-service/src/sync_from_dsn/import_blocks.rs:230
Sep 14 00:26:15 ironfish subspace-node[263358]: This is a bug. Please report it at:
Sep 14 00:26:15 ironfish subspace-node[263358]:         https://forum.autonomys.xyz
Sep 14 00:26:15 ironfish systemd[1]: subspace.service: Main process exited, code=exited, status=1/FAILURE
Sep 14 00:26:15 ironfish systemd[1]: subspace.service: Failed with result 'exit-code'.
Sep 14 00:26:15 ironfish systemd[1]: subspace.service: Consumed 59min 680ms CPU time.

I observe such a problem on two different servers updated to the latest version. The Ubuntu version, however, is the same.

Hi!
The team is aware of this problem, we are expecting a fix

1 Like

Should be fixed in sep-13-2 release, sorry for this

1 Like