Cannot block the current thread from within a runtime

Support
1

Issue Report

Environment

  • Operating System: Ubuntu 20.04
  • CPU Architecture: 3 core amd64
  • RAM: 4G
  • Storage: 80G
  • Plot Size: 1G
  • Subspace Deployment Method: CLI

Problem

My node just synced 1 hour ago and crashed like this:

2023-03-21T17:21:23.186657Z  INFO substrate: 💤 Idle (100 peers), best: #542055 (0x060e…31d5), finalized #541955 (0x8a3d…ae9a), ⬇ 505.1kiB/s ⬆ 452.5kiB/s    
2023-03-21T17:21:23.511938Z  INFO substrate: ✨ Imported #542055 (0x702d…9818)    
2023-03-21T17:21:24.547494Z  INFO substrate: ✨ Imported #542056 (0xe5d6…5757)    
2023-03-21T17:21:25.566925Z  INFO substrate: ✨ Imported #542057 (0x578a…4bea)    
2023-03-21T17:21:26.351758Z  INFO substrate: ✨ Imported #542058 (0x07a0…7da7)    
2023-03-21T17:21:27.018916Z  INFO new{user_space_pledged=Some(1000.0 MB)}:summary_path: subspace_cli::summary: close time.busy=35.3µs time.idle=9.39µs
Starting farmer ...
2023-03-21T17:21:27.020043Z  INFO subspace_sdk:🧑‍🌾 Initializing provider storage... db_path="/root/.local/share/subspace-cli/cache/providers_db" keys_limit=655360
2023-03-21T17:21:27.256011Z  INFO substrate: ✨ Imported #542059 (0xf49d…6f9a)    
2023-03-21T17:21:28.186949Z  INFO substrate: 💤 Idle (39 peers), best: #542059 (0xf49d…6f9a), finalized #541959 (0x7f70…526c), ⬇ 102.6kiB/s ⬆ 94.9kiB/s    
2023-03-21T17:21:30.059309Z  INFO subspace_sdk:🧑‍🌾 Initializing piece cache... db_path="/root/.local/share/subspace-cli/cache/piece_cache_db" size=330
2023-03-21T17:21:30.063497Z  INFO subspace_sdk:🧑‍🌾 Piece cache initialized successfully current_size=330
Farmer started successfully!
2023-03-21T17:21:30.468334Z  INFO single_disk_plot{disk_farm_index=0}: subspace_farmer::single_disk_plot: Subscribing to slot info notifications
2023-03-21T17:21:30.468850Z  INFO single_disk_plot{disk_farm_index=0}: subspace_farmer::reward_signing: Subscribing to reward signing notifications
2023-03-21T17:21:30.511069Z  WARN sync: 💔 Error importing block 0x5f17bd8e7afe23c744917cda8bd341efb89e38b7a0311483d4fc728f18900df3: block has an unknown parent    
The application panicked (crashed).
Message:  Cannot block the current thread from within a runtime. This happens because a function attempted to block the current thread while the thread is being used to drive asynchronous tasks.
Location: /home/runner/.cargo/git/checkouts/subspace-sdk-cf03f5a296714cdf/dfa443f/src/farmer.rs:647

Backtrace omitted. Run with RUST_BACKTRACE=1 environment variable to display it.
Run with RUST_BACKTRACE=full to include source snippets.
2023-03-21T17:21:30.522015Z  INFO new{user_space_pledged=Some(1000.0 MB)}: subspace_cli::summary: close time.busy=309µs time.idle=3.50s
2023-03-21T17:21:30.523741Z  INFO single_disk_plot{disk_farm_index=0}: subspace_farmer::single_disk_plot: close time.busy=54.3ms time.idle=398ms
2

Hello Nirnaeth

You running another instance on your server by any chance?

3

I checked no other instance or process. Node crashes right after it syncs.

5

Thank you, Nirnaeth

Escalating to Devs

6

I copied my whole database to my home computer which has 8 threads (4 cores) It also crashed right after it synced

7

Hello Nirnaeth

Thank you for the alert.

Opened an issue on Git. Devs work on it. Will keep you posted.

8

Thanks to @Fradique the fix is confirmed to be upgrade to the latest version of the software 🐞 [Bug]:Node crashes right after it syncs (no other instance or process running) · Issue #148 · subspace/subspace-cli · GitHub.

Related issue: 🐞 [Bug]: Synchronization issues · Issue #150 · subspace/subspace-cli · GitHub

9

How should we proceed? With v0.1.11 and a db wipe?

10

Try without a wipe first. I have been able to upgrade 0.1.10 → 0.1.11 without issue.

11

My farmer stops with no error right after it syncs

2023-03-28T12:44:27.196004Z  WARN sync: 💔 Error importing block 0x7c8e26d7db758d23ef9932fda55e1272b3e4f23d59cf3f9753280aa914697b84: block has an unknown parent    
2023-03-28T12:44:27.196184Z  WARN sync: 💔 Error importing block 0x5ff136f4f599d6eb1f67e3b24d150239cef8d575c8da9d58c6f5f4d60e3dbf4b: block has an unknown parent    
2023-03-28T12:44:27.674068Z  INFO new{user_space_pledged=Some(1000.0 MB)}:summary_path: subspace_cli::summary: close time.busy=3.88µs time.idle=6.35µs
Starting farmer ...
2023-03-28T12:44:27.674481Z  INFO subspace_sdk::farmer: Initializing provider storage... db_path="/root/.local/share/subspace-cli/cache/providers_db" keys_limit=655360
2023-03-28T12:44:27.675755Z  INFO subspace_sdk::farmer: Initializing piece cache... db_path="/root/.local/share/subspace-cli/cache/piece_cache_db" size=330
2023-03-28T12:44:27.677134Z  INFO subspace_sdk::farmer: Piece cache initialized successfully current_size=44
Farmer started successfully!
2023-03-28T12:44:27.689146Z  INFO single_disk_plot{disk_farm_index=0}: subspace_farmer::single_disk_plot: Subscribing to slot info notifications
2023-03-28T12:44:27.689272Z  INFO single_disk_plot{disk_farm_index=0}: subspace_farmer::reward_signing: Subscribing to reward signing notifications
2023-03-28T12:44:27.691654Z  INFO new{user_space_pledged=Some(1000.0 MB)}: subspace_cli::summary: close time.busy=201µs time.idle=17.4ms
2023-03-28T12:44:27.698908Z  INFO single_disk_plot{disk_farm_index=0}: subspace_farmer::single_disk_plot: close time.busy=4.76ms time.idle=5.77ms
root@ubuntu-4gb-hil-1:~#
12

With version 0.1.12 V3 everything seems fixed my node synced and running

13

Thank you for the feedback Nirnaeth!