Failed to allocate bytes exception when running Docker on aarch64

nazar-pc · September 25, 2022, 12:30pm

I was able to get the same error right after start on my poor Raspberry PI 3B+ with just 1G of RAM.

I believe my earlier guess that it is related to memory overcommitment was correct. Default value of 0 means overcommitment will sometimes be allowed and sometimes not. Setting it to 1 tells kernel to always allow overcommitment (but things can and will crash if you run out of memory).

I need huge amounts of SWAP on Raspberry PI to run Subspace node and only have USB 2.0 interfaces, so it is basically impossible, but I think you should have no issues with 8G of RAM if I was able to sync successfully with just 4G + 4G SWAP.

I also have Orange PI 3 with 2G of RAM, but kind of lazy trying it for now.

To always allow overcommitment add vm.overcommit_memory = 1 to the end of /etc/sysctl.conf and run sudo sysctl -p to apply it without reboot. After this you should be able to sync successfully.

jim-counter · September 25, 2022, 4:03pm

Was really excited to try this. But I’m afraid it’s no dice for me. Any chance I followed the steps incorrectly?

jim@raspberrypi4:~/subspace-testnet/raspberrypi4 $ cat /proc/sys/vm/overcommit_memory
1
jim@raspberrypi4:~/subspace-testnet/raspberrypi4 $ sudo docker logs subspace-node --tail=39
2022-09-25 16:01:53 Subspace    
2022-09-25 16:01:53 ✌️  version 0.1.0-unknown    
2022-09-25 16:01:53 ❤️  by Subspace Labs <https://subspace.network>, 2021-2022    
2022-09-25 16:01:53 📋 Chain specification: Subspace Gemini 2a    
2022-09-25 16:01:53 🏷  Node name: counterpoint_aarch64_pi4_hdd    
2022-09-25 16:01:53 👤 Role: AUTHORITY    
2022-09-25 16:01:53 💾 Database: ParityDb at /var/subspace/chains/subspace_gemini_2a/paritydb/full    
2022-09-25 16:01:53 ⛓  Native runtime: subspace-4 (subspace-0.tx0.au0)    

====================

Version: 0.1.0-unknown

   0: sp_panic_handler::set::{{closure}}
   1: std::panicking::rust_panic_with_hook
   2: std::panicking::begin_panic_handler::{{closure}}
   3: std::sys_common::backtrace::__rust_end_short_backtrace
   4: rust_begin_unwind
   5: core::panicking::panic_fmt
   6: sc_consensus_subspace::archiver::initialize_archiver
   7: subspace_service::new_partial
   8: subspace_service::new_full::{{closure}}
   9: <core::future::from_generator::GenFuture<T> as core::future::future::Future>::poll
  10: sc_cli::runner::Runner<C>::run_node_until_exit
  11: subspace_node::main
  12: std::sys_common::backtrace::__rust_begin_short_backtrace
  13: std::rt::lang_start::{{closure}}
  14: main
  15: __libc_start_main
  16: <unknown>


Thread 'main' panicked at 'Failed to make runtime API call during last archived block search: Application(VersionInvalid("cannot create the wasmtime engine: failed to create memory pool mapping: mmap failed to allocate 0x3080000000 bytes: Cannot allocate memory (os error 12)"))', /code/crates/sc-consensus-subspace/src/archiver.rs:72

This is a bug. Please report it at:

        https://forum.autonomys.xyz

2022-09-25 16:01:57 [PrimaryChain] Cannot create a runtime error=Other("cannot create the wasmtime engine: failed to create memory pool mapping: mmap failed to allocate 0x3080000000 bytes: Cannot allocate memory (os error 12)")

nazar-pc · September 25, 2022, 8:28pm

If cat /proc/sys/vm/overcommit_memory says 1 then you did it correctly. I guess there is some other reason then.
I’m fairly certain it is because of the OS tuning in a specific way since it worked on the device with half the RAM.
I guess I try my Orange PI 3 too then.

nazar-pc · September 26, 2022, 1:46pm

Orange PI 3 with just 2G of RAM and 4G of swap also works fine (Ubuntu 20.04). Will create an upstream bug report.

In the meantime, @jim-counter, can you try Ubuntu 20.04 on Raspberry PI 4 instead of Debian so we can exclude software side?

nazar-pc · October 18, 2022, 7:24am

@jim-counter did you have a chance to try Ubuntu distro and see if it works there?

jim-counter · October 19, 2022, 10:36pm

Hey Nazar, it’s on my ever increasing list of things to do - not forgotten about!

It takes a few days to hit the error so I’ll follow up when it does.

nazar-pc · October 20, 2022, 1:28am

I imagine if you store it on external drive, you can just move problematic node files and try under different OS.
Also should not take days to get to that height, but depends on Internet connection speed.

jim-counter · October 20, 2022, 11:36am

Hi Nazar, I just grabbed a new SD card and put Ubuntu Server 22.04 LTS on it:

jim@raspberrypi4:~$ hostnamectl
 Static hostname: raspberrypi4
       Icon name: computer
      Machine ID: 8511376505be4c62aef3d62e23265437
         Boot ID: 9cafaf328010480e9e95403d84fe0ed0
Operating System: Ubuntu 22.04.1 LTS
          Kernel: Linux 5.15.0-1017-raspi
    Architecture: arm64

Mounted the external drive and span up the node with exactly the same config as before and it’s working. A relief as I didn’t know if I’d have to wipe and sync the first few K blocks again. I note that it seems to identify the OS version weirdly:

subspace-node    | 2022-10-20 11:16:33 [PrimaryChain] 💻 Linux distribution: Ubuntu 20.04.5 LTS

But the important thing is that it’s back trying to sync and not constantly boot-looping with the previously observed crash. This makes me think this is a Raspbian specific issue.

subspace-node    | 2022-10-20 11:15:10 Subspace    
subspace-node    | 2022-10-20 11:15:10 ✌️  version 0.1.0-unknown    
subspace-node    | 2022-10-20 11:15:10 ❤️  by Subspace Labs <https://subspace.network>, 2021-2022    
subspace-node    | 2022-10-20 11:15:10 📋 Chain specification: Subspace Gemini 2a    
subspace-node    | 2022-10-20 11:15:10 🏷  Node name: counterpoint_aarch64_pi4_hdd    
subspace-node    | 2022-10-20 11:15:10 👤 Role: AUTHORITY    
subspace-node    | 2022-10-20 11:15:10 💾 Database: ParityDb at /var/subspace/chains/subspace_gemini_2a/paritydb/full    
subspace-node    | 2022-10-20 11:15:10 ⛓  Native runtime: subspace-3 (subspace-0.tx0.au0)    
subspace-node    | 2022-10-20 11:16:33 [PrimaryChain] Last archived block 5600    
subspace-node    | 2022-10-20 11:16:33 [PrimaryChain] Archiving already produced blocks 5601..=5635    
subspace-node    | 2022-10-20 11:16:33 [PrimaryChain] 🏷  Local node identity is: 12D3KooWGT1AVyHMxJiCNxA6RUEuVdSVFj52EeEp88QxwrpBa5Ru    
subspace-node    | 2022-10-20 11:16:33 [PrimaryChain] 🧑‍🌾 Starting Subspace Authorship worker    
subspace-node    | 2022-10-20 11:16:33 [PrimaryChain] 💻 Operating system: linux    
subspace-node    | 2022-10-20 11:16:33 [PrimaryChain] 💻 CPU architecture: aarch64    
subspace-node    | 2022-10-20 11:16:33 [PrimaryChain] 💻 Target environment: gnu    
subspace-node    | 2022-10-20 11:16:33 [PrimaryChain] 💻 Memory: 7807MB    
subspace-node    | 2022-10-20 11:16:33 [PrimaryChain] 💻 Kernel: 5.15.0-1017-raspi    
subspace-node    | 2022-10-20 11:16:33 [PrimaryChain] 💻 Linux distribution: Ubuntu 20.04.5 LTS    
subspace-node    | 2022-10-20 11:16:33 [PrimaryChain] 💻 Virtual machine: no    
subspace-node    | 2022-10-20 11:16:33 [PrimaryChain] 📦 Highest known block at #5735    
subspace-node    | 2022-10-20 11:16:33 [PrimaryChain] 〽️ Prometheus exporter started at 0.0.0.0:9615    
subspace-node    | 2022-10-20 11:16:33 [PrimaryChain] Running JSON-RPC HTTP server: addr=127.0.0.1:9933, allowed origins=None    
subspace-node    | 2022-10-20 11:16:33 [PrimaryChain] Running JSON-RPC WS server: addr=0.0.0.0:9944, allowed origins=None    
subspace-node    | 2022-10-20 11:16:33 [PrimaryChain] creating instance on iface 172.19.0.4    
subspace-node    | 2022-10-20 11:16:34 [PrimaryChain] 🔍 Discovered new external address for our node: /ip4/x.x.x.x/tcp/30336/p2p/12D3KooWGT1AVyHMxJiCNxA6RUEuVdSVFj52EeEp88QxwrpBa5Ru    
subspace-node    | 2022-10-20 11:16:35 [PrimaryChain] ✅ no migration for System    
subspace-node    | 2022-10-20 11:16:35 [PrimaryChain] ✅ no migration for Timestamp    
subspace-node    | 2022-10-20 11:16:35 [PrimaryChain] ✅ no migration for Subspace    
subspace-node    | 2022-10-20 11:16:35 [PrimaryChain] ✅ no migration for OffencesSubspace    
subspace-node    | 2022-10-20 11:16:35 [PrimaryChain] ✅ no migration for Rewards    
subspace-node    | 2022-10-20 11:16:35 [PrimaryChain] ✅ no migration for Balances    
subspace-node    | 2022-10-20 11:16:35 [PrimaryChain] ✅ no migration for TransactionFees    
subspace-node    | 2022-10-20 11:16:35 [PrimaryChain] ✅ no migration for TransactionPayment    
subspace-node    | 2022-10-20 11:16:35 [PrimaryChain] ✅ no migration for Utility    
subspace-node    | 2022-10-20 11:16:35 [PrimaryChain] ✅ no migration for Feeds    
subspace-node    | 2022-10-20 11:16:35 [PrimaryChain] ✅ no migration for GrandpaFinalityVerifier    
subspace-node    | 2022-10-20 11:16:35 [PrimaryChain] ✅ no migration for ObjectStore    
subspace-node    | 2022-10-20 11:16:35 [PrimaryChain] ✅ no migration for Executor    
subspace-node    | 2022-10-20 11:16:35 [PrimaryChain] ✅ no migration for RuntimeConfigs    
subspace-node    | 2022-10-20 11:16:35 [PrimaryChain] ✅ no migration for Vesting    
subspace-node    | 2022-10-20 11:16:35 [PrimaryChain] ✅ no migration for Sudo    
subspace-node    | 2022-10-20 11:16:38 [PrimaryChain] ⚙️  Syncing, target=#627537 (49 peers), best: #5806 (0xf037…5e9c), finalized #5705 (0xb658…1131), ⬇ 2.0MiB/s ⬆ 29.2kiB/s    
subspace-farmer  | 2022-10-20T11:16:42.593287Z  INFO subspace_farmer::utils: Increase file limit from soft to hard (limit is 1048576)
subspace-farmer  | 2022-10-20T11:16:42.603774Z  INFO subspace_farmer::commands::farm: Connecting to node at ws://node:9944
subspace-farmer  | 2022-10-20T11:16:42.604339Z  INFO subspace_farmer::commands::farm: Relay listening on /memory/9396317801340741873/p2p/12D3L7AV9KNc9YCP7E1Uu1tagjkFoCBRZQDnmWoC2hwqp9986VsH
subspace-farmer  | 2022-10-20T11:16:42.605142Z  INFO subspace_farmer::commands::farm: Relay listening on /ip4/172.19.0.6/tcp/40336/p2p/12D3L7AV9KNc9YCP7E1Uu1tagjkFoCBRZQDnmWoC2hwqp9986VsH
subspace-farmer  | 2022-10-20T11:16:42.605278Z  INFO subspace_farmer::commands::farm: Relay listening on /ip4/127.0.0.1/tcp/40336/p2p/12D3L7AV9KNc9YCP7E1Uu1tagjkFoCBRZQDnmWoC2hwqp9986VsH
subspace-farmer  | 2022-10-20T11:16:42.609844Z  INFO jsonrpsee_client_transport::ws: Connection established to target: Target { sockaddrs: [], host: "node", host_header: "node:9944", _mode: Plain, path_and_query: "/" }
subspace-farmer  | 2022-10-20T11:16:42.613937Z  INFO jsonrpsee_client_transport::ws: Connection established to target: Target { sockaddrs: [], host: "node", host_header: "node:9944", _mode: Plain, path_and_query: "/" }
subspace-node    | 2022-10-20 11:16:42 Accepting new connection 1/100
subspace-node    | 2022-10-20 11:16:42 Accepting new connection 2/100
subspace-farmer  | 2022-10-20T11:16:42.693980Z  INFO single_plot_farm{single_plot_farm_id=01GC9QGX7EFP01W07Q5QW49VXS}: subspace_farmer::single_plot_farm: Opening plot
subspace-farmer  | 2022-10-20T11:16:42.694358Z  INFO single_plot_farm{single_plot_farm_id=01GC9QGX7E3SEHF0T9VMWT0501}: subspace_farmer::single_plot_farm: Opening plot
subspace-farmer  | 2022-10-20T11:16:42.706454Z  INFO single_plot_farm{single_plot_farm_id=01GC9QGX7ETH4NABRFKJZDZ6RR}: subspace_farmer::single_plot_farm: Opening plot
subspace-farmer  | 2022-10-20T11:16:42.706624Z  INFO single_plot_farm{single_plot_farm_id=01GC9QGX7EDATKH4Z9H79BFK24}: subspace_farmer::single_plot_farm: Opening plot
subspace-node    | 2022-10-20 11:16:43 [PrimaryChain] Waiting for farmer to receive and acknowledge archived segment    
subspace-node    | 2022-10-20 11:16:43 [PrimaryChain] ⚙️  Syncing  0.0 bps, target=#627537 (67 peers), best: #5806 (0xf037…5e9c), finalized #5705 (0xb658…1131), ⬇ 54.1kiB/s ⬆ 11.7kiB/s    
subspace-node    | 2022-10-20 11:16:48 [PrimaryChain] Waiting for farmer to receive and acknowledge archived segment    
subspace-node    | 2022-10-20 11:16:48 [PrimaryChain] ⚙️  Syncing  0.0 bps, target=#627537 (75 peers), best: #5806 (0xf037…5e9c), finalized #5705 (0xb658…1131), ⬇ 91.2kiB/s ⬆ 22.0kiB/s    
subspace-node    | 2022-10-20 11:16:53 [PrimaryChain] Waiting for farmer to receive and acknowledge archived segment    
subspace-node    | 2022-10-20 11:16:53 [PrimaryChain] ⚙️  Syncing  0.0 bps, target=#627537 (75 peers), best: #5806 (0xf037…5e9c), finalized #5705 (0xb658…1131), ⬇ 182.1kiB/s ⬆ 106.4kiB/s    
subspace-node    | 2022-10-20 11:16:58 [PrimaryChain] Waiting for farmer to receive and acknowledge archived segment    
subspace-node    | 2022-10-20 11:16:58 [PrimaryChain] ⚙️  Syncing  0.0 bps, target=#627537 (75 peers), best: #5806 (0xf037…5e9c), finalized #5705 (0xb658…1131), ⬇ 196.1kiB/s ⬆ 141.2kiB/s    
subspace-node    | 2022-10-20 11:17:03 [PrimaryChain] Waiting for farmer to receive and acknowledge archived segment    
subspace-node    | 2022-10-20 11:17:03 [PrimaryChain] ⚙️  Syncing  0.0 bps, target=#627537 (75 peers), best: #5806 (0xf037…5e9c), finalized #5705 (0xb658…1131), ⬇ 187.7kiB/s ⬆ 120.7kiB/s    
subspace-node    | 2022-10-20 11:17:08 [PrimaryChain] Waiting for farmer to receive and acknowledge archived segment    
subspace-node    | 2022-10-20 11:17:08 [PrimaryChain] ⚙️  Syncing  0.0 bps, target=#627537 (75 peers), best: #5806 (0xf037…5e9c), finalized #5705 (0xb658…1131), ⬇ 195.8kiB/s ⬆ 136.9kiB/s    
subspace-node    | 2022-10-20 11:17:13 [PrimaryChain] Waiting for farmer to receive and acknowledge archived segment    
subspace-node    | 2022-10-20 11:17:13 [PrimaryChain] ⚙️  Syncing  0.0 bps, target=#627537 (75 peers), best: #5806 (0xf037…5e9c), finalized #5705 (0xb658…1131), ⬇ 181.3kiB/s ⬆ 119.9kiB/s    
subspace-node    | 2022-10-20 11:17:18 [PrimaryChain] Waiting for farmer to receive and acknowledge archived segment    
subspace-node    | 2022-10-20 11:17:18 [PrimaryChain] ⚙️  Syncing  0.0 bps, target=#627537 (75 peers), best: #5806 (0xf037…5e9c), finalized #5705 (0xb658…1131), ⬇ 156.3kiB/s ⬆ 83.8kiB/s    
subspace-node    | 2022-10-20 11:17:23 [PrimaryChain] Waiting for farmer to receive and acknowledge archived segment    
subspace-node    | 2022-10-20 11:17:23 [PrimaryChain] ⚙️  Syncing  0.0 bps, target=#627537 (75 peers), best: #5806 (0xf037…5e9c), finalized #5705 (0xb658…1131), ⬇ 123.2kiB/s ⬆ 60.5kiB/s    
subspace-node    | 2022-10-20 11:17:28 [PrimaryChain] Waiting for farmer to receive and acknowledge archived segment    
subspace-node    | 2022-10-20 11:17:28 [PrimaryChain] ⚙️  Syncing  0.0 bps, target=#627545 (63 peers), best: #5806 (0xf037…5e9c), finalized #5705 (0xb658…1131), ⬇ 102.4kiB/s ⬆ 36.7kiB/s    
subspace-node    | 2022-10-20 11:17:33 [PrimaryChain] Waiting for farmer to receive and acknowledge archived segment    
subspace-node    | 2022-10-20 11:17:34 [PrimaryChain] ⚙️  Syncing  0.0 bps, target=#627547 (70 peers), best: #5806 (0xf037…5e9c), finalized #5705 (0xb658…1131), ⬇ 78.6kiB/s ⬆ 8.9kiB/s

nazar-pc · October 20, 2022, 12:03pm

Thanks for testing!
Reported upstream to Substrate developers: Crash on runtime upgrade related to wasmtime (Raspbian only) · Issue #12538 · paritytech/substrate · GitHub

jim-counter · October 31, 2022, 10:10pm

I read the update from the Substrate dev. Do you think we should add a note to the docs that a known issue precludes the use of Raspbian? I should be able to pull a PR together for that if you want?

nazar-pc · October 31, 2022, 10:41pm

Not sure, probably if we get more reports of this kind. I expect by Gemini 3 it’ll “just work” out of the box.

jim-counter · October 31, 2022, 11:39pm

Will keep my eyes peeled for any reported occurrences

Topic		Replies	Views
Subspace Node (version gemini-3h) cannot run on ARM64 Support	6	249	October 1, 2024
Subspace node build failed Support	5	191	April 4, 2024
Synchronization does not fully reach, the node crashes when starting farming Support gemini , nodes , error , farmer	11	738	March 29, 2023
Subspace-node-1 \| Error: SubstrateCli(Input("Error opening spec file `gemini-3c`: No such file or directory (os error 2)")) Support	12	489	April 22, 2023
Sync issues on multiple nodes with different configurations Support	10	573	June 26, 2022

Failed to allocate bytes exception when running Docker on aarch64

Related topics