subspace-node-ubuntu-aarch64-gemini-3h-2024-feb-05 failed to run due to memory allocation failure while physical memory size is 12 GB.
Environment
- Operating System: Ubuntu 22.04
- DDR Memory Size: 12 GB
- In Docker container run the Advanced CLI:
Problem
./subspace-node-ubuntu-aarch64-gemini-3h-2024-feb-05 run --chain gemini-3h --farmer --rpc-methods unsafe --rpc-listen-on 0.0.0.0:9944 --rpc-cors all --base-path /root/subspace/ --name “NodeName”
After run the command above, an error is reported immediately.
The earlier version gemini-3g node has no such problem.
2024-02-18T10:19:30.368611Z INFO subspace_node::commands::run: Subspace
2024-02-18T10:19:30.368683Z INFO subspace_node::commands::run: _ version 0.1.0-a2e0318a757
2024-02-18T10:19:30.368716Z INFO subspace_node::commands::run: _ by Subspace Labs https://subspace.network
2024-02-18T10:19:30.368733Z INFO subspace_node::commands::run: __ Chain specification: Subspace Gemini 3h
2024-02-18T10:19:30.368773Z INFO subspace_node::commands::run: _ Node name: NodeName
2024-02-18T10:19:30.368809Z INFO subspace_node::commands::run: __ Node path: /root/subspace/
2024-02-18T10:19:30.490046Z WARN Consensus: wasm-runtime: Cannot create a runtime error=Other(“cannot create the wasmtime engine: failed to create memory pool mapping: mmap failed to allocate 0x6080000000 bytes: Cannot allocate memory (os error 12)”)
Error: SubstrateService(Other(“Failed to build a full subspace node 1: Client(VersionInvalid("Other error happened while constructing the runtime: cannot create the wasmtime engine: failed to create memory pool mapping: mmap failed to allocate 0x6080000000 bytes: Cannot allocate memory (os error 12)"))”))
Please help to have a look and give some advice.
1 Like
It’d be helpful if you provided details about the hardware as well.
Generally Subspace node and farmer should run on aarch64 just fine, but some Linux distributions (notably Raspberry PI’s OS, at least in the past) configure kernel in a way that is not compatible with larger amounts of virtual memory required.
Hardware information attached
inxi -Fxz
System: Host: MC05367412004206 Kernel: 4.19.125 aarch64 bits: 32 gcc: 7.5.0
Console: tty 0 Distro: Ubuntu 18.04.5 LTS
Machine: No /sys/class/dmi; using dmidecode: no smbios data. Old system?
CPU: 8 core AArch64 rev 14 (aarch64) (-MCP-) bmips: arch: ARM
clock speeds: max: 1804 MHz 1: 1804 MHz 2: 1804 MHz 3: 1804 MHz
4: 1804 MHz 5: 1804 MHz 6: 1804 MHz 7: 1804 MHz 8: 1804 MHz
Graphics: Card: Failed to Detect Video Card!
Display Server: X.org 5.32 driver: N/A Resolution: 1280x680
OpenGL: renderer: GDI Generic version: 1.1.0 Direct Render: No
Audio: Card Failed to Detect Sound Card! Sound: ALSA v: k4.19.125
Network: Card: Realtek RTL8111/8168/8411 PCIE Gigabit Ethernet Controller
driver: r8168 port: 1000 bus-ID: 0002:01:00.0
IF: N/A state: N/A speed: N/A duplex: N/A mac: N/A
Drives: HDD Total Size: 64.0GB (63.9% used)
ID-1: /dev/sda model: SDINDDH4 size: 58.0GB temp: 0C
ID-2: /dev/sde model: SDINDDH4 size: 4.3GB temp: 0C
ID-3: /dev/sdc model: SDINDDH4 size: 0.0GB temp: 0C
ID-4: /dev/sdb model: SDINDDH4 size: 0.0GB temp: 0C
ID-5: /dev/sdd model: SDINDDH4 size: 0.1GB temp: 0C
ID-6: /dev/sdf model: SDINDDH4 size: 1.6GB temp: 0C
Partition: ID-1: / size: 7.8G used: 6.7G (86%) fs: ext4 dev: /dev/root
ID-2: /tmp size: 38G used: 33G (88%) fs: overlay dev: N/A
ID-3: /home size: 38G used: 33G (88%) fs: overlay dev: N/A
ID-4: /opt size: 38G used: 33G (88%) fs: overlay dev: N/A
ID-5: /usr size: 38G used: 33G (88%) fs: overlay dev: N/A
ID-6: /var size: 38G used: 33G (88%) fs: overlay dev: N/A
ID-7: swap-1 size: 6.08GB used: 0.00GB (0%) fs: swap dev: /dev/zram0
Sensors: None detected - is lm-sensors installed and configured?
Info: Processes: 414 Uptime: 2 days Memory: 3828.6/11592.2MB
Init: systemd runlevel: 3 Gcc sys: 7.5.0
Client: Shell (bash 4.4.201) inxi: 2.3.56
This is a VERY old Ubuntu version, we only support 20.04+ and 22.04+ is strongly recommended because we plan to remove support for 20.04 later this year. Also your initial message indicated you have 20.04, so I’m confused.
You will likely be able to compile software for 18.04 yourself, but no promises if it will work as expected.
Either way 8-core ARM CPU with 1.8GHz will probably not farm very successfully anyway and plotting will take a long time as well.
We are running Subspace Node within the docker container.
The docker container system is running Ubuntu 22.04.
The docker container hardware info attached.
System:
Kernel: 4.19.125 aarch64 bits: 64 compiler: gcc v: 7.5.0 Console: pty pts/0
Distro: Ubuntu 22.04.2 LTS (Jammy Jellyfish)
CPU:
Info: 3x 4-core model: AArch64 variant: armv8 bits: 64 type: MT AMP arch: aarch64 rev: E cache:
L1: 3x 256 KiB (768 KiB) L2: 3x 512 KiB (1.5 MiB) L3: 3x 4 MiB (12 MiB)
Speed (MHz): avg: 2164 high: 2842 min/max: 300:710:845/1805:2419:2842 cores: 1: 1805 2: 1805
3: 1805 4: 1805 5: 2419 6: 2419 7: 2419 8: 2842 bogomips: N/A
Features: Use -f option to see features
Graphics:
Device-1: msm-dai-q6-hdmi driver: msm_dai_q6_hdmi v: N/A bus-ID: N/A
Display: server: No display server data found. Headless machine? tty: 107x25
resolution: 1920x1080
Message: GL data unavailable in console for root.
Audio:
Device-1: msm-audio-apr driver: audio_apr bus-ID: N/A
Device-2: msm-dai-q6-hdmi driver: msm_dai_q6_hdmi bus-ID: N/A
Device-3: audio-ref-clk driver: audio_ref_clk bus-ID: N/A
Device-4: audio-ref-clk driver: audio_ref_clk bus-ID: N/A
Device-5: audio-ref-clk driver: audio_ref_clk bus-ID: N/A
Device-6: audio-ref-clk driver: audio_ref_clk bus-ID: N/A
Device-7: usb-audio-qmi-dev driver: uaudio_qmi bus-ID: N/A
Device-8: audio-ref-clk driver: audio_ref_clk bus-ID: N/A
Device-9: audio-ref-clk driver: audio_ref_clk bus-ID: N/A
Device-10: audio-ref-clk driver: audio_ref_clk bus-ID: N/A
Device-11: audio-ref-clk driver: audio_ref_clk bus-ID: N/A
Device-12: msm-audio-ion driver: msm_audio_ion bus-ID: N/A
Device-13: q6core-audio driver: q6core_audio bus-ID: N/A
Device-14: voice-mhi-audio driver: voice_mhi_audio bus-ID: N/A
Network:
Device-1: Realtek RTL8111/8168/8411 PCI Express Gigabit Ethernet driver: r8168 v: 8.046.00-NAPI
port: 1000 bus-ID: 0002:01:00.0
IF: eth0 state: up speed: 1000 Mbps duplex: full mac:
IF-ID-1: bond0 state: down mac:
IF-ID-2: bonding_masters state: N/A speed: N/A duplex: N/A mac: N/A
IF-ID-3: can0 state: down mac: N/A
IF-ID-4: docker0 state: down mac:
IF-ID-5: dummy0 state: unknown speed: N/A duplex: N/A mac:
IF-ID-6: ip6_vti0 state: down mac:
IF-ID-7: ip6tnl0 state: down mac:
IF-ID-8: ip_vti0 state: down mac:
IF-ID-9: sit0 state: down mac:
Drives:
Local Storage: total: 59.62 GiB used: 0 KiB (0.0%)
ID-1: /dev/sda model: SDINDDH4-64G size: 53.98 GiB
ID-2: /dev/sdb model: SDINDDH4-64G size: 4 MiB
ID-3: /dev/sdc model: SDINDDH4-64G size: 4 MiB
ID-4: /dev/sdd model: SDINDDH4-64G size: 128 MiB
ID-5: /dev/sde model: SDINDDH4-64G size: 4 GiB
ID-6: /dev/sdf model: SDINDDH4-64G size: 1.5 GiB
Partition:
Message: No partition data found.
Swap:
ID-1: swap-1 type: zram size: 5.66 GiB used: 2.5 MiB (0.0%) dev: /dev/zram0
Sensors:
Message: No sensor data found. Is lm-sensors configured?
Info:
Processes: 3 Uptime: 2d 22h 52m Memory: 11.32 GiB used: 933.2 MiB (8.1%) Init: N/A Compilers:
gcc: N/A Packages: 253 Shell: Bash v: 5.1.16 inxi: 3.3.13
What you write makes even less sense now. But running 22.04 container on 18.04 host will not work, you’ll be hitting many glibc errors. As I said, host OS must be 20.04 or newer, older hosts (and judging by the kernel version it is old indeed) are not officially supported and will never be.
The issue here is the same as Failed to allocate bytes exception when running Docker on aarch64 and caused by kernel compiled with 39-bit address space.
Ubuntu and other distros usually don’t have these issues, but Raspberry PI OS/Raspbian/Debian do hit it.
See Crash on runtime upgrade related to wasmtime (Raspbian only) · Issue #12538 · paritytech/substrate · GitHub and links mentioned there for details and things you’d want to change for kernel recompilation to fix this on current distro you’re using.