Docker error "dependency failed to start: container subspace-node-1 is unhealthy"

Support
,
1

Issue Report

Environment

  • Operating System:
    ubuntu
  • Pulsar/Advanced CLI/Docker:
    Advanced CLI

Problem

sudo docker compose up -d
[+] Running 2/2
 ✘ Container subspace-node-1    Error                                                                                                                                                                             2.8s
 ✔ Container subspace-farmer-1  Recreated                                                                                                                                                                         0.0s
dependency failed to start: container subspace-node-1 is unhealthy

I get this error when I follow the docs

Version: 0.1.0-22d5e48b0465b3634996b6fd9bdc29447a971a15

   0: sp_panic_handler::set::{{closure}}
   1: std::panicking::rust_panic_with_hook
   2: std::panicking::begin_panic_handler::{{closure}}
   3: std::sys_common::backtrace::__rust_end_short_backtrace
   4: rust_begin_unwind
   5: core::panicking::panic_fmt
   6: fdlimit::raise_fd_limit
   7: subspace_node::main
   8: std::sys_common::backtrace::__rust_begin_short_backtrace
   9: main
  10: __libc_start_main
  11: _start


Thread 'main' panicked at 'raise_fd_limit: error calling setrlimit: Operation not permitted (os error 1)', /root/.cargo/registry/src/index.crates.io-6f17d22bba15001f/fdlimit-0.2.1/src/lib.rs:105

This is a bug. Please report it at:

this is logs of docker logs subspace-node-1

[Paste any errors or relevant logs here]
2

That is interesting, I have not seen it before. Can you post docker-compose.yml contents?

There must be something in Docker configuration or the environment that prevents that operation from working.

Also if you can add RUST_BACKTRACE=full environment variable it might print something more useful in the backtrace or else I can help you with test build to debug this further.

3

this is my docker-compose.yml

version: "3.7"
services:
  node:
    image: ghcr.io/subspace/node:gemini-3g-2023-nov-03
    volumes:
      - node-data:/var/subspace:rw
    ports:
      - "0.0.0.0:30333:30333/udp"
      - "0.0.0.0:30333:30333/tcp"
      - "0.0.0.0:30433:30433/udp"
      - "0.0.0.0:30433:30433/tcp"
    restart: unless-stopped
    command:
      [
        "--chain", "gemini-3g",
        "--base-path", "/var/subspace",
        "--blocks-pruning", "256",
        "--state-pruning", "archive-canonical",
        "--port", "30333",
        "--dsn-listen-on", "/ip4/0.0.0.0/udp/30433/quic-v1",
        "--dsn-listen-on", "/ip4/0.0.0.0/tcp/30433",
        "--rpc-cors", "all",
        "--rpc-methods", "unsafe",
        "--rpc-external",
        "--no-private-ipv4",
        "--validator",
        "--name", "subspace"
      ]
    healthcheck:
      timeout: 5s
      interval: 30s
      retries: 600

  farmer:
    depends_on:
      node:
        condition: service_healthy
    image: ghcr.io/subspace/farmer:gemini-3g-2023-nov-03
    volumes:
      - farmer-data:/var/subspace:rw
    ports:
      - "0.0.0.0:30533:30533/udp"
      - "0.0.0.0:30533:30533/tcp"
    restart: unless-stopped
    command:
      [
        "farm",
        "--node-rpc-url", "ws://node:9944",
        "--listen-on", "/ip4/0.0.0.0/udp/30533/quic-v1",
        "--listen-on", "/ip4/0.0.0.0/tcp/30533",
        "--reward-address", "st9mKkKrtPb9kYwPpZpcev6C8b2o65YekjcoXeFT9Fn41ucSs",
        "path=/var/subspace,size=400G"
      ]
volumes:
  node-data:
  farmer-data:
4

Okay, so those are recommended options, shouldn’t cause issues. What Ubuntu version are you running this on and did you customize anything on it?

5

Ubuntu 20.04.6 LTS
and no customize

6

Did you install Docker itself from stock repository or from Docker’s (as in Install Docker Engine on Ubuntu | Docker Docs)?

7

maybe like this command

sudo apt install docker-ce docker-ce-cli containerd.io
8

Hm, “maybe” isn’t very accurate. Can you check if you have docker-ce or docker.io installed?

Either way it shouldn’t crash, I have submitted Catch panic generated by `fdlimit` call by nazar-pc · Pull Request #2155 · paritytech/polkadot-sdk · GitHub to fix it and we’ll pull into one of the future Subspace limits.

Without that you need to figure out why Docker doesn’t allow your container to change file descriptor limits, this is what is failing there.