ROCm GPU support (AMD)

Feedback & Suggestions
, , ,
1

:warning: Please use #farmer-chat on Discord for regular discussion about ROCm (like plotting performance, installation questions, etc.), avoid anything that would result in a bunch of messages in this thread :pray:

Current status (as of September 27th, 2024)

So here is a test build with ROCm support (Linux only): Snapshot build · autonomys/subspace@0ca3945 · GitHub

ROCm is much more difficult (and even painful) to work with, so a lot of caveats:

  • Windows builds are not available and not planned in short term; you might be able to compile it from source, but it is awkward and painful process, I wasted too much time as is trying to make it work in CI (not 100% sure why it still fails, contributions are welcome if you figure it out)
    • That said, you should be able to run it on Windows under WSL with CLI
  • It is unlikely that there will be ROCm support in Space Acres any time soon, though I’ll be making progress towards that
  • ROCm support is mutually exclusive with CUDA at the moment, so we have a separate executable where CUDA is replaced with ROCm support, long term I’ll try to combine them
  • Unfortunately default ROCm version shipped in Ubuntu repos works, but is horrendously slow, making it literally more harmful than helpful, which means custom AMD repo and additional package installation is needed to get it to work
  • Official support is limited and different between Windows and Linux, you can count supported GPUs with your fingers and have a few left to spare
    • On linux only 79000 XT/XTX/GRE are officially supported
  • However, technically compiler can target more GPUs, so I have enabled all CDNA1-3/RDNA1-4 GPUs supported by the compiler, including APUs
    • That said I have no idea what works and what doesn’t, the only cards I have tested are known to work are 7900 XTX and 7600 XT, I’ll need community help to test as many GPUs as possible to correct list of supported GPUs if necessary
  • Only x86-64 builds have ROCm support, there is no official ROCm toolchain packaging for aarch64 at all right now

Regular CLI

On Ubuntu subspace-farmer-rocm-* executable can be used for ROCm support with corresponding CLI options being similar to CUDA and prefixed with --rocm.
As you might expect all detected GPUs will be used by default and CPU plotting is automatically disabled in such case.

Prepare your OS

In order to install necessary libraries go to Ubuntu native installation — ROCm installation (Linux) and follow these steps for your Ubuntu version:

  • Package signing key
  • Register ROCm packages

You don’t need a custom driver or full ROCm toolchain to use already compiled application, so skip all other steps.

Next install a single package with ROCm runtime:

sudo apt-get install --no-install-recommends hip-runtime-amd

And register the library so that farmer and other apps can find it:

echo "/opt/rocm/lib" | sudo tee /etc/ld.so.conf.d/rocm.conf > /dev/null
sudo ldconfig

Last step is to allow the user to access GPU for compute purposes, for this you need to add your user to render group (please never run farmer as root with sudo :pray: ):

sudo usermod -a -G render $LOGNAME

Now you’ll need to log out of your user profile and log back in or simply reboot for group changes to take effect and you’re ready to go.

Tip: you can install rocm-smi package to observe GPU load %, VRAM usage, etc.

Docker (container)

Container image now ships with a second executable /subspace-farmer-rocm (see explanation above as to why second binary is needed).

Here is how it can be used with Docker Compose, you can use this as an inspiration for other use cases:

services:
  famer:
    image: ghcr.io/autonomys/farmer:TAG
    # Beginning of ROCm-specific options
    devices:
      - /dev/kfd:/dev/kfd
      # This will make all GPUs accessible, you can also limit this
      # to individual devices like `/dev/dri/renderD128`
      - /dev/dri:/dev/dri
    security_opt:
      - seccomp:unconfined
    # `root` user can be replaced with ID of the user on the host system
    # that is in `render` group, needed to access `/dev/kfd` device
    user: root
    entrypoint: /subspace-farmer-rocm
    # End of ROCm-specific options
    ...the rest of typical options you'd normally use

WSL

You tell me, I have not tried it myself, but hypothetically it should work :man_shrugging:

If you figure out exact instructions, please create a separate forum thread and consider contributing to our official documentation.

GPU support

As mentioned above, official GPU support is very limited, so we’ll have to try and see what happens ourself.

Device Type Compatibility
Radeon RX 7900 XTX dGPU :white_check_mark:
Radeon RX 7600 XT dGPU :white_check_mark:
Radeon RX 6800 dGPU :white_check_mark:
Radeon RX 6600 dGPU :white_check_mark:
Radeon RX 5700 dGPU :x:
Radeon RX 5600 dGPU :x:

Please post in this thread dGPU/APU you have tried and whether it worked or not (and full farmer logs if there were some errors) and I’ll update the table. Ideally wait for the first solution to be signed, so we know it both plotted sector and did it correctly. The goal is to remove support for GPUs that are known to not work.

Note that I’m only interested in Ubuntu and direct derivatives for now.

3 Likes