ROCm GPU support (AMD)

Feedback & Suggestions
, , ,
1

:warning: Please use #farmer-chat on Discord for regular discussion about ROCm (like plotting performance, installation questions, etc.), avoid anything that would result in a bunch of messages in this thread :pray:

Current status (as of October 10, 2024)

So here is a test build with ROCm support (Ubuntu and Windows): Snapshot build · autonomys/subspace@3971130 · GitHub

ROCm is much more difficult (and even painful) to work with, so a lot of caveats:

  • Space Acres will be possible to compile with ROCm support for networks after Gemini 3h, though I’m not sure it’ll be included in the official builds any time soon
  • ROCm support is mutually exclusive with CUDA at the moment, so we have a separate executable where CUDA is replaced with ROCm support, long term I’ll try to combine them
  • Unfortunately default ROCm version shipped in Ubuntu repos works, but is horrendously slow, making it literally more harmful than helpful, which means custom AMD repo and additional package installation is needed on Ubuntu to get it to work
  • On Windows special runtime libraries need to be installed in addition to having standard AMD driver
  • Official support is limited and different between Windows and Linux, you can count supported GPUs with your fingers and have a few left to spare
    • On linux only 79000 XT/XTX/GRE are officially supported
  • However, technically compiler can target more GPUs, so I have enabled all CDNA1-3/RDNA1-4 GPUs supported by the compiler, including APUs
    • So far RDNA 1 doesn’t seem to work, but newer cards do, see table a the bottom of this post
  • Only x86-64 builds have ROCm support, there is no official ROCm toolchain packaging for aarch64 at all right now

Regular CLI

On Ubuntu and Windows subspace-farmer-rocm-* executable can be used for ROCm support with corresponding CLI options being similar to CUDA and prefixed with --rocm.
As you might expect all detected GPUs will be used by default and CPU plotting is automatically disabled in such case.

Prepare your OS (Ubuntu)

In order to install necessary libraries go to Ubuntu native installation — ROCm installation (Linux) and follow these steps for your Ubuntu version:

  • Package signing key
  • Register ROCm packages

You don’t need a custom driver or full ROCm toolchain to use already compiled application, so skip all other steps.

Next install a single package with ROCm runtime:

sudo apt-get install --no-install-recommends hip-runtime-amd

And register the library so that farmer and other apps can find it:

echo "/opt/rocm/lib" | sudo tee /etc/ld.so.conf.d/rocm.conf > /dev/null
sudo ldconfig

Last step is to allow the user to access GPU for compute purposes, for this you need to add your user to render group (please never run farmer as root with sudo :pray: ):

sudo usermod -a -G render $LOGNAME

Now you’ll need to log out of your user profile and log back in or simply reboot for group changes to take effect and you’re ready to go.

Prepare your OS (Windows)

In order to install necessary libraries go to https://www.amd.com/en/developer/resources/rocm-hub/hip-sdk.html and download ROCm version 6.1.2 for your Windows version.

In the installer just the HIP RTC Runtime should be enough under “HIP Runtime Compiler → HIP RTC Runtime 6.1.0”, everything else can be unchecked:

Знімок екрана з 2024-10-10 00-53-42
Знімок екрана з 2024-10-10 00-53-421196×837 192 KB

Tip: you can install rocm-smi package on Ubuntu to observe GPU load %, VRAM usage, etc.

Docker (container)

Container image now ships with a second executable /subspace-farmer-rocm (see explanation above as to why second binary is needed).

Here is how it can be used with Docker Compose, you can use this as an inspiration for other use cases:

services:
  famer:
    image: ghcr.io/autonomys/farmer:TAG
    # Beginning of ROCm-specific options
    devices:
      - /dev/kfd:/dev/kfd
      # This will make all GPUs accessible, you can also limit this
      # to individual devices like `/dev/dri/renderD128`
      - /dev/dri:/dev/dri
    security_opt:
      - seccomp:unconfined
    # `root` user can be replaced with ID of the user on the host system
    # that is in `render` group, needed to access `/dev/kfd` device
    user: root
    entrypoint: /subspace-farmer-rocm
    # End of ROCm-specific options
    ...the rest of typical options you'd normally use

WSL

You tell me, I have not tried it myself, but hypothetically it should work :man_shrugging:

If you figure out exact instructions, please create a separate forum thread and consider contributing to our official documentation.

GPU support

As mentioned above, official GPU support is very limited, so we’ll have to try and see what happens ourself.

Device Type Ubuntu Windows
Radeon RX 7900 XTX dGPU :white_check_mark: :x:
Radeon RX 7600 XT dGPU :white_check_mark: :grey_question:
Radeon RX 7600 dGPU :white_check_mark: :grey_question:
Radeon RX 6800 dGPU :white_check_mark: :grey_question:
Radeon RX 6700 XT dGPU :grey_question: :x:
Radeon RX 6600 XT dGPU :white_check_mark: :x:
Radeon RX 6600 dGPU :white_check_mark: :x:
Radeon RX 5700 XT dGPU :x: :grey_question:
Radeon RX 5700 dGPU :x: :grey_question:
Radeon RX 5600 dGPU :x: :grey_question:
AMD BC-250 APU :x: :grey_question:

:grey_question: means untested, but if it is between two supported GPUs then it’ll likely work as well.

NOTE: Consumer GPUs are not supported on Windows, AMD engineers recognized this as a bug and it should be fixed in the future.

Please post in this thread dGPU/APU you have tried and whether it worked or not (and full farmer logs if there were some errors) and I’ll update the table. Ideally wait for the first solution to be signed, so we know it both plotted sector and did it correctly. The goal is to remove support for GPUs that are known to not work.

Note that I’m only interested in Ubuntu and direct derivatives for now on Linux side and Windows.

4 Likes