Plotting stops, rewards continue, no errors - non-cluster ACLI

Support
1

Issue Report

Environment

  • Operating System: Ubuntu 24.04 (Linux Mint 22)
  • Space Acres/Advanced CLI/Docker: ACLI

Problem

I run two (non-cluster)machines on the Taurus network, with a separate machine running a node. They are each plotting to 6 SATA SSD and farming. Randomly, the creating of sectors stops without an error message, although farming continues and rewards are still received.

Machine 1
2024-10-29T00:46:57.771071Z  INFO {farm_index=1}:{sector_index=2972}: subspace_farmer::single_disk_farm::plotting: Plotting sector (39.69% complete)
2024-10-29T00:47:22.776154Z  INFO {farm_index=1}:{sector_index=2973}: subspace_farmer::single_disk_farm::plotting: Plotting sector (39.70% complete)
2024-10-29T00:47:42.779981Z  INFO {farm_index=1}:{sector_index=2974}: subspace_farmer::single_disk_farm::plotting: Plotting sector (39.72% complete)
2024-10-29T00:48:07.787120Z  INFO {farm_index=1}:{sector_index=2975}: subspace_farmer::single_disk_farm::plotting: Plotting sector (39.73% complete)
2024-10-29T00:48:32.793532Z  INFO {farm_index=1}:{sector_index=2976}: subspace_farmer::single_disk_farm::plotting: Plotting sector (39.74% complete)
2024-10-29T00:48:52.798394Z  INFO {farm_index=1}:{sector_index=2977}: subspace_farmer::single_disk_farm::plotting: Plotting sector (39.76% complete)
2024-10-29T00:49:07.082343Z  INFO {farm_index=2}: subspace_farmer::single_disk_farm::reward_signing: Successfully signed reward hash 0x09084afab6c06df6a6f99745ec57699f77215761271de06643eceacdcc0475ca
2024-10-29T01:05:58.637701Z  INFO {farm_index=2}: subspace_farmer::single_disk_farm::reward_signing: Successfully signed reward hash 0x7e925372e1a9ba57fe1dc70dbfaa6e079c8e3fd8aec9f24168ff700bf730ff29
2024-10-29T01:10:33.567281Z  INFO {farm_index=2}: subspace_farmer::single_disk_farm::reward_signing: Successfully signed reward hash 0xd67535ac1c1468674aeda0e53fab21d1b275ee66979b35ebf224a5eba7ad512a
2024-10-29T01:18:21.042458Z  INFO {farm_index=2}: subspace_farmer::single_disk_farm::reward_signing: Successfully signed reward hash 0x15f7dde5e264b16bb3d886cc712d3187da7a1d1dee19302ce21e3a777d52aca7
2024-10-29T01:34:32.465854Z  INFO {farm_index=2}: subspace_farmer::single_disk_farm::reward_signing: Successfully signed reward hash 0xf82cb03a41c902cb456fa30bc1b75542e53dc0021766b0441cfecb51b6369f1a

Machine 2
2024-10-30T05:26:38.514114Z  INFO {farm_index=1}:{sector_index=4429}: subspace_farmer::single_disk_farm::plotting: Plotting sector (59.15% complete)
2024-10-30T05:27:08.521938Z  INFO {farm_index=1}:{sector_index=4430}: subspace_farmer::single_disk_farm::plotting: Plotting sector (59.16% complete)
2024-10-30T05:27:38.529620Z  INFO {farm_index=1}:{sector_index=4431}: subspace_farmer::single_disk_farm::plotting: Plotting sector (59.17% complete)
2024-10-30T05:28:08.604416Z  INFO {farm_index=3}:{sector_index=4564}: subspace_farmer::single_disk_farm::plotting: Plotting sector (60.95% complete)
2024-10-30T05:28:37.637112Z  INFO {farm_index=4}: subspace_farmer::single_disk_farm::reward_signing: Successfully signed reward hash 0x18b8283246a076f6d99279d086e3f9902d9a4f2f1fe3e128e4003c658049810d
2024-10-30T05:28:38.671681Z  INFO {farm_index=0}:{sector_index=4056}: subspace_farmer::single_disk_farm::plotting: Plotting sector (54.17% complete)
2024-10-30T05:29:08.865410Z  INFO {farm_index=0}:{sector_index=4057}: subspace_farmer::single_disk_farm::plotting: Plotting sector (54.18% complete)
2024-10-30T05:29:30.868435Z  INFO {farm_index=0}: subspace_farmer::single_disk_farm::reward_signing: Successfully signed reward hash 0x0c4252da26f5ee5436e6b138c792b513765665c35dbd6b693c1bc723bbc218c8
2024-10-30T05:29:44.155092Z  INFO {farm_index=1}:{sector_index=4432}: subspace_farmer::single_disk_farm::plotting: Plotting sector (59.19% complete)
2024-10-30T05:30:14.161440Z  INFO {farm_index=1}:{sector_index=4433}: subspace_farmer::single_disk_farm::plotting: Plotting sector (59.20% complete)
2024-10-30T05:30:44.260210Z  INFO {farm_index=3}:{sector_index=4565}: subspace_farmer::single_disk_farm::plotting: Plotting sector (60.96% complete)
2024-10-30T05:31:14.573657Z  INFO {farm_index=0}:{sector_index=4058}: subspace_farmer::single_disk_farm::plotting: Plotting sector (54.19% complete)
2024-10-30T05:31:44.581492Z  INFO {farm_index=0}:{sector_index=4059}: subspace_farmer::single_disk_farm::plotting: Plotting sector (54.21% complete)
2024-10-30T05:32:16.588192Z  INFO {farm_index=1}: subspace_farmer::single_disk_farm::reward_signing: Successfully signed reward hash 0xdc7312daa3350ea3b10d1b992fa6c74446c9f8542676b5334d08c484b993e639
2024-10-30T05:33:00.742978Z  INFO {farm_index=3}: subspace_farmer::single_disk_farm::reward_signing: Successfully signed reward hash 0x5d3d1a2dd2daecc49a808ccad0a632133b7773f4393a496db5358a0cfcbb2e04
2024-10-30T05:33:37.878442Z  INFO {farm_index=3}: subspace_farmer::single_disk_farm::reward_signing: Successfully signed reward hash 0x37bd6e0e0c3ab1a60c30bee054ebe2c9ee78b42e2e096aa85bf009988244a9ad
2024-10-30T05:36:25.655358Z  INFO {farm_index=1}: subspace_farmer::single_disk_farm::reward_signing: Successfully signed reward hash 0x129136037963a5f27b133e0352b35704126408bff0fe3aaf705604bca2f65c29
2024-10-30T05:38:03.023615Z  INFO {farm_index=3}: subspace_farmer::single_disk_farm::reward_signing: Successfully signed reward hash 0xdd5786b1677686caa71b81f252c1343bb44a0e4d329fa56f4b5ed5a5aa1582d9
2024-10-30T05:39:16.343183Z  INFO {farm_index=5}: subspace_farmer::single_disk_farm::reward_signing: Successfully signed reward hash 0x9665a8ae12b6ba7d612d25fc21d274e56ccebd85b31d0532bea48fbd9c458900
2024-10-30T05:39:25.382051Z  INFO {farm_index=1}: subspace_farmer::single_disk_farm::reward_signing: Successfully signed reward hash 0x85e8a8a6fe447d6dd6c7a3c15f0d955dd7c89fe01ac597a2fc3c532dcb1aed8e
2024-10-30T05:40:15.553883Z  INFO {farm_index=1}: subspace_farmer::single_disk_farm::reward_signing: Successfully signed reward hash 0xddb0b80596da4c0b1165aa3d87cb0d92deb40690719fd07ea2a9470d1a23ab14
2024-10-30T05:41:29.893563Z  INFO {farm_index=5}: subspace_farmer::single_disk_farm::reward_signing: Successfully signed reward hash 0xc3b15c587ac72e826e1de75e3086861a0e633a306e617cd5a4eaee2596346bdd
2024-10-30T05:45:36.857119Z  INFO {farm_index=3}: subspace_farmer::single_disk_farm::reward_signing: Successfully signed reward hash 0xf3feab979e0fe8bb7d19f99a0c933d5d84803baa5d4681c1e3815ead9178b05b
2024-10-30T05:51:39.369060Z  INFO {farm_index=4}: subspace_farmer::single_disk_farm::reward_signing: Successfully signed reward hash 0x78a68d20b14ff79d228873407dc0d82ccc1c7ff62367c20648052a0dfc263cc3
2024-10-30T05:54:07.014832Z  INFO {farm_index=1}: subspace_farmer::single_disk_farm::reward_signing: Successfully signed reward hash 0xc1441f40818fc5b393a6a994e5985ccc6d437cf68d74d6206505f13f4e9a5578
2024-10-30T05:55:02.228995Z  INFO {farm_index=3}: subspace_farmer::single_disk_farm::reward_signing: Successfully signed reward hash 0xa92abb009f21bf28a4afac4bfd1d7596cc9f1c6cf30559683ea6a7543b66e87c
2024-10-30T05:56:00.422981Z  INFO {farm_index=3}: subspace_farmer::single_disk_farm::reward_signing: Successfully signed reward hash 0x5666c36d93d97bc05a2904c31b2cb84161943530683d770a32ac6abac0384370
2024-10-30T05:57:08.699091Z  INFO {farm_index=1}: subspace_farmer::single_disk_farm::reward_signing: Successfully signed reward hash 0x3a09d3a216e92eb66ff9d5b027952c0fe25315e33d008ce8d7764bc00f06bd25
2024-10-30T05:57:12.712289Z  INFO {farm_index=5}: subspace_farmer::single_disk_farm::reward_signing: Successfully signed reward hash 0x15c5c52bf64ffc5e1a9b60fbc2e739a803671034a3a6d9b77908accc69243477
2024-10-30T06:04:13.541038Z  INFO {farm_index=3}: subspace_farmer::single_disk_farm::reward_signing: Successfully signed reward hash 0xb7e21f99fa6f7515b922b13aaaf8cdc267905a27880dd641b33d9e975782ee12
2024-10-30T06:05:57.919062Z  INFO {farm_index=5}: subspace_farmer::single_disk_farm::reward_signing: Successfully signed reward hash 0xc5f32be35dcbd4ba2d8c006e87ab3574fee0a133eff48fc4ba4306e422b4e4a7
2024-10-30T06:06:20.997559Z  INFO {farm_index=3}: subspace_farmer::single_disk_farm::reward_signing: Successfully signed reward hash 0xeff2025509ba0827346a3df7d605faf2100cdc80c44c1bd73c167724748f0be9
2024-10-30T06:21:33.834375Z  INFO {farm_index=5}: subspace_farmer::single_disk_farm::reward_signing: Successfully signed reward hash 0xecb3348ecab1578f0ca90c8cddd4674e65e5ea9dee625f148b16ba96f782e079
2024-10-30T06:21:55.907309Z  INFO {farm_index=4}: subspace_farmer::single_disk_farm::reward_signing: Successfully signed reward hash 0xfeae6419e4d6ec32ab4c281f37458dd4b908d6d7fe421fa1f45ac385520127a5
2024-10-30T06:22:28.035741Z  INFO {farm_index=5}: subspace_farmer::single_disk_farm::reward_signing: Successfully signed reward hash 0x7cac1c892e5449332dfe8138b99e1689eecb6af8f41aaead172f470925b8095b
2024-10-30T06:23:00.149179Z  INFO {farm_index=5}: subspace_farmer::single_disk_farm::reward_signing: Successfully signed reward hash 0xd1f0b62249289a5820751c342691889f17902b9411972642372f75201b867fa7
2024-10-30T06:29:34.899642Z  INFO {farm_index=1}: subspace_farmer::single_disk_farm::reward_signing: Successfully signed reward hash 0x24fa7e4991c30cd4cfbda3cca7dcce84eb668e80c39394d124ae9e0b10db1283
2

I’m seeing this as well. No errors or warnings on either the farmer or the plotter. It’s like the farmer just stops requesting sectors from the plotter. Restarting the farmer fixes the problem, at least temporarily. I’m running a cluster, and it happens with all my farmers, whether they have a local plotter or not.

ETA: I’m running taurus-2024-oct-30 in docker containers across 4 machines.

3

Can you check your plotter logs please? Do you see the same as in this post? Plotting Stops after Error (taurus-oct-2024)

4

Similarly to Plotting Stops after Error (taurus-oct-2024) - #6 by nazar-pc there are two things I’d like to request.

First to start farmer with RUST_LOG=info,subspace_farmer=trace, which will generate a lot more logs, but should also help with debugging, there could be gigabytes of logs potentially.

Second, it’d help to dump backtraces of threads with gdb when this happens.
Here is how I’d do it:

sudo apt-get install --no-install-recommends gdb
sudo gdb /path/to/subspace-farmer PID_OF_FARMER_PROCESS

Now in gdb type the following:

set pagination off
thead apply all bt

And send me the whole output.

Then you can exit gdb.

5

Now that you say it, I do see a couple of those lines in my plotter logs:

2024-10-31T00:54:18.924935Z  WARN subspace_farmer::cluster::nats_client: Acknowledgement wait timed out response_subject=stream-response.01JBG0ZK51ME4XGG3M5AW26MGT expected_index=6 request_type=subspace_farmer::cluster::plotter::ClusterPlotterPlotSectorRequest response_type=subspace_farmer::cluster::plotter::ClusterSectorPlottingProgress
2024-10-31T00:54:26.728002Z  WARN subspace_farmer::cluster::nats_client: Acknowledgement wait timed out response_subject=stream-response.01JBG0ZXZ3H7B84N5DAKJTAEZ2 expected_index=6 request_type=subspace_farmer::cluster::plotter::ClusterPlotterPlotSectorRequest response_type=subspace_farmer::cluster::plotter::ClusterSectorPlottingProgress

I guess I don’t see them consistently, or I would have posted in the other thread.

I’ll try Nazar’s suggestion tomorrow when I’m around to watch the farmers more closely.

6

I’ve started both farmers back up with the RUST_LOG variable you asked. Will keep an eye out and see what it looks like if it stops plotting again.

I’ll set up the gbd output shortly. Do you want me to just run that randomly or does it need to be done after the farmer stops plotting?

Is there a way to actually upload my log files and the gdb output file? All I seem able to do is paste small sections of the files. I can’t find an upload button or field.

7

No I don’t have any error messages in the log relating to this issue. I’m not on a cluster so I have no NATS server etc, just one node and two separate farmers running simple ACLI startup commands.

8

Gdb only needs to be connected after plotting has stopped, just to see what farmer is waiting for to understand why it stopped plotting.

You can upload to pastebin.com or gist.github.com or anything similar and drop the link to it here.

9

Apologies but I have little idea what I’m doing here. Before including RUST_LOG in my command, I could easily see that the farmer had stopped producing sectors because (as shown in my log extracts in the OP) the log would show a stream of "Plotting sector (xx.xx% complete)) every 5-10 seconds with the occasional “Successfully signed reward hash” entry. When it stopped producing sectors, I would get hours of reward messaged only with no Plotting entries.

With the RUST_LOG turned on, the entries are completely different and I can’t really tell whether it is still plotting sectors or not. I think it is but there are lots of entries saying “Slow farming, skipping slot” which would seem to indicate some sort of issue. An extract from the log shows:

2024-10-31T07:16:36.550790Z DEBUG {farm_index=1}: subspace_farmer::single_disk_farm::farming: New slot slot_info=SlotInfo { slot_number: 654935, global_challenge: a7e835ca52a9a1ddb22a00573414c6d43526c89538ef5bb0f1954f9bd6b7f931, solution_range: 1020039543, voting_solution_range: 10200395430 }
2024-10-31T07:16:36.550794Z DEBUG {farm_index=1}: subspace_farmer::single_disk_farm::farming: Slow farming, skipping slot slot=654935
2024-10-31T07:16:36.550799Z DEBUG {farm_index=0}: subspace_farmer::single_disk_farm::farming: New slot slot_info=SlotInfo { slot_number: 654935, global_challenge: a7e835ca52a9a1ddb22a00573414c6d43526c89538ef5bb0f1954f9bd6b7f931, solution_range: 1020039543, voting_solution_range: 10200395430 }
2024-10-31T07:16:36.550808Z DEBUG {farm_index=4}: subspace_farmer::single_disk_farm::farming: Reading sectors slot=654935 sector_count=3741
2024-10-31T07:16:36.550817Z DEBUG {farm_index=5}: subspace_farmer::single_disk_farm::farming: Reading sectors slot=654935 sector_count=3741
2024-10-31T07:16:36.550824Z DEBUG {farm_index=0}: subspace_farmer::single_disk_farm::farming: Reading sectors slot=654935 sector_count=4771
2024-10-31T07:16:37.556049Z DEBUG {farm_index=2}: subspace_farmer::single_disk_farm::farming: New slot slot_info=SlotInfo { slot_number: 654936, global_challenge: cd4479fde29488be84846c4d8387d392c492cdece7cc0fd250ca86dd04bc7c75, solution_range: 1020039543, voting_solution_range: 10200395430 }
2024-10-31T07:16:37.556097Z DEBUG {farm_index=2}: subspace_farmer::single_disk_farm::farming: Slow farming, skipping slot slot=654936
2024-10-31T07:16:37.556106Z DEBUG {farm_index=1}: subspace_farmer::single_disk_farm::farming: New slot slot_info=SlotInfo { slot_number: 654936, global_challenge: cd4479fde29488be84846c4d8387d392c492cdece7cc0fd250ca86dd04bc7c75, solution_range: 1020039543, voting_solution_range: 10200395430 }
2024-10-31T07:16:37.556109Z DEBUG {farm_index=1}: subspace_farmer::single_disk_farm::farming: Slow farming, skipping slot slot=654936
2024-10-31T07:16:37.556113Z DEBUG {farm_index=4}: subspace_farmer::single_disk_farm::farming: New slot slot_info=SlotInfo { slot_number: 654936, global_challenge: cd4479fde29488be84846c4d8387d392c492cdece7cc0fd250ca86dd04bc7c75, solution_range: 1020039543, voting_solution_range: 10200395430 }
2024-10-31T07:16:37.556120Z DEBUG {farm_index=3}: subspace_farmer::single_disk_farm::farming: New slot slot_info=SlotInfo { slot_number: 654936, global_challenge: cd4479fde29488be84846c4d8387d392c492cdece7cc0fd250ca86dd04bc7c75, solution_range: 1020039543, voting_solution_range: 10200395430 }
2024-10-31T07:16:37.556124Z DEBUG {farm_index=3}: subspace_farmer::single_disk_farm::farming: Slow farming, skipping slot slot=654936
2024-10-31T07:16:37.556128Z DEBUG {farm_index=5}: subspace_farmer::single_disk_farm::farming: New slot slot_info=SlotInfo { slot_number: 654936, global_challenge: cd4479fde29488be84846c4d8387d392c492cdece7cc0fd250ca86dd04bc7c75, solution_range: 1020039543, voting_solution_range: 10200395430 }
2024-10-31T07:16:37.556135Z DEBUG {farm_index=0}: subspace_farmer::single_disk_farm::farming: New slot slot_info=SlotInfo { slot_number: 654936, global_challenge: cd4479fde29488be84846c4d8387d392c492cdece7cc0fd250ca86dd04bc7c75, solution_range: 1020039543, voting_solution_range: 10200395430 }
2024-10-31T07:16:37.556137Z DEBUG {farm_index=4}: subspace_farmer::single_disk_farm::farming: Reading sectors slot=654936 sector_count=3741
2024-10-31T07:16:37.556159Z DEBUG {farm_index=0}: subspace_farmer::single_disk_farm::farming: Reading sectors slot=654936 sector_count=4771
2024-10-31T07:16:37.556160Z DEBUG {farm_index=5}: subspace_farmer::single_disk_farm::farming: Reading sectors slot=654936 sector_count=3741
2024-10-31T07:16:38.311597Z TRACE {farm_index=0}:{sector_index=4771}: subspace_farmer::plotter::pool: All plotters are busy, will wait and try again later retry_interval=5s
2024-10-31T07:16:38.558163Z DEBUG {farm_index=1}: subspace_farmer::single_disk_farm::farming: New slot slot_info=SlotInfo { slot_number: 654937, global_challenge: 2e4dd97cf8c585c9040dabd42b7a10c8ed7a7a72b4d331385732605d74b388f3, solution_range: 1020039543, voting_solution_range: 10200395430 }
2024-10-31T07:16:38.558194Z DEBUG {farm_index=1}: subspace_farmer::single_disk_farm::farming: Slow farming, skipping slot slot=654937
2024-10-31T07:16:38.558200Z DEBUG {farm_index=0}: subspace_farmer::single_disk_farm::farming: New slot slot_info=SlotInfo { slot_number: 654937, global_challenge: 2e4dd97cf8c585c9040dabd42b7a10c8ed7a7a72b4d331385732605d74b388f3, solution_range: 1020039543, voting_solution_range: 10200395430 }
2024-10-31T07:16:38.558209Z DEBUG {farm_index=2}: subspace_farmer::single_disk_farm::farming: New slot slot_info=SlotInfo { slot_number: 654937, global_challenge: 2e4dd97cf8c585c9040dabd42b7a10c8ed7a7a72b4d331385732605d74b388f3, solution_range: 1020039543, voting_solution_range: 10200395430 }
2024-10-31T07:16:38.558213Z DEBUG {farm_index=2}: subspace_farmer::single_disk_farm::farming: Slow farming, skipping slot slot=654937
2024-10-31T07:16:38.558216Z DEBUG {farm_index=3}: subspace_farmer::single_disk_farm::farming: New slot slot_info=SlotInfo { slot_number: 654937, global_challenge: 2e4dd97cf8c585c9040dabd42b7a10c8ed7a7a72b4d331385732605d74b388f3, solution_range: 1020039543, voting_solution_range: 10200395430 }
2024-10-31T07:16:38.558220Z DEBUG {farm_index=3}: subspace_farmer::single_disk_farm::farming: Slow farming, skipping slot slot=654937
2024-10-31T07:16:38.558240Z DEBUG {farm_index=5}: subspace_farmer::single_disk_farm::farming: New slot slot_info=SlotInfo { slot_number: 654937, global_challenge: 2e4dd97cf8c585c9040dabd42b7a10c8ed7a7a72b4d331385732605d74b388f3, solution_range: 1020039543, voting_solution_range: 10200395430 }
2024-10-31T07:16:38.558241Z DEBUG {farm_index=0}: subspace_farmer::single_disk_farm::farming: Reading sectors slot=654937 sector_count=4771
2024-10-31T07:16:38.558248Z DEBUG {farm_index=4}: subspace_farmer::single_disk_farm::farming: New slot slot_info=SlotInfo { slot_number: 654937, global_challenge: 2e4dd97cf8c585c9040dabd42b7a10c8ed7a7a72b4d331385732605d74b388f3, solution_range: 1020039543, voting_solution_range: 10200395430 }
2024-10-31T07:16:38.558277Z DEBUG {farm_index=5}: subspace_farmer::single_disk_farm::farming: Reading sectors slot=654937 sector_count=3741
2024-10-31T07:16:38.558283Z DEBUG {farm_index=4}: subspace_farmer::single_disk_farm::farming: Reading sectors slot=654937 sector_count=3741
2024-10-31T07:16:39.569159Z DEBUG {farm_index=3}: subspace_farmer::single_disk_farm::farming: New slot slot_info=SlotInfo { slot_number: 654938, global_challenge: 10005ec9ecb1541ff0d6b1f27b52c2e46fbbe442a237c5e82b3ebe1680dc88f1, solution_range: 1020039543, voting_solution_range: 10200395430 }
2024-10-31T07:16:39.569190Z DEBUG {farm_index=3}: subspace_farmer::single_disk_farm::farming: Slow farming, skipping slot slot=654938
2024-10-31T07:16:39.569197Z DEBUG {farm_index=5}: subspace_farmer::single_disk_farm::farming: New slot slot_info=SlotInfo { slot_number: 654938, global_challenge: 10005ec9ecb1541ff0d6b1f27b52c2e46fbbe442a237c5e82b3ebe1680dc88f1, solution_range: 1020039543, voting_solution_range: 10200395430 }

I’m going to run the gdb trace and try to upload the full log and gdb trace for you to look at and let me know what you think.

10

My previous message has been hidden pending review but I’ve managed to do the gdb trace and the results are here (I think):

<script src="https://gist.github.com/Badfish1/e22eba20c59622eefd126baacafe8393.js"></script>

I can’t upload the farmer log, it keeps giving me an error that it timed out so I think it might be too large - it went from 4Mb to 61 Mb once I turned on the RUST variable.

11

I was unable to install gdb in the docker farmer container. I got the following:

# apt-get install --no-install-recommends gdb
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
E: Unable to locate package gdb

I’m adding the RUST_LOG variable to my compose file now and will upload the results next time the farmer hangs.

12

Right, apt-get update should have been used first.

Either way I think I located the issue and Snapshot build · autonomys/subspace@9d3ba5d · GitHub should fix it, please give it a try once it is built and let me know how it goes.

13

Thanks! I updated all my containers and will let you know how it goes.

1 Like
14

Fix was shipped in Release taurus-2024-nov-01 · autonomys/subspace · GitHub

2 Likes