Cluster Farm plotter/farmer panic

Support
1

Issue Report

Environment

  • Linux Mint 21.03
  • Snapshot build #344

Problem

Running Snapshot Build #344, farmer and plotter ran fine for around 12 hours, but then this happened in the farmer log:

2024-06-10T01:08:55.475342Z  INFO {farm_index=6}:{sector_index=3581}: subspace_farmer::single_disk_farm::plotting: Plotting sector (99.64% complete)
2024-06-10T01:09:15.322013Z  INFO {farm_index=16}: subspace_farmer::reward_signing: Successfully signed reward hash 0x48690ec9bae25b2a401e2714401bdfec9374cf262902003213404943ae0c3a09
thread 'thread 'thread 'thread '<unnamed>thread '<unnamed><unnamed>thread 'thread 'thread 'thread '<unnamed>' panicked at thread 'thread '' panicked at thread 'thread 'thread 'thread '<unnamed><unnamed>' panicked at ' panicked at thread '<unnamed>thread '<unnamed>thread 'thread '<unnamed>thread 'thread '<unnamed>/home/ubuntu/actions-runner/_work/subspace/subspace/crates/subspace-farmer-components/src/reading.rsthread '<unnamed>thread '<unnamed>/home/ubuntu/actions-runner/_work/subspace/subspace/crates/subspace-farmer-components/src/reading.rsthread 'thread 'thread '<unnamed><unnamed><unnamed>' panicked at ' panicked at /home/ubuntu/actions-runner/_work/subspace/subspace/crates/subspace-farmer-components/src/reading.rs/home/ubuntu/actions-runner/_work/subspace/subspace/crates/subspace-farmer-components/src/reading.rs<unnamed>' panicked at <unnamed>' panicked at <unnamed><unnamed>' panicked at <unnamed><unnamed>' panicked at :<unnamed>' panicked at <unnamed>' panicked at :<unnamed><unnamed><unnamed>' panicked at ' panicked at ' panicked at /home/ubuntu/actions-runner/_work/subspace/subspace/crates/subspace-farmer-components/src/reading.rs/home/ubuntu/actions-runner/_work/subspace/subspace/crates/subspace-farmer-components/src/reading.rs::' panicked at /home/ubuntu/actions-runner/_work/subspace/subspace/crates/subspace-farmer-components/src/reading.rs' panicked at /home/ubuntu/actions-runner/_work/subspace/subspace/crates/subspace-farmer-components/src/reading.rs' panicked at ' panicked at /home/ubuntu/actions-runner/_work/subspace/subspace/crates/subspace-farmer-components/src/reading.rs' panicked at ' panicked at /home/ubuntu/actions-runner/_work/subspace/subspace/crates/subspace-farmer-components/src/reading.rs222' panicked at /home/ubuntu/actions-runner/_work/subspace/subspace/crates/subspace-farmer-components/src/reading.rs' panicked at /home/ubuntu/actions-runner/_work/subspace/subspace/crates/subspace-farmer-components/src/reading.rs222' panicked at ' panicked at ' panicked at /home/ubuntu/actions-runner/_work/subspace/subspace/crates/subspace-farmer-components/src/reading.rs/home/ubuntu/actions-runner/_work/subspace/subspace/crates/subspace-farmer-components/src/reading.rs/home/ubuntu/actions-runner/_work/subspace/subspace/crates/subspace-farmer-components/src/reading.rs::222222/home/ubuntu/actions-runner/_work/subspace/subspace/crates/subspace-farmer-components/src/reading.rs:/home/ubuntu/actions-runner/_work/subspace/subspace/crates/subspace-farmer-components/src/reading.rs:/home/ubuntu/actions-runner/_work/subspace/subspace/crates/subspace-farmer-components/src/reading.rs/home/ubuntu/actions-runner/_work/subspace/subspace/crates/subspace-farmer-components/src/reading.rs:/home/ubuntu/actions-runner/_work/subspace/subspace/crates/subspace-farmer-components/src/reading.rs/home/ubuntu/actions-runner/_work/subspace/subspace/crates/subspace-farmer-components/src/reading.rs::/home/ubuntu/actions-runner/_work/subspace/subspace/crates/subspace-farmer-components/src/reading.rs:/home/ubuntu/actions-runner/_work/subspace/subspace/crates/subspace-farmer-components/src/reading.rs::/home/ubuntu/actions-runner/_work/subspace/subspace/crates/subspace-farmer-components/src/reading.rs/home/ubuntu/actions-runner/_work/subspace/subspace/crates/subspace-farmer-components/src/reading.rs/home/ubuntu/actions-runner/_work/subspace/subspace/crates/subspace-farmer-components/src/reading.rs:::222222:::222:222::222::22230:222:22230:::222222222::3030222:222:222222:222222::
222:222::
222222222:::3030:
:
:30:30::30::30encoded_chunk_used implies proof exists for this chunk; qed:30:30encoded_chunk_used implies proof exists for this chunk; qed:::303030:
:
encoded_chunk_used implies proof exists for this chunk; qedencoded_chunk_used implies proof exists for this chunk; qed30:
30:
3030:
3030:

30:
30:

303030:
:
:
encoded_chunk_used implies proof exists for this chunk; qedencoded_chunk_used implies proof exists for this chunk; qed

:
encoded_chunk_used implies proof exists for this chunk; qed:
encoded_chunk_used implies proof exists for this chunk; qed:
:
encoded_chunk_used implies proof exists for this chunk; qed:
:
encoded_chunk_used implies proof exists for this chunk; qednote: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
:
encoded_chunk_used implies proof exists for this chunk; qed:
encoded_chunk_used implies proof exists for this chunk; qed:
:
:
encoded_chunk_used implies proof exists for this chunk; qedencoded_chunk_used implies proof exists for this chunk; qedencoded_chunk_used implies proof exists for this chunk; qed

encoded_chunk_used implies proof exists for this chunk; qed
encoded_chunk_used implies proof exists for this chunk; qed
encoded_chunk_used implies proof exists for this chunk; qedencoded_chunk_used implies proof exists for this chunk; qed
encoded_chunk_used implies proof exists for this chunk; qedencoded_chunk_used implies proof exists for this chunk; qed
encoded_chunk_used implies proof exists for this chunk; qed
encoded_chunk_used implies proof exists for this chunk; qed
encoded_chunk_used implies proof exists for this chunk; qedencoded_chunk_used implies proof exists for this chunk; qedencoded_chunk_used implies proof exists for this chunk; qed

This repeats several times, interspersed with messages showing many, but not every, farm errored and stopped.

2024-06-10T01:13:00.873236Z ERROR subspace_farmer::commands::cluster::farmer: Farm errored and stopped farm_index=11 error=Background task reading-11 panicked
2024-06-10T01:13:02.360908Z ERROR subspace_farmer::commands::cluster::farmer: Farm errored and stopped farm_index=19 error=Background task reading-19 panicked
2024-06-10T01:13:02.587607Z ERROR subspace_farmer::commands::cluster::farmer: Farm errored and stopped farm_index=8 error=Background task reading-8 panicked
2024-06-10T01:13:11.337165Z ERROR subspace_farmer::commands::cluster::farmer: Farm errored and stopped farm_index=20 error=Background task reading-20 panicked
2024-06-10T01:13:12.132915Z ERROR subspace_farmer::commands::cluster::farmer: Farm errored and stopped farm_index=2 error=Background task reading-2 panicked
2024-06-10T01:13:30.875186Z ERROR subspace_farmer::commands::cluster::farmer: Farm errored and stopped farm_index=11 error=Background task reading-11 panicked
2024-06-10T01:13:32.362616Z ERROR subspace_farmer::commands::cluster::farmer: Farm errored and stopped farm_index=19 error=Background task reading-19 panicked
2024-06-10T01:13:32.588622Z ERROR subspace_farmer::commands::cluster::farmer: Farm errored and stopped farm_index=8 error=Background task reading-8 panicked
2024-06-10T01:13:41.338968Z ERROR subspace_farmer::commands::cluster::farmer: Farm errored and stopped farm_index=20 error=Background task reading-20 panicked
2024-06-10T01:13:42.133655Z ERROR subspace_farmer::commands::cluster::farmer: Farm errored and stopped farm_index=2 error=Background task reading-2 panicked
2024-06-10T01:14:00.876219Z ERROR subspace_farmer::commands::cluster::farmer: Farm errored and stopped farm_index=11 error=Background task reading-11 panicked
2024-06-10T01:14:02.363602Z ERROR subspace_farmer::commands::cluster::farmer: Farm errored and stopped farm_index=19 error=Background task reading-19 panicked
2024-06-10T01:14:02.589822Z ERROR subspace_farmer::commands::cluster::farmer: Farm errored and stopped farm_index=8 error=Background task reading-8 panicked
2024-06-10T01:14:11.340271Z ERROR subspace_farmer::commands::cluster::farmer: Farm errored and stopped farm_index=20 error=Background task reading-20 panicked
2024-06-10T01:14:12.134495Z ERROR subspace_farmer::commands::cluster::farmer: Farm errored and stopped farm_index=2 error=Background task reading-2 panicked
2024-06-10T01:14:30.877707Z ERROR subspace_farmer::commands::cluster::farmer: Farm errored and stopped farm_index=11 error=Background task reading-11 panicked
2024-06-10T01:14:32.364965Z ERROR subspace_farmer::commands::cluster::farmer: Farm errored and stopped farm_index=19 error=Background task reading-19 panicked
2024-06-10T01:14:32.590260Z ERROR subspace_farmer::commands::cluster::farmer: Farm errored and stopped farm_index=8 error=Background task reading-8 panicked
2024-06-10T01:14:41.341829Z ERROR subspace_farmer::commands::cluster::farmer: Farm errored and stopped farm_index=20 error=Background task reading-20 panicked
2024-06-10T01:14:42.135935Z ERROR subspace_farmer::commands::cluster::farmer: Farm errored and stopped farm_index=2 error=Background task reading-2 panicked
2024-06-10T01:15:00.878207Z ERROR subspace_farmer::commands::cluster::farmer: Farm errored and stopped farm_index=11 error=Background task reading-11 panicked
2024-06-10T01:15:02.366533Z ERROR subspace_farmer::commands::cluster::farmer: Farm errored and stopped farm_index=19 error=Background task reading-19 panicked
2024-06-10T01:15:02.591602Z ERROR subspace_farmer::commands::cluster::farmer: Farm errored and stopped farm_index=8 error=Background task reading-8 panicked
2024-06-10T01:15:07.534845Z  INFO {farm_index=0}: subspace_farmer::reward_signing: Successfully signed reward hash 0x72dcaaf8c39bc85b7abc6f61f797566cba934f0b2da3d65856e112d0a64f148c
2024-06-10T01:15:11.342923Z ERROR subspace_farmer::commands::cluster::farmer: Farm errored and stopped farm_index=20 error=Background task reading-20 panicked
2024-06-10T01:15:12.137914Z ERROR subspace_farmer::commands::cluster::farmer: Farm errored and stopped farm_index=2 error=Background task reading-2 panicked
2024-06-10T01:15:30.879886Z ERROR subspace_farmer::commands::cluster::farmer: Farm errored and stopped farm_index=11 error=Background task reading-11 panicked
2024-06-10T01:15:32.367286Z ERROR subspace_farmer::commands::cluster::farmer: Farm errored and stopped farm_index=19 error=Background task reading-19 panicked
2024-06-10T01:15:32.592291Z ERROR subspace_farmer::commands::cluster::farmer: Farm errored and stopped farm_index=8 error=Background task reading-8 panicked
2024-06-10T01:15:41.343124Z ERROR subspace_farmer::commands::cluster::farmer: Farm errored and stopped farm_index=20 error=Background task reading-20 panicked
2024-06-10T01:15:42.139011Z ERROR subspace_farmer::commands::cluster::farmer: Farm errored and stopped farm_index=2 error=Background task reading-2 panicked
2024-06-10T01:16:00.881574Z ERROR subspace_farmer::commands::cluster::farmer: Farm errored and stopped farm_index=11 error=Background task reading-11 panicked
2024-06-10T01:16:02.368494Z ERROR subspace_farmer::commands::cluster::farmer: Farm errored and stopped farm_index=19 error=Background task reading-19 panicked
2024-06-10T01:16:02.593476Z ERROR subspace_farmer::commands::cluster::farmer: Farm errored and stopped farm_index=8 error=Background task reading-8 panicked
2024-06-10T01:16:11.344520Z ERROR subspace_farmer::commands::cluster::farmer: Farm errored and stopped farm_index=20 error=Background task reading-20 panicked
2024-06-10T01:16:12.140110Z ERROR subspace_farmer::commands::cluster::farmer: Farm errored and stopped farm_index=2 error=Background task reading-2 panicked
2024-06-10T01:16:30.882631Z ERROR subspace_farmer::commands::cluster::farmer: Farm errored and stopped farm_index=11 error=Background task reading-11 panicked
2024-06-10T01:16:32.370140Z ERROR subspace_farmer::commands::cluster::farmer: Farm errored and stopped farm_index=19 error=Background task reading-19 panicked
2024-06-10T01:16:32.594172Z ERROR subspace_farmer::commands::cluster::farmer: Farm errored and stopped farm_index=8 error=Background task reading-8 panicked
2024-06-10T01:16:41.345086Z ERROR subspace_farmer::commands::cluster::farmer: Farm errored and stopped farm_index=20 error=Background task reading-20 panicked
2024-06-10T01:16:42.141801Z ERROR subspace_farmer::commands::cluster::farmer: Farm errored and stopped farm_index=2 error=Background task reading-2 panicked

Note that at this point I was not farming any remote plots. This was on a threadripper with 512GB RAM farming 9 local 15.36TB U.2 drives. For various unimportant reasons I was not attempting to connect remote farms to it. So LAN bandwidth issues wouldn’t be a cause.

I stopped and restart the farm and plotter. Farming seems to be okay, but the plotting will not resume normal operation. Normally this computer would farm 4 plots in 8 minutes under its current load and --plotting-thread-pool-size set to 8, to only use 50% CPU at the moment. Here you see it taking over an hour and not properly finishing more than one set of sectors.

2024-06-10T01:36:47.697203Z  INFO {farm_index=28}: subspace_farmer::reward_signing: Subscribing to reward signing notifications
2024-06-10T01:37:15.972999Z  INFO {farm_index=22}:{sector_index=3584}: subspace_farmer::single_disk_farm::plotting: Plotting sector (99.72% complete)
2024-06-10T01:37:15.977241Z  INFO {farm_index=22}:{sector_index=3585}: subspace_farmer::single_disk_farm::plotting: Plotting sector (99.75% complete)
2024-06-10T01:37:15.980352Z  INFO {farm_index=22}:{sector_index=3586}: subspace_farmer::single_disk_farm::plotting: Plotting sector (99.78% complete)
2024-06-10T01:37:15.981108Z  INFO {farm_index=22}:{sector_index=3587}: subspace_farmer::single_disk_farm::plotting: Plotting sector (99.81% complete)
2024-06-10T01:37:15.981629Z  INFO {farm_index=22}:{sector_index=3588}: subspace_farmer::single_disk_farm::plotting: Plotting sector (99.83% complete)
2024-06-10T01:57:02.628793Z  INFO {farm_index=16}: subspace_farmer::reward_signing: Successfully signed reward hash 0xc436c1afc00d599bfe4af572d535d7699ac9cca4f3a9e92dacde8856c1d98456
2024-06-10T01:57:57.610233Z  INFO {farm_index=21}: subspace_farmer::reward_signing: Successfully signed reward hash 0xf3216d3c74771a712539bac14083e3f832c80ef317e583502b49ea75e435d449
2024-06-10T02:00:00.660483Z  INFO {farm_index=27}: subspace_farmer::reward_signing: Successfully signed reward hash 0xde585463a4ac78b7e45216f4da25dd55ecd983b8d32f90ce1e13ad14c2017993
2024-06-10T02:02:03.689117Z  INFO {farm_index=18}:{sector_index=3576}: subspace_farmer::single_disk_farm::plotting: Plotting sector (99.50% complete)
2024-06-10T02:02:03.693960Z  INFO {farm_index=18}:{sector_index=3577}: subspace_farmer::single_disk_farm::plotting: Plotting sector (99.53% complete)
2024-06-10T02:02:03.702234Z  INFO {farm_index=18}:{sector_index=3578}: subspace_farmer::single_disk_farm::plotting: Plotting sector (99.55% complete)
2024-06-10T02:02:03.710421Z  INFO {farm_index=18}:{sector_index=3579}: subspace_farmer::single_disk_farm::plotting: Plotting sector (99.58% complete)
2024-06-10T02:02:03.718307Z  INFO {farm_index=18}:{sector_index=3580}: subspace_farmer::single_disk_farm::plotting: Plotting sector (99.61% complete)
2024-06-10T02:02:25.979595Z  WARN {farm_index=22}:{sector_index=3585}: subspace_farmer::single_disk_farm::plotting: Failed to plot sector, retrying in 1s error=Low-level plotting error: Timed out without ping from plotter
2024-06-10T02:02:25.982758Z  WARN {farm_index=22}:{sector_index=3586}: subspace_farmer::single_disk_farm::plotting: Failed to plot sector, retrying in 1s error=Low-level plotting error: Timed out without ping from plotter
2024-06-10T02:02:25.983918Z  WARN {farm_index=22}:{sector_index=3587}: subspace_farmer::single_disk_farm::plotting: Failed to plot sector, retrying in 1s error=Low-level plotting error: Timed out without ping from plotter
2024-06-10T02:02:25.983969Z  WARN {farm_index=22}:{sector_index=3588}: subspace_farmer::single_disk_farm::plotting: Failed to plot sector, retrying in 1s error=Low-level plotting error: Timed out without ping from plotter
2024-06-10T02:02:25.988062Z  WARN {farm_index=22}:{sector_index=3584}: subspace_farmer::single_disk_farm::plotting: Failed to plot sector, retrying in 1s error=Low-level plotting error: Timed out without ping from plotter
2024-06-10T02:05:42.695127Z  INFO {farm_index=2}: subspace_farmer::reward_signing: Successfully signed reward hash 0x1301b2cb6ff523c4f548462429e597f67ca2ed3874dcf9d1e28ab07cfa686ccb
2024-06-10T02:09:29.654573Z  INFO {farm_index=25}: subspace_farmer::reward_signing: Successfully signed reward hash 0xf1bbf395a4687fd07c862a54a9fbd786888f062d61e8a1f12ccb98e0d98d3e26
2024-06-10T02:09:34.757776Z  INFO {farm_index=24}: subspace_farmer::reward_signing: Successfully signed reward hash 0x734ba618aedacdceeda487eb1551ed041a947efaa58b1c5c781c536674139ba1
2024-06-10T02:26:06.780974Z  INFO {farm_index=2}: subspace_farmer::reward_signing: Successfully signed reward hash 0xbf7a4c43519e37340c9f3c035928b2ed4e870156484db72276b908190a95843c
2024-06-10T02:35:53.399883Z  WARN {farm_index=18}:{sector_index=3580}: subspace_farmer::single_disk_farm::plotting: Failed to plot sector, retrying in 1s error=Low-level plotting error: Timed out without ping from plotter
2024-06-10T02:35:53.400678Z  WARN {farm_index=18}:{sector_index=3578}: subspace_farmer::single_disk_farm::plotting: Failed to plot sector, retrying in 1s error=Low-level plotting error: Timed out without ping from plotter
2024-06-10T02:35:53.400715Z  WARN {farm_index=18}:{sector_index=3576}: subspace_farmer::single_disk_farm::plotting: Failed to plot sector, retrying in 1s error=Low-level plotting error: Timed out without ping from plotter
2024-06-10T02:35:53.400725Z  WARN {farm_index=18}:{sector_index=3579}: subspace_farmer::single_disk_farm::plotting: Failed to plot sector, retrying in 1s error=Low-level plotting error: Timed out without ping from plotter
2024-06-10T02:35:53.400733Z  WARN {farm_index=18}:{sector_index=3577}: subspace_farmer::single_disk_farm::plotting: Failed to plot sector, retrying in 1s error=Low-level plotting error: Timed out without ping from plotter

(The last set of Failed to Plot Sector messages there were due to me control-c’ing the plotter, since it clearly wasn’t doing enough to justify continuing to run it at this point in time.)

This is the most detail I can provide at the moment, many distractions atm, sorry.

2

This looks like a hardware issue to me unless you were running gemini-3h-2024-may-23 release, which produces unreadable sectors.

3

I never ran the may 23 release. And this box has been supremely stable. I was even running it in Efficiency Mode, so it’s the opposite of overclocked.

So it would pretty much have to be drives, or the PCIe-U.2 adapters the drives are on. It wouldn’t be shocking to find out the latter are problematic. I’ll try to figure out if all the affected farms were on the same adapter.

If I leave the plotter running long enough once that is resolved, any chance it’ll ever revert to normal operation/speed?

4

Undervolting and similar derivation from the spec can also result in instability, there was at least one person in Discord with an issue related to undervolting (though if you limit it by power, it might be different). Also this is often caused by memory being unstable, not CPU.

It is unlikely that drives would result in such issues without anything in kernel logs. If you run scrubbing, does it see an issue with this farm?

I made a note to look into handling such issue without panicking completely, the severity if a bit too extreme for this situation, however unexpected it is.

5

It wasn’t any actual undervolting, just a defined power profile option in the BIOS (the other options being High Performance Mode and, intriguingly, “Maximum IO Performance Mode”, tho I haven’t tried that one yet.

Unfortunately I did a reboot for unrelated reasons after the issue happened, so that time frame isn’t showing up in the conveniently accessible kernel logs. But I’ll keep digging for them.

It wasn’t just one farm though, it was multiple, but not all of them. Per the middle bit of log I posted, it appears to have been farms 2, 8, 11, 19, and 20. Which now that I look at the sequence is pretty odd and makes it unlikely to have been an adapter issue, since that would be on at least 4 different drives, but only one or two affected plots on each (these drives have 4 plots of 3.8TB each). There’s a total of 9 15.36TB drives involved, with 29 plots.

Very weird.

6

IIRC Ubuntu doesn’t preserve previous boot logs by default, at least it wasn’t the case at some point in the past.

7

Consider missing PoS proof errors as non-fatal by nazar-pc · Pull Request #2838 · subspace/subspace · GitHub will change this to non-fatal error instead.

8

Thank you!

Any idea what might make subsequent restarts of the plotter take up to 7x longer than normal?

I haven’t let it run much longer than an hour following that, as I am running some other important things atm and couldn’t sacrifice that much power for no perceptible gain.

9

There are a few reports of this in last hours, which makes me think it is a regression in test builds. Can you try running a stable release and see if it crashes in a similar way? Also I’m curious if you see successfully signed rewards with this experimental build.

Answers will help to narrow down the issue.

10

I read through the changes that were included into build #344 comparing to gemini-3h-2024-may-24 and have not found anything obviously problematic so far. This doesn’t mean there isn’t anything, but I do not see anything yet.

The error is about some sector details on disk not matching reality leading to farmer trying to generate a proof and not finding it at the offset where it is supposed to exist.

11

Hi Nazar,

I did farm and successfully signed hashes for quite a few hours on 344 yesterday with no issues. The error that I posted didn’t happen until I had been both farming and plotting for more than 12 hours I believe.

Unfortunately at the moment I cannot run the plotter for 12+ hours hoping to reproduce the error. It’s a long story. My apologies. When my system is freed up from testing other things I will try to get it working again with the main release and will let you know.

12

Thanks. I’m just worried this might be a regression, in which case I’d like to get as many details as possible so I can fix it.

13

So I’ve been running just the cluster farmer on jun-11 for around 15 hours or so with no issues. Just farming seems to be fine. So I figured I’d give the plotter another try. Immediate errors:

2024-06-12T14:38:56.469181Z  INFO {public_key=8e93a8555afe0a0aecd75a72f459de29616218d5345ed52f41b64f71b3fdaf41 sector_index=3580}: subspace_farmer::cluster::plotter: Plot sector request
2024-06-12T14:38:56.469660Z  INFO {public_key=8e93a8555afe0a0aecd75a72f459de29616218d5345ed52f41b64f71b3fdaf41 sector_index=3581}: subspace_farmer::cluster::plotter: Plot sector request
thread 'plotting-2.3' panicked at /home/ubuntu/actions-runner/_work/subspace/subspace/crates/subspace-farmer-components/src/plotting.rs:526:44:
Piece getter must returns valid pieces of history that contain proper scalar bytes; qed: "Invalid scalar"
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
thread 'plotting-3.2' panicked at /home/ubuntu/actions-runner/_work/subspace/subspace/crates/subspace-farmer-components/src/plotting.rs:526:44:
Piece getter must returns valid pieces of history that contain proper scalar bytes; qed: "Invalid scalar"
thread 'plotting-3.1' panicked at /home/ubuntu/actions-runner/_work/subspace/subspace/crates/subspace-farmer-components/src/plotting.rs:526:44:
Piece getter must returns valid pieces of history that contain proper scalar bytes; qed: "Invalid scalar"
thread 'plotting-3.0' panicked at /home/ubuntu/actions-runner/_work/subspace/subspace/crates/subspace-farmer-components/src/plotting.rs:526:44:
Piece getter must returns valid pieces of history that contain proper scalar bytes; qed: "Invalid scalar"
thread 'plotting-3.4' panicked at /home/ubuntu/actions-runner/_work/subspace/subspace/crates/subspace-farmer-components/src/plotting.rs:526:44:
Piece getter must returns valid pieces of history that contain proper scalar bytes; qed: "Invalid scalar"
thread 'plotting-2.1' panicked at /home/ubuntu/actions-runner/_work/subspace/subspace/crates/subspace-farmer-components/src/plotting.rs:526:44:
Piece getter must returns valid pieces of history that contain proper scalar bytes; qed: "Invalid scalar"
thread 'plotting-2.0' panicked at /home/ubuntu/actions-runner/_work/subspace/subspace/crates/subspace-farmer-components/src/plotting.rs:526:44:
Piece getter must returns valid pieces of history that contain proper scalar bytes; qed: "Invalid scalar"
thread 'plotting-2.2' panicked at /home/ubuntu/actions-runner/_work/subspace/subspace/crates/subspace-farmer-components/src/plotting.rs:526:44:
Piece getter must returns valid pieces of history that contain proper scalar bytes; qed: "Invalid scalar"
2024-06-12T14:39:10.172046Z  INFO {public_key=3cc3aca01a650d67abb7b38885864db50a280dcc7a59b016426a08ab4cf8ec04 sector_index=3576}: subspace_farmer::cluster::plotter: Plot sector request
2024-06-12T14:39:10.176644Z  INFO {public_key=3cc3aca01a650d67abb7b38885864db50a280dcc7a59b016426a08ab4cf8ec04 sector_index=3577}: subspace_farmer::cluster::plotter: Plot sector request
thread 'plotting-2.1' panicked at /home/ubuntu/actions-runner/_work/subspace/subspace/crates/subspace-farmer-components/src/plotting.rs:526:44:
Piece getter must returns valid pieces of history that contain proper scalar bytes; qed: "Invalid scalar"
thread 'plotting-2.0' panicked at /home/ubuntu/actions-runner/_work/subspace/subspace/crates/subspace-farmer-components/src/plotting.rs:526:44:
Piece getter must returns valid pieces of history that contain proper scalar bytes; qed: "Invalid scalar"
thread 'plotting-2.2' panicked at /home/ubuntu/actions-runner/_work/subspace/subspace/crates/subspace-farmer-components/src/plotting.rs:526:44:
Piece getter must returns valid pieces of history that contain proper scalar bytes; qed: "Invalid scalar"
thread 'plotting-2.3' panicked at /home/ubuntu/actions-runner/_work/subspace/subspace/crates/subspace-farmer-components/src/plotting.rs:526:44:
Piece getter must returns valid pieces of history that contain proper scalar bytes; qed: "Invalid scalar"
2024-06-12T14:39:15.195526Z  INFO {public_key=10759201c55c1e79f60d5c241f780d2fc5bc79b5a583de4753c4310a97dd6e34 sector_index=3580}: subspace_farmer::cluster::plotter: Plot sector request
thread 'plotting-2.4' panicked at /home/ubuntu/actions-runner/_work/subspace/subspace/crates/subspace-farmer-components/src/plotting.rs:526:44:
Piece getter must returns valid pieces of history that contain proper scalar bytes; qed: "Invalid scalar"
thread 'plotting-2.1' panicked at /home/ubuntu/actions-runner/_work/subspace/subspace/crates/subspace-farmer-components/src/plotting.rs:526:44:
Piece getter must returns valid pieces of history that contain proper scalar bytes; qed: "Invalid scalar"
thread 'plotting-2.2' panicked at /home/ubuntu/actions-runner/_work/subspace/subspace/crates/subspace-farmer-components/src/plotting.rs:526:44:
Piece getter must returns valid pieces of history that contain proper scalar bytes; qed: "Invalid scalar"
thread 'plotting-2.0' panicked at /home/ubuntu/actions-runner/_work/subspace/subspace/crates/subspace-farmer-components/src/plotting.rs:526:44:
Piece getter must returns valid pieces of history that contain proper scalar bytes; qed: "Invalid scalar"
2024-06-12T14:39:19.437972Z  INFO {public_key=10759201c55c1e79f60d5c241f780d2fc5bc79b5a583de4753c4310a97dd6e34 sector_index=3581}: subspace_farmer::cluster::plotter: Plot sector request
thread 'plotting-2.3' panicked at /home/ubuntu/actions-runner/_work/subspace/subspace/crates/subspace-farmer-components/src/plotting.rs:526:44:
Piece getter must returns valid pieces of history that contain proper scalar bytes; qed: "Invalid scalar"
thread 'plotting-2.0' panicked at /home/ubuntu/actions-runner/_work/subspace/subspace/crates/subspace-farmer-components/src/plotting.rs:526:44:
Piece getter must returns valid pieces of history that contain proper scalar bytes; qed: "Invalid scalar"
thread 'plotting-2.4' panicked at /home/ubuntu/actions-runner/_work/subspace/subspace/crates/subspace-farmer-components/src/plotting.rs:526:44:
Piece getter must returns valid pieces of history that contain proper scalar bytes; qed: "Invalid scalar"
thread 'plotting-2.1' panicked at /home/ubuntu/actions-runner/_work/subspace/subspace/crates/subspace-farmer-components/src/plotting.rs:526:44:
Piece getter must returns valid pieces of history that contain proper scalar bytes; qed: "Invalid scalar"
2024-06-12T14:39:23.825842Z  INFO {public_key=10759201c55c1e79f60d5c241f780d2fc5bc79b5a583de4753c4310a97dd6e34 sector_index=3582}: subspace_farmer::cluster::plotter: Plot sector request
2024-06-12T14:39:27.841231Z  INFO subspace_farmer::utils: Received SIGINT, shutting down farmer...
thread 'plotting-2.0' panicked at /home/ubuntu/actions-runner/_work/subspace/subspace/crates/subspace-farmer-components/src/plotting.rs:526:44:
Piece getter must returns valid pieces of history that contain proper scalar bytes; qed: "Invalid scalar"
thread 'plotting-2.1' panicked at /home/ubuntu/actions-runner/_work/subspace/subspace/crates/subspace-farmer-components/src/plotting.rs:526:44:
Piece getter must returns valid pieces of history that contain proper scalar bytes; qed: "Invalid scalar"
thread 'plotting-2.7' panicked at /home/ubuntu/actions-runner/_work/subspace/subspace/crates/subspace-farmer-components/src/plotting.rs:526:44:
Piece getter must returns valid pieces of history that contain proper scalar bytes; qed: "Invalid scalar"
thread 'plotting-2.2' panicked at /home/ubuntu/actions-runner/_work/subspace/subspace/crates/subspace-farmer-components/src/plotting.rs:526:44:
Piece getter must returns valid pieces of history that contain proper scalar bytes; qed: "Invalid scalar"

So I’ve shut it down again.

Note, btw, that I do not have a /home/ubuntu directory. Then again, I don’t have a /home/qwinn/actions-runner/ directory either, so.

14

The paths with code are generated by build system, so they correspond to whatever server that was building the app had.

So you are using farming cluster, did you upgrade all components or just some of them?

There were breaking changes in the last release (I probably should have mentioned that in release notes) and it may result in such issues indeed.

15

All components are using the jun-11 release farmer executable.

And yes, I’m using cluster, hence the title of this thread.

So you’re saying #344 broke my stuff? Do I have any options to fix it? I mean, I’m fine with just farming what I already have plotted for the remainder of 3h, I didn’t have that far to go, tho hopefully I don’t start expiring in large quantities before its over.

16

I do not think it should have broken your stuff, at least I do not see how it possibly could. I think you may have an issue with cache specifically, can you try to rename the piece cache file and let it sync again to see if it helps?

17

Eh, sure. New sync started. I assume this will be a few hours.

I forget, can I keep successfully farming while this is going on? Or will it not farm until cache sync is done?

18

Cache sync should not impact farming in any way

19

Ok cool. Farming again then. At current rate looks like my TR will finish syncing the cache in about 63 hours (I set 200 gig cache per instructions). When that’s done I’ll try firing up the plotter again and report back.

20

Thanks, appreciate all the testing and reporting!