What is the reason for my reward so little

Support
,
1

What is the reason for my reward so little

Environment

  • Operating System:ubuntu 22.04
  • Space Acres/Advanced CLI/Docker:Advanced CLI

Problem

Cluster mode was adopted, but now 90% of rewards are skipped

2024-11-25T12:14:48.484599Z ERROR {farm_index=4}: subspace_farmer::single_disk_farm::farming: Failed to prove, scheduling sector for replotting slot=1623348 sector_index=1408 error=Record reading error: Invalid chunk at location 46 s-bucket 0 encoded true, possible disk corruption: Invalid scalar
2024-11-25T15:25:05.805324Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::farming: Failed to prove, scheduling sector for replotting slot=1634743 sector_index=2100 error=Record reading error: Missing PoS proof for s-bucket 0
2024-11-25T15:28:30.641055Z  INFO {farm_index=3}: subspace_farmer::single_disk_farm::reward_signing: Successfully signed reward hash 0x636d6b3a6c73efc0177eb5cca2b4a78b4f96d2deb997687845b1bb59035f9225
2024-11-25T20:00:10.842711Z ERROR {farm_index=0}: subspace_farmer::single_disk_farm::farming: Failed to prove, scheduling sector for replotting slot=1651217 sector_index=364 error=Record reading error: Missing PoS proof for s-bucket 0
2024-11-25T20:01:23.997321Z ERROR {farm_index=4}: subspace_farmer::single_disk_farm::farming: Failed to prove, scheduling sector for replotting slot=1651290 sector_index=1596 error=Record reading error: Invalid chunk at location 995 s-bucket 1 encoded true, possible disk corruption: Invalid scalar
2024-11-25T21:00:47.411405Z ERROR {farm_index=2}: subspace_farmer::single_disk_farm::farming: Failed to prove, scheduling sector for replotting slot=1654847 sector_index=373 error=Record reading error: Missing PoS proof for s-bucket 0
2

error
error973×597 71.5 KB

3

Thank you for creating the post, but can you add the other information requested from Discord?

Full farmer logs
Full node logs would be nice as well
Information that you added in Discord such as Hardware and scrub results
Any errors you may have had while plotting
Version you are on now and if possible, the version you were on when plotting

Were the drives used for any prior testnet?

And since you are running a cluster now, the startup parameters for each role.

Thanks!

4

I strongly suspect when you configured cluster you reused cache file from another network (Gemini or Taurus), that is the most plausible explanation here.

Since consensus is based on archival storage of the blockchain itself, using pieces from a different blockchain will lead to invalid sectors and solutions. Pieces that are received from the network are validated, but local pieces from cache are implicitly assumed to be correct for efficiency purposes.

5

Cluster I just switched over these two days, so I can be sure that my cache file must be the latest, is re-downloaded back.

6

the drives used for Gemini 3h

7 
./subspaceFarmer cluster --prometheus-listen-on 0.0.0.0:7171 --nats-server nats://10.1.22.253:4222 controller --base-path /data/subspace/controller --node-rpc-url ws://10.1.22.253:9944
./subspaceFarmer cluster --prometheus-listen-on 0.0.0.0:8181 --nats-server nats://10.1.22.253:4222 cache path=/data/subspace/cache,size=200GiB
./subspaceFarmer cluster --prometheus-listen-on 0.0.0.0:8181 --nats-server nats://10.1.22.253:4222 plotter
./subspaceFarmer cluster --prometheus-listen-on 0.0.0.0:9191 --nats-server nats://10.1.22.253:4222 farmer --reward-address subotW1z1Qd5kYJykkp1WLqFahHHAP59gPW6NqnPSixWFbDLc  path=/data/subspace/farm2,size=3839152934KiB path=/data/subspace/farm3,size=3839152934KiB path=/data/subspace/farm5,size=958989062KiB path=/data/subspace/farm6,size=1919049454KiB path=/data/subspace/farm7,size=3749147381KiB | tee -a /data/subspace/farm.log
8

I think the safest would be to simply replot, the thing is clearly corrupted and you’ll be banned by other network participants if you send them invalid pieces from local cache.

9

My node does not have a public IP, and the number of node connections is almost always in single digits. Is this also the reason?

10

It will reduce download speed and may impact success of rewards in case of block reorgs, but it will not result in invalid pieces

11

Do I need to reset my cache completely? Or I just need to reset farm?

12

Delete both farms and caches, I suspect caches are corrupted, in which cache you’ll replot broken pieces again unless you sync cache from scratch

13 
rm -rf

Is this method undesirable, need to adopt the following way can?

sudo mkfs.ext4 -F -m 0 -T largefile4 /dev/nvme0n1
14

Or do you have a better idea?

15

You can remove the files, format the disks, doesn’t really matter

16

In fact, all my disks are deleted cache files and farm files in the main network re-plot, but now it is a large area of the problem.

Other people use non-main network cache, causing cache synchronization when suffering from pollution, and then my cache synchronization to these abnormal cache, is there such a possibility?

17

When your farmer downloads pieces from DSN it will verify whether they belong to the correct network, so this should not actually be the case, you’ll see messages like these when bad piece is returned:

2024-11-26T14:09:17.106031Z WARN {download_id=6594952138070911796}: subspace_farmer::farmer_piece_getter::piece_validator: Received invalid piece from peer piece_index=57 source_peer_id=12D3KooWK7Q7Z7oX96LSJtWPx877Y4PNhx2dKxPzCt9uXrfADz7S

18

Why can’t CLI detect that there’s something wrong with that data and replot that block alone? Is disk scrubbing too expensive?

19

Cost of full verification is the same as plotting (even more in some sense). Scrubbing checks checksums, detecting some in-memory corruptions and disk corruptions, but it doesn’t detect invalid data being written to begin with.

20 
2024-11-26T17:05:46.103400Z  WARN {download_id=708083724034619860}: subspace_farmer::farmer_piece_getter::piece_validator: Received invalid piece from peer piece_index=24466 source_peer_id=12D3KooWMKC7HVEJ1oeXtuHhZbgzWe4YYiy7ub5WKnEKZGgvWFJC
2024-11-26T17:06:28.717817Z  WARN {download_id=708083724034619860}: subspace_farmer::farmer_piece_getter::piece_validator: Received invalid piece from peer piece_index=1040 source_peer_id=12D3KooWMojH4chWcCRonACQwBEkRdHdLLM1UQvhHBGkkLEdCW7a
2024-11-26T17:06:30.103716Z  INFO subspace_farmer::farmer_cache: Piece cache sync 0.10% complete
2024-11-26T17:06:53.189571Z  INFO subspace_farmer::farmer_cache: Piece cache sync 0.20% complete

I formatted my node, nats, controller,cache disk, restarted sync cache, and got this warning