Recovering missing piece failed

xorinox · September 15, 2023, 2:55am

Hello, I see a farmer stuck in a cycle failing to recover missing piece. The farmer never leaves this cycle. This farmer uses Sept-13-2 right now, but error also occurred with Sept-11. I saw some users reporting the same issue/error message on discord also.

Blockquote
2023-09-14T16:47:42.836080Z INFO single_disk_farm{disk_farm_index=0}: subspace_farmer::single_disk_farm::plotting: Node is synced, resuming plotting
2023-09-14T16:47:42.836139Z INFO single_disk_farm{disk_farm_index=1}: subspace_farmer::single_disk_farm::plotting: Node is synced, resuming plotting
2023-09-14T16:48:51.106731Z ERROR yamux::connection: 7d36dc01: maximum number of streams reached
2023-09-14T17:19:58.324072Z INFO single_disk_farm{disk_farm_index=0}: subspace_farmer_components::segment_reconstruction: Recovering missing piece… missing_piece_index=6086
2023-09-14T17:20:22.417245Z INFO single_disk_farm{disk_farm_index=1}: subspace_farmer_components::segment_reconstruction: Recovering missing piece… missing_piece_index=1920
2023-09-14T20:14:11.132484Z ERROR single_disk_farm{disk_farm_index=0}: subspace_farmer_components::segment_reconstruction: Recovering missing piece failed. missing_piece_index=6086 received_pieces=35 required_pieces_number=128
2023-09-14T20:14:11.132597Z INFO single_disk_farm{disk_farm_index=0}: subspace_farmer_components::segment_reconstruction: Recovering missing piece… missing_piece_index=6086
2023-09-14T20:16:07.707771Z ERROR single_disk_farm{disk_farm_index=1}: subspace_farmer_components::segment_reconstruction: Recovering missing piece failed. missing_piece_index=1920 received_pieces=48 required_pieces_number=128
2023-09-14T20:16:07.708105Z INFO single_disk_farm{disk_farm_index=1}: subspace_farmer_components::segment_reconstruction: Recovering missing piece… missing_piece_index=1695

I think the farmer had finished the plotting previously. This is one of my earliest farmers. Must have gone through most re-plotting cycles of all my farmers. Not sure if that matters.

Blockquote
farm4_path=“/chia/scratch/disk04/subspace/plots/”
farm5_path=“/chia/scratch/disk05/subspace/plots/”

cmd="/home/srv_subspace/subspace/target/x86_64-unknown-linux-gnu/production/subspace-farmer
farm
path=$farm4_path,size=6200G
path=$farm5_path,size=6200G
–reward-address …
"

nazar-pc · September 15, 2023, 2:38pm

This is probably the reason. Have you modified something, also did you change default CLI parameters related to networking? It can result in this (exceeding number of streams).

xorinox · September 15, 2023, 6:18pm

Self compiled but unchanged code (I have only changed the code on one single machine and reverted that after no success). Node and farmers run with defaults, only changed (from defaults) configuration parameters have to do with the ports.

nazar-pc · September 15, 2023, 11:41pm

We have seen this rarely, we’ll investigate why this happens. I do not have a good answer for you right now. Overall this shouldn’t be fatal, did you see plotting continue eventually or was it stuck permanently?

xorinox · September 16, 2023, 1:51am

It was stuck in there for a while, several hours, so i restarted, meanwhile also reinstalled sep-11, the nodes with sep-13-2 all lost sync and didn’t get back on track. with sep-11 they all resynced.

but this particular reported issue, i didn’t see it again so far.

Rabinovitch · September 19, 2023, 5:57pm

Oh, this is a long-known problem. Farmer does not come out of this state for days (I just did not check longer). This has been discussed many times in discord, the solution helps with reducing the number of allowed connections for the farmer and, possibly, for the node too.

nazar-pc · September 19, 2023, 7:47pm

This should no longer be necessary, recent releases addressed this specific issue, specifically sep-05 release if I’m not mistaken. As mentioned in other threads, things are changing and we do make improvements that should remove the need of workarounds.

Rabinovitch · September 19, 2023, 8:56pm

But I continue to get such an amazing experience with the latest release, 13-2.

nazar-pc · September 19, 2023, 9:13pm

Well, then would be great to report it here right away and help developers with debugging so we can improve things going forward.

Everything except --dsn-in-connections on the node is irrelevant for DSN, so I have no idea why would you customize it at all. And for farmer I’m not sure how it would help either to be completely honest.

Are you sure it is those parameters that help and not a coincidence? Can be checked by creating backup of the node data and starting once without those customized arguments and another time with them.

Rabinovitch · September 19, 2023, 9:39pm

I just gave an example from discord, it’s not my name in the screenshot.
Right now I am observing the problem being discussed:

It’s a dedicated farmer machine (the node is running on another machine).
Node’s startup parameters:

Description=Subspace Node
After=network.target
[Service]
Type=simple
User=admfc
ExecStart=/home/admfc/subspace/target/production/subspace-node \
--in-peers 10 --out-peers 10 --max-parallel-downloads 5 --dsn-in-connections 20 \
--rpc-external --rpc-methods=unsafe \
--rpc-cors all --rpc-max-connections="200" \
--chain="gemini-3f" \
--execution="wasm" \
--max-runtime-instances 99 \
--blocks-pruning="256" \
--state-pruning="archive" \
--no-private-ipv4 \
--validator \
--name="saturnia-len25"
Restart=on-failure
RestartSec=10
LimitNOFILE=1024000
[Install]
WantedBy=multi-user.target

Startup parameters of a problematic farmer:

[Unit]
Description=Subspace Farmer
After=network.target
[Service]
Type=simple
User=admfc
ExecStart=/home/admfc/subspace/target/production/subspace-farmer farm --reward-address=5HK8bjVVXbUPYPCRPNyZDdAGN5ae7M7aCKVyeKZUtDbNHDVB \
--listen-on /ip4/0.0.0.0/tcp/41325 \
--node-rpc-url ws://192.168.11.35:9944 \
--in-connections 10 --out-connections 10 \
path=/media/firecuda-1/plot0,size=893G \
path=/media/Viper-1/plot0,size=1933G \
path=/media/Viper-2/plot0,size=1933G \
path=/media/Viper-3/plot0,size=1933G \
path=/media/Viper-4/plot0,size=1933G
Restart=always
RestartSec=10
LimitNOFILE=1024000
[Install]
WantedBy=multi-user.target

There is another dedicated (remote) farmer connected to this node (in addition to the farmer working on the same machine where the node itself), it has finished plotting today. It has the same startup parameters. Is it possible that the node does not support more than two farmers (which are already connected to it)?

Rabinovitch · September 19, 2023, 10:40pm

At the moment, reducing the number of plots from 5 to 3 has helped, the plotting has started.

nazar-pc · September 20, 2023, 4:54am

Looks like they are having issues with those arguments added, not without, am I reading it correctly?

It would be ideal if people didn’t tweak those parameters unless requested by developers, it wasn’t once or twice that it was the reason things didn’t work as expected.

No, node can work with many more farmers implementation-wise.

Rabinovitch · September 20, 2023, 7:22am

With those parameters added plotters at least start plotting. The community does not resort to their help out of boredom…
Of course, perhaps it’s not about these para, but about the limit on the maximum number of plots that farmer agrees to plot, but this is not mentioned anywhere. What remains for the community? Look for solutions to their problems on their own. And if these measures help, so be it.

Rabinovitch · September 20, 2023, 7:24am

Don’t you think this is an empty argument? After all, it seems that no one has really said yet why the plotter enters this conditionally eternal cycle of recovering missing pieces, what causes this behavior and how to deal with it…

xorinox · September 20, 2023, 1:29pm

I have a handful of farmers also suffering from that issue. But I also see a couple more suspicious things in the log… I can sen you the log later

nazar-pc · September 20, 2023, 9:47pm

I understand that users are not doing it for no reason, but at the same time thigns might be fixed on the latest release and in some cases evidence is anecdotal.

It would be great to have such things shared on the forum, we can then request more detailed logs with and without those parameters to understand why it is helping (if it does) to make it work well by default. In the end the goal is to make it “just work”.

nazar-pc · September 20, 2023, 11:58pm

Interesting question actually, what was the first release you have used? I’m asking because before aug-21 there was a bug in the code that may result in node being banned on DSN. If you wiped the plot after upgrade or started with aug-21 or later release then this is not applicable.

xorinox · September 21, 2023, 3:33am

Have a farmer doesn’t even start to plot, setup was newly created. sep-13-2, trying 21 now

Blockquote
2023-09-21T00:55:50.998877Z INFO subspace_farmer::commands::farm: Collecting already plotted pieces (this will take some time)…
2023-09-21T00:55:50.998883Z INFO subspace_farmer::commands::farm: Finished collecting already plotted pieces successfully
2023-09-21T00:55:50.998936Z INFO subspace_farmer::piece_cache: Initializing piece cache
2023-09-21T00:55:51.000043Z INFO single_disk_farm{disk_farm_index=0}: subspace_farmer::single_disk_farm: Subscribing to slot info notifications
2023-09-21T00:55:51.000199Z INFO single_disk_farm{disk_farm_index=0}: subspace_farmer::reward_signing: Subscribing to reward signing notifications
2023-09-21T00:55:51.000216Z INFO single_disk_farm{disk_farm_index=1}: subspace_farmer::single_disk_farm: Subscribing to slot info notifications
2023-09-21T00:55:51.000220Z INFO single_disk_farm{disk_farm_index=1}: subspace_farmer::reward_signing: Subscribing to reward signing notifications
2023-09-21T00:55:51.000233Z INFO single_disk_farm{disk_farm_index=2}: subspace_farmer::single_disk_farm: Subscribing to slot info notifications
2023-09-21T00:55:51.000237Z INFO single_disk_farm{disk_farm_index=2}: subspace_farmer::reward_signing: Subscribing to reward signing notifications
2023-09-21T00:55:51.000378Z INFO single_disk_farm{disk_farm_index=3}: subspace_farmer::single_disk_farm: Subscribing to slot info notifications
2023-09-21T00:55:51.000484Z INFO single_disk_farm{disk_farm_index=3}: subspace_farmer::reward_signing: Subscribing to reward signing notifications
2023-09-21T00:55:51.001288Z INFO single_disk_farm{disk_farm_index=0}: subspace_farmer::single_disk_farm::plotting: Subscribing to archived segments
2023-09-21T00:55:51.004252Z INFO single_disk_farm{disk_farm_index=1}: subspace_farmer::single_disk_farm::plotting: Subscribing to archived segments
2023-09-21T00:55:51.004865Z INFO subspace_farmer::commands::farm::dsn: DSN listening on /ip4/127.0.0.1/tcp/30406/p2p/…
2023-09-21T00:55:51.004974Z INFO subspace_farmer::commands::farm::dsn: DSN listening on /ip4/10.1.100.30/tcp/30406/p2p/…
2023-09-21T00:55:51.009945Z INFO single_disk_farm{disk_farm_index=2}: subspace_farmer::single_disk_farm::plotting: Subscribing to archived segments
2023-09-21T00:55:51.013137Z INFO single_disk_farm{disk_farm_index=3}: subspace_farmer::single_disk_farm::plotting: Subscribing to archived segments
2023-09-21T00:55:51.075974Z INFO subspace_networking::node_runner: Added observed address as external: /ip4/8.8.8.8/tcp/30406
2023-09-21T00:56:03.952011Z INFO subspace_networking::node_runner: Public address status changed. old=Unknown new=Private
2023-09-21T00:58:35.271670Z INFO subspace_farmer::piece_cache: Synchronizing cache
2023-09-21T01:06:34.280205Z INFO single_disk_farm{disk_farm_index=1}: subspace_farmer_components::segment_reconstruction: Recovering missing piece… missing_piece_index=4615
2023-09-21T01:06:34.297562Z ERROR yamux::connection: 9de79bbf: maximum number of streams reached
2023-09-21T01:06:34.329215Z ERROR yamux::connection: 4eed3dfb: maximum number of streams reached
2023-09-21T01:06:34.330569Z ERROR yamux::connection: 07407985: maximum number of streams reached
2023-09-21T01:06:34.341485Z INFO single_disk_farm{disk_farm_index=3}: subspace_farmer_components::segment_reconstruction: Recovering missing piece… missing_piece_index=6329
2023-09-21T01:06:34.373631Z ERROR yamux::connection: 2857d6fb: maximum number of streams reached
2023-09-21T01:06:34.374652Z ERROR yamux::connection: 42b44e86: maximum number of streams reached
2023-09-21T01:06:34.378987Z ERROR yamux::connection: c04644bf: maximum number of streams reached
2023-09-21T01:06:34.380793Z INFO single_disk_farm{disk_farm_index=2}: subspace_farmer_components::segment_reconstruction: Recovering missing piece… missing_piece_index=4526
2023-09-21T01:06:34.381492Z INFO single_disk_farm{disk_farm_index=1}: subspace_farmer_components::segment_reconstruction: Recovering missing piece… missing_piece_index=7436
2023-09-21T01:06:34.390924Z INFO single_disk_farm{disk_farm_index=1}: subspace_farmer_components::segment_reconstruction: Recovering missing piece… missing_piece_index=2497
2023-09-21T01:06:34.391230Z INFO single_disk_farm{disk_farm_index=1}: subspace_farmer_components::segment_reconstruction: Recovering missing piece… missing_piece_index=4726
2023-09-21T01:06:34.392037Z INFO single_disk_farm{disk_farm_index=0}: subspace_farmer_components::segment_reconstruction: Recovering missing piece… missing_piece_index=2285
2023-09-21T01:06:34.392894Z INFO single_disk_farm{disk_farm_index=0}: subspace_farmer_components::segment_reconstruction: Recovering missing piece… missing_piece_index=5042
2023-09-21T01:06:34.395350Z INFO single_disk_farm{disk_farm_index=3}: subspace_farmer_components::segment_reconstruction: Recovering missing piece… missing_piece_index=8152
2023-09-21T01:06:34.434777Z ERROR yamux::connection: 1065f142: maximum number of streams reached
2023-09-21T01:06:34.447526Z INFO single_disk_farm{disk_farm_index=0}: subspace_farmer_components::segment_reconstruction: Recovering missing piece… missing_piece_index=5339
2023-09-21T01:06:34.450208Z INFO single_disk_farm{disk_farm_index=3}: subspace_farmer_components::segment_reconstruction: Recovering missing piece… missing_piece_index=7529
2023-09-21T01:06:34.460856Z INFO single_disk_farm{disk_farm_index=3}: subspace_farmer_components::segment_reconstruction: Recovering missing piece… missing_piece_index=3691
2023-09-21T01:06:34.467599Z INFO single_disk_farm{disk_farm_index=0}: subspace_farmer_components::segment_reconstruction: Recovering missing piece… missing_piece_index=7869
2023-09-21T01:06:34.515328Z INFO single_disk_farm{disk_farm_index=2}: subspace_farmer_components::segment_reconstruction: Recovering missing piece… missing_piece_index=3257
2023-09-21T01:06:34.516873Z INFO single_disk_farm{disk_farm_index=2}: subspace_farmer_components::segment_reconstruction: Recovering missing piece… missing_piece_index=3360
2023-09-21T01:06:34.523081Z ERROR yamux::connection: bf45c269: maximum number of streams reached

Blockquote
cmd="/home/srv_subspace/subspace/target/x86_64-unknown-linux-gnu/production/subspace-farmer
farm
path=$farm1_path,size=750G
path=$farm2_path,size=750G
path=$farm3_path,size=750G
path=$farm4_path,size=750G
path=$farm5_path,size=750G
path=$farm6_path,size=750G
path=$farm7_path,size=750G
–reward-address …
–node-rpc-url ws://10.1.100.32:30501
–listen-on /ip4/0.0.0.0/tcp/30406
"

Rabinovitch · September 21, 2023, 7:20pm

3 plots, no connection limitations (though this is the third farmer on the same LAN/ISP).

Rabinovitch · September 21, 2023, 8:29pm

This is the first time I see that a farmer has started plotting in such a situation…

Topic		Replies	Views
My node syncs blocks fine but farmer never draws, this has been going on for a long time. Support cli , error , farmer	2	308	September 14, 2023
Error - Recovering missing piece Support	11	210	November 15, 2023
Farmer Errors with 0 Rewards Support	7	528	May 3, 2023
Segment_reconstruction: Recovering missing piece on free disk Support gemini , nodes	1	167	September 14, 2023
Farmer starting error Support	1	292	April 24, 2023

Recovering missing piece failed

Related topics