Custom error: Solution was ignored

Support
, ,
1

Issue Report

Environment

  • Ubuntu 22:
  • Pulsar/Advanced CLI/Docker:

Problem

I
I found this error message in the log, and the ratio of this error to obtaining Vote Reward is approximately 1:8. Yesterday there were 287 errors and 2674 Vote Rewards

To solve this problem, three new node servers will be built to distribute pressure. The server CPU is EPYC 7773X *2 RAM 512GB, and DISK IO is low

On the NODE server, I placed 24 pic 7.68T U2 hard drives. The plotting has been completed, but this error still exists on the node.

In such an environment, network issues can be ruled out as it is running on a local server. 2. There is no problem with the disk, and I have continuously observed that the disk IO wait is extremely low. 3. The CPU usage rate remains stable at 25% -35%.

I don’t understand where the problem lies or how I should start to solve it.

IMG_0973
IMG_09731179×949 177 KB

0127985329685c6809f2262991ec5d30
0127985329685c6809f2262991ec5d301383×149 13.2 KB

5c21903307a788a8bc58dd2a2dca56d5
5c21903307a788a8bc58dd2a2dca56d51347×145 12.6 KB

[WARN {farm_index=19}: subspace_farmer::single_disk_farm::farming: Failed to send solution to node, skipping further proving for this slot slot=10995381 sector_index=5482 error=RPC call failed: ErrorObject { code: ServerError(-32001), message: "Custom error: Solution was ignored", data: None }]
2

Try to benchmark proving on that machine with the drives: Tips & Tricks | Farm from Anywhere

There is usually a good reason why this happens, frequently caused by bad disks, but shouldn’t really be common with U.2 drives. 7773X has really low base frequency, so unless it is boosting to its max 3.5 GHz it might be a problem. Either way benchmakrking is the first step to understanding it. You can also check CPU and I/O utilization during this (note that right now prover in benchmark will use all CPU cores, while real prover is limited to 32 threads by default, I just made a note for myself to correct that in the future).

3

To solve this problem, can I adopt the following solution:

use a PC machine, configured with I9-14900KF 64G, as the NODE server.

4

Absolutely no idea. It is literally impossible to answer without knowing what the root cause is, which is exactly why I suggested to start with benchmarking.