Failed to add provider record

Support
1

Issue Report

Environment

  • Operating System: Ubuntu 20.04.5 LTS
  • CPU Architecture: 6 vCPU Cores AMD EPYC 7282 x86_64
  • RAM: 16 GB
  • Storage: 400GB SSD
  • Plot Size: 10GB
  • Subspace Deployment Method: Sustrate CLI

Problem

Everything started up normally without errors, after a while errors appeared in the logs and the node stopped climbing, after the restart it continues climbing, but after a while errors appear again.

Logs node

2023-03-01 19:16:23 [PrimaryChain] Failed to add provider record: ProviderRecord { key: Key(b"\xf8\xb9\xce\x05 \xd3k\x03\xd2\xd4ZN\xb9rg \x86\xb5\xf3\xd6r\xd8\x154\xbe\xe2\xd6\xf9\xfb+\xd5\xe0\x85\xc4f\x8b\xfc"), provider: PeerId("12D3KooWPT8XWwuHVjdokuH9d5csbMUHazDBJrJAuqVrr6ueqZoT"), expires: Some(Instant { tv_sec: 2780838, tv_nsec: 58602998 }), addresses: ["/ip4/172.18.0.1/tcp/30433", "/ip4/65.109.92.26/tcp/30433", "/ip6/2a01:4f9:3051:4651::2/tcp/30433"] } err=MaxProvidedKeys
2023-03-01 19:16:23 [PrimaryChain] Failed to add provider record: ProviderRecord { key: Key(b"\xf8\xb9\xce\x05 \xd2A\xa0\xc8\xa4\x9f\xe0\xfa.\x1c\x17>H*\x91[\xd0\xd1\x9d-\x0b\xaa]l\x90\xeb\x81\x0e\xaf\xf4\x8eE"), provider: PeerId("12D3KooWPT8XWwuHVjdokuH9d5csbMUHazDBJrJAuqVrr6ueqZoT"), expires: Some(Instant { tv_sec: 2780838, tv_nsec: 116448249 }), addresses: ["/ip4/172.18.0.1/tcp/30433", "/ip4/65.109.92.26/tcp/30433", "/ip6/2a01:4f9:3051:4651::2/tcp/30433"] } err=MaxProvidedKeys
2023-03-01 19:16:23 [PrimaryChain] Failed to add provider record: ProviderRecord { key: Key(b"\xf8\xb9\xce\x05 \x15o\xc5\x15-\xd4\xb1\xa5\x02\x0b!\x9e\xf6;\xea\xf4g\x1a\xb9\x9e;\x95\xda\xb9\x12\x18\xe9\xca\x9bI\x87\x7f"), provider: PeerId("12D3KooWPT8XWwuHVjdokuH9d5csbMUHazDBJrJAuqVrr6ueqZoT"), expires: Some(Instant { tv_sec: 2780838, tv_nsec: 189478197 }), addresses: ["/ip4/172.18.0.1/tcp/30433", "/ip4/65.109.92.26/tcp/30433", "/ip6/2a01:4f9:3051:4651::2/tcp/30433"] } err=MaxProvidedKeys
2023-03-01 19:16:23 [PrimaryChain] Failed to add provider record: ProviderRecord { key: Key(b"\xf8\xb9\xce\x05 /\x1d\xf0\x07[\0\0T\xbc\xa4B\xc1\x05\x80\xfa\x91M\xcb\xf1p\x87\xe3\xcf9Vz\x14\xa9S\xdc\x8cN"), provider: PeerId("12D3KooWPT8XWwuHVjdokuH9d5csbMUHazDBJrJAuqVrr6ueqZoT"), expires: Some(Instant { tv_sec: 2780838, tv_nsec: 260390094 }), addresses: ["/ip4/172.18.0.1/tcp/30433", "/ip4/65.109.92.26/tcp/30433", "/ip6/2a01:4f9:3051:4651::2/tcp/30433"] } err=MaxProvidedKeys
2023-03-01 19:16:24 [PrimaryChain] Failed to add provider record: ProviderRecord { key: Key(b"\xf8\xb9\xce\x05 \xbcft\x9d\xde\r\xb2Rn\xacY\xcc\xbb\xbc\xc4c\xfa6\xaf\xd5\x03O\xb4\x83\xb6\xa8\xf1$\xc2:0\xb1"), provider: PeerId("12D3KooWPT8XWwuHVjdokuH9d5csbMUHazDBJrJAuqVrr6ueqZoT"), expires: Some(Instant { tv_sec: 2780838, tv_nsec: 361647962 }), addresses: ["/ip4/172.18.0.1/tcp/30433", "/ip4/65.109.92.26/tcp/30433", "/ip6/2a01:4f9:3051:4651::2/tcp/30433"] } err=MaxProvidedKeys
2023-03-01 19:16:24 [PrimaryChain] Failed to add provider record: ProviderRecord { key: Key(b"\xf8\xb9\xce\x05 \xb1\xa4\x9e\x13;\xee\xe3\x18\x19\xbe\x12Pc\x14!a\xbb\x83\xce\xb0\xd7\x18\xa5\xbb\x04c\x8a7\x1ci\xdc\x87"), provider: PeerId("12D3KooWPT8XWwuHVjdokuH9d5csbMUHazDBJrJAuqVrr6ueqZoT"), expires: Some(Instant { tv_sec: 2780838, tv_nsec: 449059646 }), addresses: ["/ip4/172.18.0.1/tcp/30433", "/ip4/65.109.92.26/tcp/30433", "/ip6/2a01:4f9:3051:4651::2/tcp/30433"] } err=MaxProvidedKeys
2023-03-01 19:16:24 [PrimaryChain] Failed to add provider record: ProviderRecord { key: Key(b"\xf8\xb9\xce\x05 \xd5\x80\xb3]\xf7\x12\x15u\x1c\xf0\x03\xf4z\xd5\x8eH\xe6\xa4\x9eh\xa1\xcfXDn\xe3\xb9.\xbfI\x06\xff"), provider: PeerId("12D3KooWPT8XWwuHVjdokuH9d5csbMUHazDBJrJAuqVrr6ueqZoT"), expires: Some(Instant { tv_sec: 2780838, tv_nsec: 564472575 }), addresses: ["/ip4/172.18.0.1/tcp/30433", "/ip4/65.109.92.26/tcp/30433", "/ip6/2a01:4f9:3051:4651::2/tcp/30433"] } err=MaxProvidedKeys
2023-03-01 19:16:24 [PrimaryChain] Failed to add provider record: ProviderRecord { key: Key(b"\xf8\xb9\xce\x05 \x12[V\x84NY\x8e\x05z\xc5\x1a\x15J\x17\x8b\x0f9\x9b\xca\xb9{6\xbb\x8b\xb9\xad\x06\xae8W\xd9\x99"), provider: PeerId("12D3KooWPT8XWwuHVjdokuH9d5csbMUHazDBJrJAuqVrr6ueqZoT"), expires: Some(Instant { tv_sec: 2780838, tv_nsec: 624206958 }), addresses: ["/ip4/172.18.0.1/tcp/30433", "/ip4/65.109.92.26/tcp/30433", "/ip6/2a01:4f9:3051:4651::2/tcp/30433"] } err=MaxProvidedKeys
2023-03-01 19:16:24 [PrimaryChain] Failed to add provider record: ProviderRecord { key: Key(b"\xf8\xb9\xce\x05 mGZ\xdf\xc7\xe6\xf0\x88\xdd\xe6J:8\xc6I\xba\xd3O\xb4\x18<\x96\xc4\xc1\xd5\xf4F{<)(\xd7"), provider: PeerId("12D3KooWPT8XWwuHVjdokuH9d5csbMUHazDBJrJAuqVrr6ueqZoT"), expires: Some(Instant { tv_sec: 2780838, tv_nsec: 748746677 }), addresses: ["/ip4/172.18.0.1/tcp/30433", "/ip4/65.109.92.26/tcp/30433", "/ip6/2a01:4f9:3051:4651::2/tcp/30433"] } err=MaxProvidedKeys

Logs Farmer

2023-03-01T18:16:57.311695Z  WARN single_disk_plot{disk_farm_index=0}: subspace_networking::utils::piece_provider: Couldn't get a piece from DSN. Retrying... piece_index=68871
2023-03-01T18:16:57.403300Z  WARN single_disk_plot{disk_farm_index=0}: subspace_networking::utils::piece_provider: Couldn't get a piece from DSN. Retrying... piece_index=127657
2023-03-01T18:16:57.511438Z  WARN single_disk_plot{disk_farm_index=0}: subspace_networking::utils::piece_provider: Couldn't get a piece from DSN. Retrying... piece_index=96090
2023-03-01T18:16:57.540692Z  WARN single_disk_plot{disk_farm_index=0}: subspace_networking::utils::piece_provider: Couldn't get a piece from DSN. Retrying... piece_index=116539
2023-03-01T18:16:57.634485Z  WARN single_disk_plot{disk_farm_index=0}: subspace_networking::utils::piece_provider: Couldn't get a piece from DSN. Retrying... piece_index=73590
2023-03-01T18:16:57.659329Z ERROR yamux::connection: 7f8d265e: socket error: decode error: i/o error: unexpected end of file
2023-03-01T18:16:57.659359Z ERROR yamux::connection: 5417162f: socket error: decode error: i/o error: unexpected end of file
2023-03-01T18:16:57.685152Z  WARN single_disk_plot{disk_farm_index=0}: subspace_networking::utils::piece_provider: Couldn't get a piece from DSN. Retrying... piece_index=89832
2023-03-01T18:16:57.685227Z  WARN single_disk_plot{disk_farm_index=0}: subspace_networking::utils::piece_provider: Couldn't get a piece from DSN. Retrying... piece_index=69506
2023-03-01T18:16:57.699441Z  WARN single_disk_plot{disk_farm_index=0}: subspace_networking::utils::piece_provider: Couldn't get a piece from DSN. Retrying... piece_index=134506
2023-03-01T18:16:57.780378Z  WARN single_disk_plot{disk_farm_index=0}: subspace_networking::utils::piece_provider: Couldn't get a piece from DSN. Retrying... piece_index=74337
2023-03-01T18:16:57.865429Z  WARN single_disk_plot{disk_farm_index=0}: subspace_networking::utils::piece_provider: Couldn't get a piece from DSN. Retrying... piece_index=60852
2023-03-01T18:16:57.869103Z  WARN single_disk_plot{disk_farm_index=0}: subspace_networking::utils::piece_provider: Couldn't get a piece from DSN. Retrying... piece_index=33008
2023-03-01T18:16:57.869147Z  WARN single_disk_plot{disk_farm_index=0}: subspace_networking::utils::piece_provider: Couldn't get a piece from DSN. Retrying... piece_index=86236
2023-03-01T18:16:57.869154Z  WARN single_disk_plot{disk_farm_index=0}: subspace_networking::utils::piece_provider: Couldn't get a piece from DSN. Retrying... piece_index=58333
2023-03-01T18:16:57.869160Z  WARN single_disk_plot{disk_farm_index=0}: subspace_networking::utils::piece_provider: Couldn't get a piece from DSN. Retrying... piece_index=46334
2023-03-01T18:16:57.869608Z  WARN single_disk_plot{disk_farm_index=0}: subspace_networking::utils::piece_provider: Couldn't get a piece from DSN. Retrying... piece_index=84168
1 Like
2

I got same problem, how to fix it thabnk you

1 Like
3

Hello King

Normal, non-fatal error. It is retrying. Reported on Discord as well. Have you left it running for a while to see if things improve?Discord

4

Hi sir how to check log without that warn thank sir

5

Hello King. Understand warns still persist after you left running for a while, please provide you environment configuration and will escalate it to the team.

In the link below, the resolution of similar issue:

7

Hi team, i also have a similar error
[PrimaryChain] Failed to add provider record: ProviderRecord { key: Key(b"\xf8\xb9\xce\x05 \x97\x99$\xd1jz\x0b\xc9\x18y\xd9GA\x1e\xb7NK\xa9\xb6Fr\x9acOP5\xcc\xdc}r\xb1\xb7"), provider: PeerId(“12D3KooWCx8qyhAhMBAZ3LHY7N5Vy9urJMThYLQ9XEo8gyTSURAJ”), expires: Some(Instant { tv_sec: 173002, tv_nsec: 836506461 }), addresses: [“/ip4/173.249.30.179/tcp/30433”, “/ip6/2a02:c207:2118:4248::1/tcp/30433”] } err=MaxProvidedKeys OS: Ubuntu 20.04.5 LTS
CPU: 4xAMD EPYC 7282
RAM: 8 Gb Storage: 100 Gb SSD
Subspace Deployment Method: Advanced CLI

8

Hello Andrey

Non-fatal error. Still persisting after running for a while?

9

Had another report of the [PrimaryChain] Failed to add provider record: ProviderRecord problem. The user is going to add details. Pinging @Support-L2.

10

Running on Windows Server 2019 with 384gb of ram and 2 @ E5-2697a CPU’s. When I first started syncing I was running the simple CLI but after about 4-5 days I switched to the advanced CLI as I have 96 TB of SSD’s I want to use and I was told the simple one wouldn’t support multiple files. After about 5-6 days it was finally synced and I started up the farmer in another powershell window. That one just fills up with the Couldn’t get a peice from DSN. Retrying… and it does not appear to be farming at all.

I have started and stopped both the farmer and node processes quite a bit. Yesterday I started having the Failed to add provider record error on the node. I assumed it was related to the farmer issue but after letting things stay stopped overnight and syncing the node again I am getting the error without the farmer process running.

I am currently not synced and I can restart the node process and usually within 10 minutes it will start to give me this Failed to add provider error again. Once that happens it doesn’t appear to be syncing anymore. I guess it might be, but the screen is being filled with so many error messages that it is hard to tell. I can use the scroll bar on the powershell window to go back pretty far and I don’t see any messages about the sync or the node numbers so I am assuming that it is no longer syncing.

Here is the command I am using to start the node:

./subspace-node --chain gemini-3c
–execution wasm --blocks-pruning archive
–state-pruning archive --dsn-disable-private-ips
–no-private-ipv4 --validator
–in-peers 5 --in-peers-light 5
–out-peers 5 --name Wezl

11

Did you ever solve this?