Gemini 3a prerelease testing - Couldn't get a piece from DSN

jim-counter · December 13, 2022, 4:59pm

Issue Report

Environment

Operating System: Ubuntu 22.04 LTS
CPU Architecture: Intel x64
RAM: 16GiB
Storage: NVME
Plot Size: 100GiB
Subspace Deployment Method: Docker (though this issue has also been reported using a systemd service and Windows CLI too)

Problem

Error reported in farmer logs Couldn't get a piece from DSN, farmer does not seem to be plotting.

Steps to reproduce

Update docker-compose.yml with images ghcr.io/subspace/node:gemini-3a-2022-dec-06 and ghcr.io/subspace/farmer:gemini-3a-2022-dec-06.
Docker compose up.

Expected result

Node and farmer run without error and start syncing.

What happens instead

Node starts syncing (seemingly happily).
Farmer continually reports warning messages (as below).

subspace-farmer  | 2022-12-13T16:56:34.107970Z  INFO single_disk_plot{single_disk_plot_id=01GM65BM72RGVHKTS78840ZC5Q}: subspace_farmer::single_disk_plot::piece_receiver: get_value returned no piece-by-sector provider piece_index=24 key=Multihash { code: 11771130, size: 32, digest: [247, 45, 186, 229, 4, 219, 147, 98, 71, 247, 188, 104, 1, 199, 57, 92, 45, 143, 175, 174, 100, 241, 124, 21, 202, 124, 56, 151, 172, 130, 99, 102, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] }
subspace-farmer  | 2022-12-13T16:56:34.107998Z  WARN single_disk_plot{single_disk_plot_id=01GM65BM72RGVHKTS78840ZC5Q}: subspace_farmer::single_disk_plot::piece_receiver: Couldn't get a piece from DSN. Retrying... piece_index=24
subspace-farmer  | 2022-12-13T16:56:34.164336Z  INFO single_disk_plot{single_disk_plot_id=01GM65BM72RGVHKTS78840ZC5Q}: subspace_farmer::single_disk_plot::piece_receiver: get_value returned no piece-by-sector provider piece_index=19 key=Multihash { code: 11771130, size: 32, digest: [69, 34, 200, 228, 250, 54, 139, 83, 97, 33, 186, 160, 139, 209, 89, 115, 230, 220, 14, 71, 66, 245, 166, 96, 128, 167, 50, 59, 235, 27, 34, 43, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] }
subspace-farmer  | 2022-12-13T16:56:34.164378Z  WARN single_disk_plot{single_disk_plot_id=01GM65BM72RGVHKTS78840ZC5Q}: subspace_farmer::single_disk_plot::piece_receiver: Couldn't get a piece from DSN. Retrying... piece_index=19
subspace-farmer  | 2022-12-13T16:56:35.342528Z  INFO single_disk_plot{single_disk_plot_id=01GM65BM72RGVHKTS78840ZC5Q}: subspace_farmer::single_disk_plot::piece_receiver: get_value returned no piece-by-sector provider piece_index=1 key=Multihash { code: 11771130, size: 32, digest: [29, 189, 125, 11, 86, 26, 65, 210, 60, 42, 70, 154, 212, 47, 189, 112, 213, 67, 139, 174, 130, 111, 111, 214, 7, 65, 49, 144, 195, 124, 54, 59, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] }
subspace-farmer  | 2022-12-13T16:56:35.342561Z  WARN single_disk_plot{single_disk_plot_id=01GM65BM72RGVHKTS78840ZC5Q}: subspace_farmer::single_disk_plot::piece_receiver: Couldn't get a piece from DSN. Retrying... piece_index=1
subspace-farmer  | 2022-12-13T16:56:35.914947Z  INFO single_disk_plot{single_disk_plot_id=01GM65BM72RGVHKTS78840ZC5Q}: subspace_farmer::single_disk_plot::piece_receiver: get_value returned no piece-by-sector provider piece_index=149 key=Multihash { code: 11771130, size: 32, digest: [141, 102, 63, 125, 10, 184, 69, 24, 118, 204, 186, 93, 247, 66, 57, 90, 241, 212, 229, 165, 230, 90, 44, 166, 27, 226, 166, 37, 168, 159, 113, 58, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] }
subspace-farmer  | 2022-12-13T16:56:35.915006Z  WARN single_disk_plot{single_disk_plot_id=01GM65BM72RGVHKTS78840ZC5Q}: subspace_farmer::single_disk_plot::piece_receiver: Couldn't get a piece from DSN. Retrying... piece_index=149
subspace-farmer  | 2022-12-13T16:56:35.960265Z  INFO single_disk_plot{single_disk_plot_id=01GM65BM72RGVHKTS78840ZC5Q}: subspace_farmer::single_disk_plot::piece_receiver: get_value returned no piece-by-sector provider piece_index=57 key=Multihash { code: 11771130, size: 32, digest: [55, 131, 186, 209, 7, 49, 109, 225, 59, 102, 248, 30, 240, 5, 213, 87, 134, 80, 4, 72, 17, 49, 201, 149, 246, 79, 156, 159, 15, 82, 128, 116, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] }
subspace-farmer  | 2022-12-13T16:56:35.960299Z  WARN single_disk_plot{single_disk_plot_id=01GM65BM72RGVHKTS78840ZC5Q}: subspace_farmer::single_disk_plot::piece_receiver: Couldn't get a piece from DSN. Retrying... piece_index=57

jim-counter · December 13, 2022, 5:09pm

Complete startup logs in case they will help. Note this is a slightly different setup to the environment in the original report but same result.

Attaching to subspace-farmer, subspace-node
subspace-node    | 2022-12-13 17:04:40 Subspace    
subspace-node    | 2022-12-13 17:04:40 ✌️  version 0.1.0-d06ad0bad8a9cac203d0e0a9b27611b9f0f48e2a    
subspace-node    | 2022-12-13 17:04:40 ❤️  by Subspace Labs <https://subspace.network>, 2021-2022    
subspace-node    | 2022-12-13 17:04:40 📋 Chain specification: Subspace Gemini 3a    
subspace-node    | 2022-12-13 17:04:40 🏷  Node name: counterpoint_vps_london    
subspace-node    | 2022-12-13 17:04:40 👤 Role: AUTHORITY    
subspace-node    | 2022-12-13 17:04:40 💾 Database: ParityDb at /var/subspace/chains/subspace_gemini_3a/paritydb/full    
subspace-node    | 2022-12-13 17:04:40 ⛓  Native runtime: subspace-0 (subspace-0.tx0.au0)    
subspace-node    | 2022-12-13 17:04:41 [PrimaryChain] 🔨 Initializing Genesis block/state (state: 0xdd09…c28f, header-hash: 0xcabf…b605)    
subspace-node    | 2022-12-13 17:04:41 [DSN] DSN instance configured. allow_non_global_addresses_in_dht=true peer_id=12D3KooWGTPLpX7kPnwr7gLdBTguhWrxNz8J5h41GSDzoBfN8Vey
subspace-node    | 2022-12-13 17:04:41 [DSN] Subspace networking initialized: Node ID is 12D3KooWGTPLpX7kPnwr7gLdBTguhWrxNz8J5h41GSDzoBfN8Vey
subspace-node    | 2022-12-13 17:04:41 [PrimaryChain] Starting archiving from genesis    
subspace-node    | 2022-12-13 17:04:41 [PrimaryChain] Archiving already produced blocks 0..=0    
subspace-node    | 2022-12-13 17:04:45 [DSN] Processing a segment. segment_index=0
subspace-node    | 2022-12-13 17:04:45 [DSN] Segment processed. segment_index=0
subspace-node    | 2022-12-13 17:04:45 [PrimaryChain] 🏷  Local node identity is: 12D3KooWGTPLpX7kPnwr7gLdBTguhWrxNz8J5h41GSDzoBfN8Vey    
subspace-node    | 2022-12-13 17:04:45 [PrimaryChain] 🧑‍🌾 Starting Subspace Authorship worker    
subspace-node    | 2022-12-13 17:04:45 [PrimaryChain] 💻 Operating system: linux    
subspace-node    | 2022-12-13 17:04:45 [PrimaryChain] 💻 CPU architecture: x86_64    
subspace-node    | 2022-12-13 17:04:45 [PrimaryChain] 💻 Target environment: gnu    
subspace-node    | 2022-12-13 17:04:45 [PrimaryChain] 💻 CPU: AMD EPYC 7282 16-Core Processor    
subspace-node    | 2022-12-13 17:04:45 [PrimaryChain] 💻 CPU cores: 8    
subspace-node    | 2022-12-13 17:04:45 [PrimaryChain] 💻 Memory: 30096MB    
subspace-node    | 2022-12-13 17:04:45 [PrimaryChain] 💻 Kernel: 5.4.0-105-generic    
subspace-node    | 2022-12-13 17:04:45 [PrimaryChain] 💻 Linux distribution: Ubuntu 20.04.5 LTS    
subspace-node    | 2022-12-13 17:04:45 [PrimaryChain] 💻 Virtual machine: yes    
subspace-node    | 2022-12-13 17:04:45 [PrimaryChain] 📦 Highest known block at #0    
subspace-node    | 2022-12-13 17:04:45 [PrimaryChain] Running JSON-RPC HTTP server: addr=127.0.0.1:9933, allowed origins=None    
subspace-node    | 2022-12-13 17:04:45 [PrimaryChain] Running JSON-RPC WS server: addr=0.0.0.0:9944, allowed origins=None    
subspace-node    | 2022-12-13 17:04:45 [PrimaryChain] 〽️ Prometheus exporter started at 127.0.0.1:9615    
subspace-node    | 2022-12-13 17:04:46 [PrimaryChain] 🔍 Discovered new external address for our node: /ip4/REDACTED/tcp/30333/p2p/12D3KooWGTPLpX7kPnwr7gLdBTguhWrxNz8J5h41GSDzoBfN8Vey    
subspace-node    | 2022-12-13 17:04:50 [PrimaryChain] ⚙️  Syncing, target=#114179 (31 peers), best: #12 (0x3a86…cc4b), finalized #0 (0xcabf…b605), ⬇ 1.1MiB/s ⬆ 25.6kiB/s    
subspace-node    | 2022-12-13 17:04:55 [PrimaryChain] ⚙️  Syncing  4.4 bps, target=#114180 (34 peers), best: #34 (0x696c…2d7f), finalized #0 (0xcabf…b605), ⬇ 90.3kiB/s ⬆ 4.2kiB/s    
subspace-node    | 2022-12-13 17:05:00 [PrimaryChain] ⚙️  Syncing  3.6 bps, target=#114179 (35 peers), best: #52 (0x6a47…dad3), finalized #0 (0xcabf…b605), ⬇ 20.0kiB/s ⬆ 3.6kiB/s    
subspace-node    | 2022-12-13 17:05:05 [PrimaryChain] ⚙️  Syncing  3.9 bps, target=#114181 (35 peers), best: #72 (0xbbe2…bd54), finalized #0 (0xcabf…b605), ⬇ 23.8kiB/s ⬆ 3.4kiB/s    
subspace-node    | 2022-12-13 17:05:10 [PrimaryChain] ⚙️  Syncing  3.7 bps, target=#114179 (36 peers), best: #91 (0x0473…bee4), finalized #0 (0xcabf…b605), ⬇ 24.4kiB/s ⬆ 3.9kiB/s    
subspace-farmer  | 2022-12-13T17:05:11.204323Z  INFO subspace_farmer::utils: Increase file limit from soft to hard (limit is 1048576)
subspace-farmer  | 2022-12-13T17:05:11.209615Z  INFO subspace_farmer::commands::farm: Connecting to node RPC at ws://subspace-node:9944
subspace-farmer  | 2022-12-13T17:05:11.212358Z  INFO jsonrpsee_client_transport::ws: Connection established to target: Target { sockaddrs: [], host: "subspace-node", host_header: "subspace-node:9944", _mode: Plain, path_and_query: "/" }
subspace-farmer  | 2022-12-13T17:05:11.214481Z  INFO subspace_farmer::commands::farm: Record cache DB configured. record_cache_db_path="/var/subspace/records_cache_db" record_cache_size=65536
subspace-farmer  | 2022-12-13T17:05:11.214688Z  WARN jsonrpsee_core::client::async_client: Custom("[backend]: frontend dropped; terminate client")
subspace-farmer  | 2022-12-13T17:05:11.218470Z  INFO subspace_networking::behavior::custom_record_store: New record cache initialized.
subspace-farmer  | 2022-12-13T17:05:11.219659Z  INFO subspace_networking::create: DSN instance configured. allow_non_global_addresses_in_dht=true peer_id=12D3KooWCLz6Tkr6cJbGhGKxq6bp3GfPGWnHqXyH2rMkPPMeJWH8
subspace-farmer  | 2022-12-13T17:05:11.221028Z  INFO subspace_farmer::commands::farm: Connecting to node RPC at ws://subspace-node:9944
subspace-farmer  | 2022-12-13T17:05:11.223337Z  INFO jsonrpsee_client_transport::ws: Connection established to target: Target { sockaddrs: [], host: "subspace-node", host_header: "subspace-node:9944", _mode: Plain, path_and_query: "/" }
subspace-node    | 2022-12-13 17:05:11 Accepting new connection 1/100
subspace-node    | 2022-12-13 17:05:11 Accepting new connection 1/100
subspace-farmer  | 2022-12-13T17:05:11.405051Z  INFO subspace_farmer::single_disk_plot: Subscribing to slot info notifications
subspace-farmer  | 2022-12-13T17:05:11.405336Z  INFO subspace_farmer::reward_signing: Subscribing to reward signing notifications
subspace-farmer  | 2022-12-13T17:05:12.410208Z  INFO single_disk_plot{single_disk_plot_id=01GM6752HTY9C6FMEBH2GRDNT2}: subspace_farmer::single_disk_plot::piece_receiver: get_value returned no piece-by-sector provider piece_index=77 key=Multihash { code: 11771130, size: 32, digest: [160, 128, 92, 136, 43, 61, 174, 82, 142, 254, 90, 72, 48, 218, 38, 230, 150, 108, 38, 73, 51, 241, 122, 178, 179, 231, 34, 131, 108, 250, 230, 233, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] }
subspace-farmer  | 2022-12-13T17:05:12.410292Z  WARN single_disk_plot{single_disk_plot_id=01GM6752HTY9C6FMEBH2GRDNT2}: subspace_farmer::single_disk_plot::piece_receiver: Couldn't get a piece from DSN. Retrying... piece_index=77
subspace-farmer  | 2022-12-13T17:05:12.410351Z  INFO single_disk_plot{single_disk_plot_id=01GM6752HTY9C6FMEBH2GRDNT2}: subspace_farmer::single_disk_plot::piece_receiver: get_value returned no piece-by-sector provider piece_index=22 key=Multihash { code: 11771130, size: 32, digest: [183, 193, 157, 61, 140, 162, 58, 135, 1, 241, 219, 208, 159, 226, 207, 155, 143, 197, 55, 168, 155, 94, 86, 172, 62, 186, 98, 109, 236, 143, 187, 159, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] }
subspace-farmer  | 2022-12-13T17:05:12.410360Z  WARN single_disk_plot{single_disk_plot_id=01GM6752HTY9C6FMEBH2GRDNT2}: subspace_farmer::single_disk_plot::piece_receiver: Couldn't get a piece from DSN. Retrying... piece_index=22
subspace-farmer  | 2022-12-13T17:05:12.410366Z  INFO single_disk_plot{single_disk_plot_id=01GM6752HTY9C6FMEBH2GRDNT2}: subspace_farmer::single_disk_plot::piece_receiver: get_value returned no piece-by-sector provider piece_index=103 key=Multihash { code: 11771130, size: 32, digest: [24, 192, 130, 146, 46, 101, 56, 129, 192, 204, 223, 122, 215, 2, 41, 1, 241, 47, 11, 45, 186, 46, 83, 54, 241, 92, 86, 44, 75, 24, 156, 41, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] }
subspace-farmer  | 2022-12-13T17:05:12.410373Z  WARN single_disk_plot{single_disk_plot_id=01GM6752HTY9C6FMEBH2GRDNT2}: subspace_farmer::single_disk_plot::piece_receiver: Couldn't get a piece from DSN. Retrying... piece_index=103
subspace-farmer  | 2022-12-13T17:05:12.410395Z  INFO single_disk_plot{single_disk_plot_id=01GM6752HTY9C6FMEBH2GRDNT2}: subspace_farmer::single_disk_plot::piece_receiver: get_value returned no piece-by-sector provider piece_index=165 key=Multihash { code: 11771130, size: 32, digest: [46, 119, 158, 232, 142, 184, 252, 9, 64, 79, 64, 219, 190, 168, 63, 25, 237, 190, 41, 183, 95, 126, 10, 162, 7, 236, 76, 223, 135, 237, 95, 104, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] }
subspace-farmer  | 2022-12-13T17:05:12.410402Z  WARN single_disk_plot{single_disk_plot_id=01GM6752HTY9C6FMEBH2GRDNT2}: subspace_farmer::single_disk_plot::piece_receiver: Couldn't get a piece from DSN. Retrying... piece_index=165
subspace-farmer  | 2022-12-13T17:05:12.410425Z  INFO single_disk_plot{single_disk_plot_id=01GM6752HTY9C6FMEBH2GRDNT2}: subspace_farmer::single_disk_plot::piece_receiver: get_value returned no piece-by-sector provider piece_index=218 key=Multihash { code: 11771130, size: 32, digest: [56, 182, 124, 113, 53, 42, 37, 203, 139, 190, 168, 181, 131, 215, 146, 73, 12, 169, 50, 164, 87, 169, 72, 116, 249, 24, 0, 245, 29, 164, 122, 168, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] }
subspace-farmer  | 2022-12-13T17:05:12.410435Z  WARN single_disk_plot{single_disk_plot_id=01GM6752HTY9C6FMEBH2GRDNT2}: subspace_farmer::single_disk_plot::piece_receiver: Couldn't get a piece from DSN. Retrying... piece_index=218
subspace-farmer  | 2022-12-13T17:05:12.410440Z  INFO single_disk_plot{single_disk_plot_id=01GM6752HTY9C6FMEBH2GRDNT2}: subspace_farmer::single_disk_plot::piece_receiver: get_value returned no piece-by-sector provider piece_index=60 key=Multihash { code: 11771130, size: 32, digest: [39, 123, 215, 194, 18, 122, 94, 12, 183, 255, 32, 104, 136, 77, 24, 202, 46, 137, 137, 233, 108, 217, 177, 75, 124, 225, 52, 10, 32, 155, 122, 173, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] }
subspace-farmer  | 2022-12-13T17:05:12.410447Z  WARN single_disk_plot{single_disk_plot_id=01GM6752HTY9C6FMEBH2GRDNT2}: subspace_farmer::single_disk_plot::piece_receiver: Couldn't get a piece from DSN. Retrying... piece_index=60
subspace-farmer  | 2022-12-13T17:05:12.410451Z  INFO single_disk_plot{single_disk_plot_id=01GM6752HTY9C6FMEBH2GRDNT2}: subspace_farmer::single_disk_plot::piece_receiver: get_value returned no piece-by-sector provider piece_index=34 key=Multihash { code: 11771130, size: 32, digest: [148, 137, 59, 84, 230, 43, 163, 109, 108, 30, 133, 234, 149, 170, 100, 200, 59, 251, 186, 74, 228, 170, 173, 191, 182, 60, 50, 214, 52, 18, 27, 76, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] }
subspace-farmer  | 2022-12-13T17:05:12.410458Z  WARN single_disk_plot{single_disk_plot_id=01GM6752HTY9C6FMEBH2GRDNT2}: subspace_farmer::single_disk_plot::piece_receiver: Couldn't get a piece from DSN. Retrying... piece_index=34
subspace-farmer  | 2022-12-13T17:05:12.410462Z  INFO single_disk_plot{single_disk_plot_id=01GM6752HTY9C6FMEBH2GRDNT2}: subspace_farmer::single_disk_plot::piece_receiver: get_value returned no piece-by-sector provider piece_index=122 key=Multihash { code: 11771130, size: 32, digest: [165, 25, 99, 234, 240, 52, 149, 52, 35, 128, 66, 22, 80, 203, 47, 56, 69, 131, 237, 22, 185, 204, 212, 49, 183, 94, 4, 114, 43, 171, 1, 177, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] }
subspace-farmer  | 2022-12-13T17:05:12.410469Z  WARN single_disk_plot{single_disk_plot_id=01GM6752HTY9C6FMEBH2GRDNT2}: subspace_farmer::single_disk_plot::piece_receiver: Couldn't get a piece from DSN. Retrying... piece_index=122
subspace-farmer  | 2022-12-13T17:05:12.410473Z  INFO single_disk_plot{single_disk_plot_id=01GM6752HTY9C6FMEBH2GRDNT2}: subspace_farmer::single_disk_plot::piece_receiver: get_value returned no piece-by-sector provider piece_index=16 key=Multihash { code: 11771130, size: 32, digest: [232, 62, 240, 151, 7, 5, 132, 246, 43, 240, 105, 185, 185, 164, 196, 12, 132, 192, 176, 60, 8, 74, 200, 253, 95, 46, 116, 141, 77, 7, 204, 219, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] }

jim-counter · December 13, 2022, 5:21pm

And another one…

subspace-node    | 2022-12-13 17:18:40 Subspace    
subspace-node    | 2022-12-13 17:18:40 ✌️  version 0.1.0-d06ad0bad8a9cac203d0e0a9b27611b9f0f48e2a    
subspace-node    | 2022-12-13 17:18:40 ❤️  by Subspace Labs <https://subspace.network>, 2021-2022    
subspace-node    | 2022-12-13 17:18:40 📋 Chain specification: Subspace Gemini 3a    
subspace-node    | 2022-12-13 17:18:40 🏷  Node name: counterpoint_server_germany    
subspace-node    | 2022-12-13 17:18:40 👤 Role: AUTHORITY    
subspace-node    | 2022-12-13 17:18:40 💾 Database: ParityDb at /var/subspace/chains/subspace_gemini_3a/paritydb/full    
subspace-node    | 2022-12-13 17:18:40 ⛓  Native runtime: subspace-0 (subspace-0.tx0.au0)    
subspace-node    | 2022-12-13 17:18:40 [PrimaryChain] 🔨 Initializing Genesis block/state (state: 0xdd09…c28f, header-hash: 0xcabf…b605)    
subspace-node    | 2022-12-13 17:18:40 [DSN] DSN instance configured. allow_non_global_addresses_in_dht=true peer_id=12D3KooWPA5mJMpm44fCAYTvLk4nvi7QWeWdmnUmG2PnfRVcoE8z
subspace-node    | 2022-12-13 17:18:40 [DSN] Subspace networking initialized: Node ID is 12D3KooWPA5mJMpm44fCAYTvLk4nvi7QWeWdmnUmG2PnfRVcoE8z
subspace-node    | 2022-12-13 17:18:40 [PrimaryChain] Starting archiving from genesis    
subspace-node    | 2022-12-13 17:18:40 [PrimaryChain] Archiving already produced blocks 0..=0    
subspace-node    | 2022-12-13 17:18:41 [DSN] Processing a segment. segment_index=0
subspace-node    | 2022-12-13 17:18:41 [DSN] Segment processed. segment_index=0
subspace-node    | 2022-12-13 17:18:41 [PrimaryChain] 🏷  Local node identity is: 12D3KooWPA5mJMpm44fCAYTvLk4nvi7QWeWdmnUmG2PnfRVcoE8z    
subspace-node    | 2022-12-13 17:18:41 [PrimaryChain] 🧑‍🌾 Starting Subspace Authorship worker    
subspace-node    | 2022-12-13 17:18:41 [PrimaryChain] 💻 Operating system: linux    
subspace-node    | 2022-12-13 17:18:41 [PrimaryChain] 💻 CPU architecture: x86_64    
subspace-node    | 2022-12-13 17:18:41 [PrimaryChain] 💻 Target environment: gnu    
subspace-node    | 2022-12-13 17:18:41 [PrimaryChain] 💻 CPU: AMD Ryzen 9 5950X 16-Core Processor    
subspace-node    | 2022-12-13 17:18:41 [PrimaryChain] 💻 CPU cores: 16    
subspace-node    | 2022-12-13 17:18:41 [PrimaryChain] 💻 Memory: 128743MB    
subspace-node    | 2022-12-13 17:18:41 [PrimaryChain] 💻 Kernel: 5.15.0-56-generic    
subspace-node    | 2022-12-13 17:18:41 [PrimaryChain] 💻 Linux distribution: Ubuntu 20.04.5 LTS    
subspace-node    | 2022-12-13 17:18:41 [PrimaryChain] 💻 Virtual machine: no    
subspace-node    | 2022-12-13 17:18:41 [PrimaryChain] 📦 Highest known block at #0    
subspace-node    | 2022-12-13 17:18:41 [PrimaryChain] 〽️ Prometheus exporter started at 0.0.0.0:9615    
subspace-node    | 2022-12-13 17:18:41 [PrimaryChain] Running JSON-RPC HTTP server: addr=127.0.0.1:9933, allowed origins=None    
subspace-node    | 2022-12-13 17:18:41 [PrimaryChain] Running JSON-RPC WS server: addr=0.0.0.0:9944, allowed origins=None    
subspace-node    | 2022-12-13 17:18:42 [PrimaryChain] 🔍 Discovered new external address for our node: /ip4/REDACTED/tcp/30333/p2p/12D3KooWPA5mJMpm44fCAYTvLk4nvi7QWeWdmnUmG2PnfRVcoE8z    
subspace-node    | 2022-12-13 17:18:46 [PrimaryChain] ⚙️  Syncing, target=#114334 (35 peers), best: #39 (0x7ab8…ab9c), finalized #0 (0xcabf…b605), ⬇ 1.1MiB/s ⬆ 26.1kiB/s    
subspace-node    | 2022-12-13 17:18:51 [PrimaryChain] ⚙️  Syncing  6.6 bps, target=#114333 (35 peers), best: #72 (0xbbe2…bd54), finalized #0 (0xcabf…b605), ⬇ 75.7kiB/s ⬆ 1.4kiB/s    
subspace-node    | 2022-12-13 17:18:56 [PrimaryChain] ⚙️  Syncing  7.0 bps, target=#114333 (35 peers), best: #107 (0x5e23…fae9), finalized #7 (0xd6e7…7738), ⬇ 403.5kiB/s ⬆ 0.4kiB/s    
subspace-node    | 2022-12-13 17:19:01 [PrimaryChain] ⚙️  Syncing 11.0 bps, target=#114333 (35 peers), best: #162 (0x8a63…be88), finalized #62 (0x046d…e6f0), ⬇ 39.1kiB/s ⬆ 1.7kiB/s    
subspace-node    | 2022-12-13 17:19:06 [PrimaryChain] ⚙️  Syncing  8.0 bps, target=#114333 (35 peers), best: #202 (0xd4b8…edfe), finalized #102 (0xc79e…9c20), ⬇ 28.0kiB/s ⬆ 0.7kiB/s    
subspace-node    | 2022-12-13 17:19:11 Accepting new connection 1/100
subspace-node    | 2022-12-13 17:19:11 Accepting new connection 1/100
subspace-node    | 2022-12-13 17:19:11 [PrimaryChain] ⚙️  Syncing  5.8 bps, target=#114336 (35 peers), best: #231 (0x2101…a67e), finalized #131 (0xe7f8…0752), ⬇ 70.0kiB/s ⬆ 1.2kiB/s    
subspace-node    | 2022-12-13 17:19:16 [PrimaryChain] ⚙️  Syncing  8.2 bps, target=#114336 (35 peers), best: #272 (0x3153…ec32), finalized #172 (0x9ef7…4e2e), ⬇ 27.9kiB/s ⬆ 1.6kiB/s    
subspace-node    | 2022-12-13 17:19:21 [PrimaryChain] ⚙️  Syncing 14.0 bps, target=#114337 (35 peers), best: #342 (0x87f4…f516), finalized #242 (0x243d…be78), ⬇ 12.8kiB/s ⬆ 0.5kiB/s    
subspace-node    | 2022-12-13 17:19:26 [PrimaryChain] ⚙️  Syncing 12.4 bps, target=#114337 (35 peers), best: #404 (0xae48…a0b4), finalized #304 (0xa531…1bca), ⬇ 6.1kiB/s ⬆ 0.4kiB/s    
subspace-farmer  | 2022-12-13T17:19:11.439085Z  INFO subspace_farmer::utils: Increase file limit from soft to hard (limit is 1048576)
subspace-farmer  | 2022-12-13T17:19:11.440667Z  INFO subspace_farmer::commands::farm: Connecting to node RPC at ws://subspace-node:9944
subspace-farmer  | 2022-12-13T17:19:11.441180Z  INFO jsonrpsee_client_transport::ws: Connection established to target: Target { sockaddrs: [], host: "subspace-node", host_header: "subspace-node:9944", _mode: Plain, path_and_query: "/" }
subspace-farmer  | 2022-12-13T17:19:11.441589Z  INFO subspace_farmer::commands::farm: Record cache DB configured. record_cache_db_path="/var/subspace/records_cache_db" record_cache_size=65536
subspace-farmer  | 2022-12-13T17:19:11.441593Z  WARN jsonrpsee_core::client::async_client: Custom("[backend]: frontend dropped; terminate client")
subspace-farmer  | 2022-12-13T17:19:11.442156Z  INFO subspace_networking::behavior::custom_record_store: New record cache initialized.
subspace-farmer  | 2022-12-13T17:19:11.442361Z  INFO subspace_networking::create: DSN instance configured. allow_non_global_addresses_in_dht=true peer_id=12D3KooWCT95ewj2hkDJN5EmmaZeYHikLSmfsXtg4iR553KFX9LW
subspace-farmer  | 2022-12-13T17:19:11.442828Z  INFO subspace_farmer::commands::farm: Connecting to node RPC at ws://subspace-node:9944
subspace-farmer  | 2022-12-13T17:19:11.443077Z  INFO jsonrpsee_client_transport::ws: Connection established to target: Target { sockaddrs: [], host: "subspace-node", host_header: "subspace-node:9944", _mode: Plain, path_and_query: "/" }
subspace-farmer  | 2022-12-13T17:19:11.476384Z  INFO subspace_farmer::single_disk_plot: Subscribing to slot info notifications
subspace-farmer  | 2022-12-13T17:19:11.476400Z  INFO subspace_farmer::reward_signing: Subscribing to reward signing notifications
subspace-farmer  | 2022-12-13T17:19:12.478785Z  INFO single_disk_plot{single_disk_plot_id=01GM67YQ2MPBDFQ6YRC70RCCQF}: subspace_farmer::single_disk_plot::piece_receiver: get_value returned no piece-by-sector provider piece_index=241 key=Multihash { code: 11771130, size: 32, digest: [40, 113, 75, 154, 44, 35, 226, 128, 240, 193, 231, 217, 86, 199, 52, 64, 182, 33, 199, 196, 33, 151, 46, 58, 158, 236, 96, 76, 185, 162, 115, 184, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] }
subspace-farmer  | 2022-12-13T17:19:12.478809Z  WARN single_disk_plot{single_disk_plot_id=01GM67YQ2MPBDFQ6YRC70RCCQF}: subspace_farmer::single_disk_plot::piece_receiver: Couldn't get a piece from DSN. Retrying... piece_index=241
subspace-farmer  | 2022-12-13T17:19:12.478823Z  INFO single_disk_plot{single_disk_plot_id=01GM67YQ2MPBDFQ6YRC70RCCQF}: subspace_farmer::single_disk_plot::piece_receiver: get_value returned no piece-by-sector provider piece_index=9 key=Multihash { code: 11771130, size: 32, digest: [94, 103, 203, 218, 236, 123, 235, 65, 113, 254, 125, 188, 51, 224, 41, 192, 20, 119, 76, 189, 107, 34, 241, 37, 239, 202, 72, 111, 118, 100, 160, 255, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] }
subspace-farmer  | 2022-12-13T17:19:12.478829Z  WARN single_disk_plot{single_disk_plot_id=01GM67YQ2MPBDFQ6YRC70RCCQF}: subspace_farmer::single_disk_plot::piece_receiver: Couldn't get a piece from DSN. Retrying... piece_index=9
subspace-farmer  | 2022-12-13T17:19:12.478835Z  INFO single_disk_plot{single_disk_plot_id=01GM67YQ2MPBDFQ6YRC70RCCQF}: subspace_farmer::single_disk_plot::piece_receiver: get_value returned no piece-by-sector provider piece_index=245 key=Multihash { code: 11771130, size: 32, digest: [152, 58, 7, 192, 170, 146, 168, 45, 103, 31, 36, 195, 219, 17, 18, 185, 213, 164, 95, 249, 90, 232, 47, 79, 200, 137, 141, 86, 249, 199, 202, 117, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] }
subspace-farmer  | 2022-12-13T17:19:12.478840Z  WARN single_disk_plot{single_disk_plot_id=01GM67YQ2MPBDFQ6YRC70RCCQF}: subspace_farmer::single_disk_plot::piece_receiver: Couldn't get a piece from DSN. Retrying... piece_index=245
subspace-farmer  | 2022-12-13T17:19:12.478876Z  INFO single_disk_plot{single_disk_plot_id=01GM67YQ2MPBDFQ6YRC70RCCQF}: subspace_farmer::single_disk_plot::piece_receiver: get_value returned no piece-by-sector provider piece_index=106 key=Multihash { code: 11771130, size: 32, digest: [248, 188, 13, 75, 227, 47, 120, 20, 123, 243, 134, 73, 22, 254, 221, 172, 203, 133, 223, 46, 218, 50, 89, 51, 133, 76, 210, 107, 188, 172, 180, 128, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] }
subspace-farmer  | 2022-12-13T17:19:12.478882Z  WARN single_disk_plot{single_disk_plot_id=01GM67YQ2MPBDFQ6YRC70RCCQF}: subspace_farmer::single_disk_plot::piece_receiver: Couldn't get a piece from DSN. Retrying... piece_index=106
subspace-farmer  | 2022-12-13T17:19:12.478943Z  INFO single_disk_plot{single_disk_plot_id=01GM67YQ2MPBDFQ6YRC70RCCQF}: subspace_farmer::single_disk_plot::piece_receiver: get_value returned no piece-by-sector provider piece_index=195 key=Multihash { code: 11771130, size: 32, digest: [251, 232, 85, 157, 156, 69, 130, 159, 201, 205, 87, 98, 31, 188, 211, 199, 241, 59, 96, 164, 19, 80, 118, 151, 42, 197, 125, 71, 16, 52, 34, 168, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] }
subspace-farmer  | 2022-12-13T17:19:12.478949Z  WARN single_disk_plot{single_disk_plot_id=01GM67YQ2MPBDFQ6YRC70RCCQF}: subspace_farmer::single_disk_plot::piece_receiver: Couldn't get a piece from DSN. Retrying... piece_index=195
subspace-farmer  | 2022-12-13T17:19:12.478952Z  INFO single_disk_plot{single_disk_plot_id=01GM67YQ2MPBDFQ6YRC70RCCQF}: subspace_farmer::single_disk_plot::piece_receiver: get_value returned no piece-by-sector provider piece_index=35 key=Multihash { code: 11771130, size: 32, digest: [37, 75, 24, 122, 248, 181, 162, 128, 141, 61, 54, 222, 119, 43, 186, 7, 61, 22, 58, 127, 22, 73, 241, 75, 102, 102, 20, 61, 15, 114, 115, 255, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] }
subspace-farmer  | 2022-12-13T17:19:12.478958Z  WARN single_disk_plot{single_disk_plot_id=01GM67YQ2MPBDFQ6YRC70RCCQF}: subspace_farmer::single_disk_plot::piece_receiver: Couldn't get a piece from DSN. Retrying... piece_index=35
subspace-farmer  | 2022-12-13T17:19:12.478962Z  INFO single_disk_plot{single_disk_plot_id=01GM67YQ2MPBDFQ6YRC70RCCQF}: subspace_farmer::single_disk_plot::piece_receiver: get_value returned no piece-by-sector provider piece_index=24 key=Multihash { code: 11771130, size: 32, digest: [247, 45, 186, 229, 4, 219, 147, 98, 71, 247, 188, 104, 1, 199, 57, 92, 45, 143, 175, 174, 100, 241, 124, 21, 202, 124, 56, 151, 172, 130, 99, 102, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] }
subspace-farmer  | 2022-12-13T17:19:12.478968Z  WARN single_disk_plot{single_disk_plot_id=01GM67YQ2MPBDFQ6YRC70RCCQF}: subspace_farmer::single_disk_plot::piece_receiver: Couldn't get a piece from DSN. Retrying... piece_index=24
subspace-farmer  | 2022-12-13T17:19:12.478972Z  INFO single_disk_plot{single_disk_plot_id=01GM67YQ2MPBDFQ6YRC70RCCQF}: subspace_farmer::single_disk_plot::piece_receiver: get_value returned no piece-by-sector provider piece_index=181 key=Multihash { code: 11771130, size: 32, digest: [10, 241, 157, 86, 57, 1, 243, 12, 247, 190, 20, 12, 222, 69, 206, 1, 212, 181, 234, 57, 180, 105, 63, 28, 196, 146, 84, 63, 98, 87, 220, 88, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] }
subspace-farmer  | 2022-12-13T17:19:12.478977Z  WARN single_disk_plot{single_disk_plot_id=01GM67YQ2MPBDFQ6YRC70RCCQF}: subspace_farmer::single_disk_plot::piece_receiver: Couldn't get a piece from DSN. Retrying... piece_index=181
subspace-farmer  | 2022-12-13T17:19:12.478981Z  INFO single_disk_plot{single_disk_plot_id=01GM67YQ2MPBDFQ6YRC70RCCQF}: subspace_farmer::single_disk_plot::piece_receiver: get_value returned no piece-by-sector provider piece_index=249 key=Multihash { code: 11771130, size: 32, digest: [66, 55, 134, 92, 193, 210, 38, 162, 164, 5, 250, 81, 33, 135, 4, 2, 37, 56, 80, 219, 168, 80, 36, 42, 102, 70, 42, 182, 216, 97, 124, 191, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] }
subspace-farmer  | 2022-12-13T17:19:12.478986Z  WARN single_disk_plot{single_disk_plot_id=01GM67YQ2MPBDFQ6YRC70RCCQF}: subspace_farmer::single_disk_plot::piece_receiver: Couldn't get a piece from DSN. Retrying... piece_index=249
subspace-farmer  | 2022-12-13T17:19:12.478990Z  INFO single_disk_plot{single_disk_plot_id=01GM67YQ2MPBDFQ6YRC70RCCQF}: subspace_farmer::single_disk_plot::piece_receiver: get_value returned no piece-by-sector provider piece_index=170 key=Multihash { code: 11771130, size: 32, digest: [94, 105, 244, 116, 103, 225, 148, 23, 52, 168, 129, 219, 88, 139, 191, 186, 226, 55, 22, 33, 41, 242, 252, 132, 112, 128, 164, 243, 221, 184, 162, 149, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] }
subspace-farmer  | 2022-12-13T17:19:12.478995Z  WARN single_disk_plot{single_disk_plot_id=01GM67YQ2MPBDFQ6YRC70RCCQF}: subspace_farmer::single_disk_plot::piece_receiver: Couldn't get a piece from DSN. Retrying... piece_index=170
subspace-farmer  | 2022-12-13T17:19:12.478999Z  INFO single_disk_plot{single_disk_plot_id=01GM67YQ2MPBDFQ6YRC70RCCQF}: subspace_farmer::single_disk_plot::piece_receiver: get_value returned no piece-by-sector provider piece_index=11 key=Multihash { code: 11771130, size: 32, digest: [83, 50, 194, 39, 103, 192, 178, 217, 125, 32, 22, 246, 168, 239, 120, 222, 173, 2, 155, 120, 7, 166, 176, 226, 172, 7, 43, 102, 21, 204, 92, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] }
subspace-farmer  | 2022-12-13T17:19:12.479004Z  WARN single_disk_plot{single_disk_plot_id=01GM67YQ2MPBDFQ6YRC70RCCQF}: subspace_farmer::single_disk_plot::piece_receiver: Couldn't get a piece from DSN. Retrying... piece_index=11
subspace-farmer  | 2022-12-13T17:19:12.479008Z  INFO single_disk_plot{single_disk_plot_id=01GM67YQ2MPBDFQ6YRC70RCCQF}: subspace_farmer::single_disk_plot::piece_receiver: get_value returned no piece-by-sector provider piece_index=247 key=Multihash { code: 11771130, size: 32, digest: [127, 186, 180, 46, 206, 152, 163, 228, 253, 69, 62, 102, 251, 127, 141, 16, 133, 181, 26, 113, 76, 131, 88, 237, 220, 190, 238, 152, 185, 53, 49, 153, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] }

zug · December 13, 2022, 6:18pm

and another one from me ( ±same environment but Ubuntu 20.04.5 LTS and Docker version 20.10.21 )

Attaching to subspace-farmer-1, subspace-node-1
subspace-node-1    | 2022-12-13 18:15:28 Subspace
subspace-node-1    | 2022-12-13 18:15:28 ✌️  version 0.1.0-d06ad0bad8a9cac203d0e0a9b27611b9f0f48e2a
subspace-node-1    | 2022-12-13 18:15:28 ❤️  by Subspace Labs <https://subspace.network>, 2021-2022
subspace-node-1    | 2022-12-13 18:15:28 📋 Chain specification: Subspace Gemini 3a
subspace-node-1    | 2022-12-13 18:15:28 🏷  Node name: zug-docker
subspace-node-1    | 2022-12-13 18:15:28 👤 Role: AUTHORITY
subspace-node-1    | 2022-12-13 18:15:28 💾 Database: ParityDb at /var/subspace/chains/subspace_gemini_3a/paritydb/full
subspace-node-1    | 2022-12-13 18:15:28 ⛓  Native runtime: subspace-0 (subspace-0.tx0.au0)
subspace-node-1    | 2022-12-13 18:15:28 [DSN] DSN instance configured. allow_non_global_addresses_in_dht=true peer_id=12D3KooWBqhJ3xFM9dUTu8LwPe66TscgorNBj6ktmpLEZJRofZko
subspace-node-1    | 2022-12-13 18:15:28 [DSN] Subspace networking initialized: Node ID is 12D3KooWBqhJ3xFM9dUTu8LwPe66TscgorNBj6ktmpLEZJRofZko
subspace-node-1    | 2022-12-13 18:15:32 [PrimaryChain] Last archived block 776
subspace-node-1    | 2022-12-13 18:15:32 [PrimaryChain] Archiving already produced blocks 777..=2952
subspace-node-1    | 2022-12-13 18:15:55 [PrimaryChain] 🏷  Local node identity is: 12D3KooWBqhJ3xFM9dUTu8LwPe66TscgorNBj6ktmpLEZJRofZko
subspace-node-1    | 2022-12-13 18:15:55 [PrimaryChain] 🧑‍🌾 Starting Subspace Authorship worker
subspace-node-1    | 2022-12-13 18:15:55 [PrimaryChain] 💻 Operating system: linux
subspace-node-1    | 2022-12-13 18:15:55 [PrimaryChain] 💻 CPU architecture: x86_64
subspace-node-1    | 2022-12-13 18:15:55 [PrimaryChain] 💻 Target environment: gnu
subspace-node-1    | 2022-12-13 18:15:55 [PrimaryChain] 💻 CPU: Intel Core Processor (Haswell, no TSX, IBRS)
subspace-node-1    | 2022-12-13 18:15:55 [PrimaryChain] 💻 CPU cores: 6
subspace-node-1    | 2022-12-13 18:15:55 [PrimaryChain] 💻 Memory: 7861MB
subspace-node-1    | 2022-12-13 18:15:55 [PrimaryChain] 💻 Kernel: 5.4.0-135-generic
subspace-node-1    | 2022-12-13 18:15:55 [PrimaryChain] 💻 Linux distribution: Ubuntu 20.04.5 LTS
subspace-node-1    | 2022-12-13 18:15:55 [PrimaryChain] 💻 Virtual machine: yes
subspace-node-1    | 2022-12-13 18:15:55 [PrimaryChain] 📦 Highest known block at #3052
subspace-node-1    | 2022-12-13 18:15:55 [PrimaryChain] 〽️ Prometheus exporter started at 127.0.0.1:9615
subspace-node-1    | 2022-12-13 18:15:55 [PrimaryChain] Running JSON-RPC HTTP server: addr=127.0.0.1:9933, allowed origins=None
subspace-node-1    | 2022-12-13 18:15:55 [PrimaryChain] Running JSON-RPC WS server: addr=0.0.0.0:9944, allowed origins=None
subspace-node-1    | 2022-12-13 18:15:56 [PrimaryChain] 🔍 Discovered new external address for our node: /ip4/62.106.104.15/tcp/30333/p2p/12D3KooWBqhJ3xFM9dUTu8LwPe66TscgorNBj6ktmpLEZJRofZko
subspace-farmer-1  | 2022-12-13T18:15:59.172254Z  INFO subspace_farmer::utils: Increase file limit from soft to hard (limit is 1048576)
subspace-farmer-1  | 2022-12-13T18:15:59.176059Z  INFO subspace_farmer::commands::farm: Connecting to node RPC at ws://node:9944
subspace-farmer-1  | 2022-12-13T18:15:59.177788Z  INFO jsonrpsee_client_transport::ws: Connection established to target: Target { sockaddrs: [], host: "node", host_header: "node:9944", _mode: Plain, path_and_query: "/" }
subspace-farmer-1  | 2022-12-13T18:15:59.178863Z  WARN jsonrpsee_core::client::async_client: Custom("[backend]: frontend dropped; terminate client")
subspace-farmer-1  | 2022-12-13T18:15:59.178901Z  INFO subspace_farmer::commands::farm: Record cache DB configured. record_cache_db_path="/var/subspace/records_cache_db" record_cache_size=65536
subspace-farmer-1  | 2022-12-13T18:15:59.180385Z  INFO subspace_networking::behavior::custom_record_store: New record cache initialized.
subspace-farmer-1  | 2022-12-13T18:15:59.181209Z  INFO subspace_networking::create: DSN instance configured. allow_non_global_addresses_in_dht=true peer_id=12D3KooWEzeHHgEmQt7W7pyv34dswBsduRoyDFbxSkVNmJ6nU34e
subspace-farmer-1  | 2022-12-13T18:15:59.182999Z  INFO subspace_farmer::commands::farm: Connecting to node RPC at ws://node:9944
subspace-farmer-1  | 2022-12-13T18:15:59.183726Z  INFO jsonrpsee_client_transport::ws: Connection established to target: Target { sockaddrs: [], host: "node", host_header: "node:9944", _mode: Plain, path_and_query: "/" }
subspace-farmer-1  | 2022-12-13T18:15:59.196267Z  INFO subspace_farmer::single_disk_plot: Subscribing to slot info notifications
subspace-farmer-1  | 2022-12-13T18:15:59.196292Z  INFO subspace_farmer::reward_signing: Subscribing to reward signing notifications
subspace-node-1    | 2022-12-13 18:15:59 Accepting new connection 1/100
subspace-node-1    | 2022-12-13 18:15:59 Accepting new connection 1/100
subspace-farmer-1  | 2022-12-13T18:16:00.200523Z  INFO single_disk_plot{single_disk_plot_id=01GM69TXKAG3R45R2SKCHVRND1}: subspace_farmer::single_disk_plot::piece_receiver: get_value returned no piece-by-sector provider piece_index=496 key=Multihash { code: 11771130, size: 32, digest: [37, 219, 125, 25, 103, 64, 74, 81, 149, 83, 132, 8, 66, 133, 92, 16, 103, 238, 87, 158, 254, 194, 38, 164, 160, 127, 248, 66, 171, 175, 116, 50, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] }
subspace-farmer-1  | 2022-12-13T18:16:00.200601Z  WARN single_disk_plot{single_disk_plot_id=01GM69TXKAG3R45R2SKCHVRND1}: subspace_farmer::single_disk_plot::piece_receiver: Couldn't get a piece from DSN. Retrying... piece_index=496
subspace-farmer-1  | 2022-12-13T18:16:00.200657Z  INFO single_disk_plot{single_disk_plot_id=01GM69TXKAG3R45R2SKCHVRND1}: subspace_farmer::single_disk_plot::piece_receiver: get_value returned no piece-by-sector provider piece_index=265 key=Multihash { code: 11771130, size: 32, digest: [188, 217, 114, 105, 139, 63, 188, 9, 32, 238, 89, 133, 66, 13, 29, 149, 113, 230, 202, 232, 30, 228, 116, 213, 190, 54, 206, 136, 104, 16, 207, 54, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] }
subspace-farmer-1  | 2022-12-13T18:16:00.200675Z  WARN single_disk_plot{single_disk_plot_id=01GM69TXKAG3R45R2SKCHVRND1}: subspace_farmer::single_disk_plot::piece_receiver: Couldn't get a piece from DSN. Retrying... piece_index=265
subspace-farmer-1  | 2022-12-13T18:16:00.200689Z  INFO single_disk_plot{single_disk_plot_id=01GM69TXKAG3R45R2SKCHVRND1}: subspace_farmer::single_disk_plot::piece_receiver: get_value returned no piece-by-sector provider piece_index=57 key=Multihash { code: 11771130, size: 32, digest: [55, 131, 186, 209, 7, 49, 109, 225, 59, 102, 248, 30, 240, 5, 213, 87, 134, 80, 4, 72, 17, 49, 201, 149, 246, 79, 156, 159, 15, 82, 128, 116, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] }
subspace-farmer-1  | 2022-12-13T18:16:00.200704Z  WARN single_disk_plot{single_disk_plot_id=01GM69TXKAG3R45R2SKCHVRND1}: subspace_farmer::single_disk_plot::piece_receiver: Couldn't get a piece from DSN. Retrying... piece_index=57
subspace-farmer-1  | 2022-12-13T18:16:00.200920Z  INFO single_disk_plot{single_disk_plot_id=01GM69TXKAG3R45R2SKCHVRND1}: subspace_farmer::single_disk_plot::piece_receiver: get_value returned no piece-by-sector provider piece_index=8 key=Multihash { code: 11771130, size: 32, digest: [72, 174, 155, 173, 71, 117, 135, 58, 15, 176, 114, 61, 253, 184, 42, 62, 73, 114, 166, 28, 216, 69, 18, 184, 36, 249, 38, 24, 90, 211, 176, 113, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] }
subspace-farmer-1  | 2022-12-13T18:16:00.200939Z  WARN single_disk_plot{single_disk_plot_id=01GM69TXKAG3R45R2SKCHVRND1}: subspace_farmer::single_disk_plot::piece_receiver: Couldn't get a piece from DSN. Retrying... piece_index=8
subspace-farmer-1  | 2022-12-13T18:16:00.200956Z  INFO single_disk_plot{single_disk_plot_id=01GM69TXKAG3R45R2SKCHVRND1}: subspace_farmer::single_disk_plot::piece_receiver: get_value returned no piece-by-sector provider piece_index=345 key=Multihash { code: 11771130, size: 32, digest: [89, 126, 11, 246, 224, 114, 244, 36, 169, 249, 81, 17, 127, 128, 59, 170, 188, 140, 121, 171, 229, 44, 68, 124, 6, 162, 254, 31, 100, 123, 119, 244, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] }
subspace-farmer-1  | 2022-12-13T18:16:00.200971Z  WARN single_disk_plot{single_disk_plot_id=01GM69TXKAG3R45R2SKCHVRND1}: subspace_farmer::single_disk_plot::piece_receiver: Couldn't get a piece from DSN. Retrying... piece_index=345
subspace-farmer-1  | 2022-12-13T18:16:00.200982Z  INFO single_disk_plot{single_disk_plot_id=01GM69TXKAG3R45R2SKCHVRND1}: subspace_farmer::single_disk_plot::piece_receiver: get_value returned no piece-by-sector provider piece_index=353 key=Multihash { code: 11771130, size: 32, digest: [30, 91, 237, 212, 158, 81, 89, 47, 187, 255, 42, 23, 119, 125, 1, 86, 230, 45, 64, 203, 46, 118, 255, 145, 213, 97, 68, 180, 81, 141, 67, 8, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] }
subspace-farmer-1  | 2022-12-13T18:16:00.200996Z  WARN single_disk_plot{single_disk_plot_id=01GM69TXKAG3R45R2SKCHVRND1}: subspace_farmer::single_disk_plot::piece_receiver: Couldn't get a piece from DSN. Retrying... piece_index=353
subspace-farmer-1  | 2022-12-13T18:16:00.201031Z  INFO single_disk_plot{single_disk_plot_id=01GM69TXKAG3R45R2SKCHVRND1}: subspace_farmer::single_disk_plot::piece_receiver: get_value returned no piece-by-sector provider piece_index=17 key=Multihash { code: 11771130, size: 32, digest: [198, 93, 39, 152, 252, 218, 106, 238, 80, 254, 222, 229, 4, 236, 1, 216, 180, 142, 55, 49, 122, 239, 77, 110, 126, 201, 101, 238, 116, 165, 7, 128, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] }
subspace-farmer-1  | 2022-12-13T18:16:00.201047Z  WARN single_disk_plot{single_disk_plot_id=01GM69TXKAG3R45R2SKCHVRND1}: subspace_farmer::single_disk_plot::piece_receiver: Couldn't get a piece from DSN. Retrying... piece_index=17
subspace-farmer-1  | 2022-12-13T18:16:00.201059Z  INFO single_disk_plot{single_disk_plot_id=01GM69TXKAG3R45R2SKCHVRND1}: subspace_farmer::single_disk_plot::piece_receiver: get_value returned no piece-by-sector provider piece_index=351 key=Multihash { code: 11771130, size: 32, digest: [217, 228, 60, 137, 56, 58, 247, 155, 140, 2, 131, 144, 221, 229, 255, 147, 195, 252, 238, 26, 162, 22, 28, 100, 78, 108, 109, 248, 13, 173, 7, 196, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] }
subspace-farmer-1  | 2022-12-13T18:16:00.201073Z  WARN single_disk_plot{single_disk_plot_id=01GM69TXKAG3R45R2SKCHVRND1}: subspace_farmer::single_disk_plot::piece_receiver: Couldn't get a piece from DSN. Retrying... piece_index=351
subspace-farmer-1  | 2022-12-13T18:16:00.201082Z  INFO single_disk_plot{single_disk_plot_id=01GM69TXKAG3R45R2SKCHVRND1}: subspace_farmer::single_disk_plot::piece_receiver: get_value returned no piece-by-sector provider piece_index=490 key=Multihash { code: 11771130, size: 32, digest: [108, 19, 193, 0, 11, 50, 146, 132, 37, 48, 181, 181, 149, 179, 97, 107, 253, 194, 78, 61, 149, 15, 198, 52, 120, 16, 116, 189, 248, 161, 251, 118, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] }
subspace-farmer-1  | 2022-12-13T18:16:00.201096Z  WARN single_disk_plot{single_disk_plot_id=01GM69TXKAG3R45R2SKCHVRND1}: subspace_farmer::single_disk_plot::piece_receiver: Couldn't get a piece from DSN. Retrying... piece_index=490
subspace-farmer-1  | 2022-12-13T18:16:00.201107Z  INFO single_disk_plot{single_disk_plot_id=01GM69TXKAG3R45R2SKCHVRND1}: subspace_farmer::single_disk_plot::piece_receiver: get_value returned no piece-by-sector provider piece_index=131 key=Multihash { code: 11771130, size: 32, digest: [59, 226, 184, 232, 163, 123, 53, 63, 109, 107, 52, 153, 255, 146, 69, 120, 175, 5, 76, 94, 26, 146, 153, 139, 193, 78, 169, 164, 87, 130, 152, 255, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] }
subspace-farmer-1  | 2022-12-13T18:16:00.201127Z  WARN single_disk_plot{single_disk_plot_id=01GM69TXKAG3R45R2SKCHVRND1}: subspace_farmer::single_disk_plot::piece_receiver: Couldn't get a piece from DSN. Retrying... piece_index=131
subspace-farmer-1  | 2022-12-13T18:16:00.201454Z  INFO single_disk_plot{single_disk_plot_id=01GM69TXKAG3R45R2SKCHVRND1}: subspace_farmer::single_disk_plot::piece_receiver: get_value returned no piece-by-sector provider piece_index=156 key=Multihash { code: 11771130, size: 32, digest: [174, 184, 173, 186, 189, 60, 100, 181, 218, 166, 7, 252, 25, 154, 112, 130, 107, 67, 60, 206, 184, 254, 220, 200, 156, 133, 120, 228, 255, 2, 240, 113, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] }
subspace-farmer-1  | 2022-12-13T18:16:00.201845Z  WARN single_disk_plot{single_disk_plot_id=01GM69TXKAG3R45R2SKCHVRND1}: subspace_farmer::single_disk_plot::piece_receiver: Couldn't get a piece from DSN. Retrying... piece_index=156
subspace-farmer-1  | 2022-12-13T18:16:00.201868Z  INFO single_disk_plot{single_disk_plot_id=01GM69TXKAG3R45R2SKCHVRND1}: subspace_farmer::single_disk_plot::piece_receiver: get_value returned no piece-by-sector provider piece_index=47 key=Multihash { code: 11771130, size: 32, digest: [67, 111, 85, 118, 145, 249, 54, 127, 23, 228, 87, 221, 135, 21, 76, 242, 154, 106, 101, 251, 57, 2, 61, 151, 239, 248, 135, 54, 219, 18, 32, 51, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0, 0] }
subspace-farmer-1  | 2022-12-13T18:16:00.201884Z  WARN single_disk_plot{single_disk_plot_id=01GM69TXKAG3R45R2SKCHVRND1}: subspace_farmer::single_disk_plot::piece_receiver: Couldn't get a piece from DSN. Retrying... piece_index=47
subspace-node-1    | 2022-12-13 18:16:00 [PrimaryChain] ⚙️  Syncing, target=#114976 (29 peers), best: #3073 (0x2df5…130e), finalized #2973 (0x6338…3557), ⬇ 1.5MiB/s ⬆ 23.8kiB/s

ImmaZoni · December 13, 2022, 7:05pm

Spoke with the development team regarding this bug. This actually isn’t a bug but rather a warning, it is something to be expected on a Decentralized Storage Network. Which this version utilizes, instead of the normal node sync. Because of this there is nothing you need to do as a user with this warning, its likely it will come up occasionally but as long as there aren’t other more catastrophic errors it can be ignored.

jim-counter · December 13, 2022, 7:18pm

Thanks for confirming. I actually had an error message on the original environment but after a restart it did not come back. I confused one being related to the other.

xorinox · January 26, 2023, 4:07pm

Hello, thanks for that explanation. The issue I have with it, the warning is posted hundreds of times (estimate) per second, which blows up any log file in no time (pipe stdout to log). If that warning is normal and expected, maybe it’s rather a debug message, that the user with default configuration does not need to see.

ALPHANOMICON · January 26, 2023, 5:07pm

Glad you got it figured out friend

edbaff1ed · March 18, 2023, 12:40pm

Is this still normal as of march 15 release? I get a deluge of these warnings. Farmer has been working for a few hours and spilled 0.5GB of these into the log file.

jim-counter · March 18, 2023, 2:18pm

Yes, these have been described as annoying but can be ignored. Though the log growth could be problematic which could end up more than just annoying.

jrwashburn · June 9, 2023, 5:10pm

In the subspace-farmer-ubuntu-x86_64-v3-gemini-3d-2023-may-23 build (and maybe earlier, but noticed now), this seems to have been upgraded from WARN to ERROR after 3 retries - at least on mine. e.g.

Jun 09 13:07:16 jwcop subspace-farmer[328887]: 2023-06-09T17:07:16.805785Z ERROR single_disk_plot{disk_farm_index=0}: subspace_networking::utils::piece_provider: Couldn't get a piece from DSN. No retries left. piece_index=2853734 current_attempt=3 max_retries=3
Jun 09 13:07:16 jwcop subspace-farmer[328887]: 2023-06-09T17:07:16.805813Z  INFO single_disk_plot{disk_farm_index=0}: subspace_farmer_components::segment_reconstruction: Recovering missing piece... missing_piece_index=2853734
Jun 09 13:07:22 jwcop subspace-farmer[328887]: 2023-06-09T17:07:22.462034Z ERROR single_disk_plot{disk_farm_index=0}: subspace_networking::utils::piece_provider: Couldn't get a piece from DSN. No retries left. piece_index=1768870 current_attempt=3 max_retries=3
Jun 09 13:07:22 jwcop subspace-farmer[328887]: 2023-06-09T17:07:22.462068Z  INFO single_disk_plot{disk_farm_index=0}: subspace_farmer_components::segment_reconstruction: Recovering missing piece... missing_piece_index=1768870
Jun 09 13:07:31 jwcop subspace-farmer[328887]: 2023-06-09T17:07:31.635648Z ERROR single_disk_plot{disk_farm_index=0}: subspace_farmer_components::segment_reconstruction: Recovering missing piece failed. missing_piece_index=2760575
Jun 09 13:07:40 jwcop subspace-farmer[328887]: 2023-06-09T17:07:40.781859Z ERROR single_disk_plot{disk_farm_index=0}: subspace_farmer_components::segment_reconstruction: Recovering missing piece failed. missing_piece_index=1946316
Jun 09 13:07:42 jwcop subspace-farmer[328887]: 2023-06-09T17:07:42.781325Z ERROR single_disk_plot{disk_farm_index=0}: subspace_farmer_components::segment_reconstruction: Recovering missing piece failed. missing_piece_index=3280899

BSV47 · June 13, 2023, 4:37pm

Topic		Replies	Views
Farming not working on gemini-3d Support	18	889	April 28, 2023
Stop accrual of coins Support gemini-3d	4	396	May 3, 2023
Farmers are unable to plot with the Advanced CLI Support farmer , gemini-3e , advanced-cli	9	435	July 14, 2023
Gemini 3c pre-release testing - how do i know plotting is working? Testing	5	618	February 24, 2023
Run farmer cultivator error Support	3	290	March 24, 2023