thread 'plotting-1.1' panicked at crates/subspace-proof-of-space/src/chiapos/table.rs:305:49:
index out of bounds: the len is 15113 but the index is 18446744073019990790
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Error: Background task plotting-1 panicked
That index looks a bit large I would say.
Is this issue replicating or just a one off runtime? I would be curious what the rust backtrace may say to help the devs pinpoint where this error is coming from. (See this thread here for tips on this)
Can you provide some system specifications on the farmer in which this error occured? Particularly interested in cpu, ram, & ssd size, and plot size etc.
Happened for the first time. I have been running this system on and off for some time now. Might be just a glitch. RAM is pretty tight with 6400MHz and CL32 @ 64GB. So might be a bit flip or whatever.
AMD 7950X (no OC)
64GB DDR5-6400MHz CL32 (XMP profile, no OC)
4TB NVMe with following plots:
path=/media/max/NVME0/plot0,size=1000GiB \
path=/media/max/NVME0/plot4,size=2000GiB \
As long as this doesn’t happen again and not reported by anyone else, I guess it’s not an issue. Just wanted to report it here in case anyone else had this as well but ignored it.
After farmer restart no issues. I’ve set RUST_BACKTRACE=1 in case it ever happens again.
Duplicate of both Index out of bounds and Chia Pos Plotting Error?
Indication of unstable hardware, typically RAM. Subspace can be quite intense and uncovers instabilities that might otherwise result in very rare glitches that you would write off on solar flares.
Can this potentially ruin the whole block or even the plot?
It can result in broken sector plotted, but if your computer reads different data from RAM comparing to what was written, results of what computer does are quite literally unpredictable and broken sector (that can be fixed with scrub) is the least of your problems.
Agree. There might actually be many more errors which stay under radar like wrong FP operations which are not as easily caught as out-of-bound errors.
I’ll try to bump the RAM voltage slightly. It’s quite low anyways, something like 1.135V which is really not a lot considering the clocks of these particular RAM sticks.
In my case, it looks like the plot, which has been finished for a relatively long time, has been damaged. Because when I leave even one of the first two pilots, errors appear.
Feb 23 02:34:42 ironfish2 systemd[1]: subspacefarm.service: Scheduled restart job, restart counter is at 3.
Feb 23 02:34:42 ironfish2 systemd[1]: Stopped Subspace Farmer.
Feb 23 02:34:42 ironfish2 systemd[1]: subspacefarm.service: Consumed 1h 40min 45.382s CPU time.
Feb 23 02:34:42 ironfish2 systemd[1]: Started Subspace Farmer.
Feb 23 02:34:42 ironfish2 subspace-farmer[9893]: 2024-02-22T19:34:42.475070Z INFO subspace_farmer::commands::farm: Connecting to node RPC url=ws://127.0.0.1:9944
Feb 23 02:34:42 ironfish2 subspace-farmer[9893]: 2024-02-22T19:34:42.477061Z INFO subspace_networking::constructor: DSN instance configured. allow_non_global_addresses_in_dht=false peer_id=12D3KooWNirY8GEysoysW7rskZVHE6QxU23ghfmFJ7jrwmmRyZsf protocol_version=/subspace/2/0c121c75f4ef450f40619e1fca9d1e8e7fbabc42c895bc4790801e85d5a91c34
Feb 23 02:34:42 ironfish2 subspace-farmer[9893]: 2024-02-22T19:34:42.477579Z INFO libp2p_swarm: local_peer_id=12D3KooWNirY8GEysoysW7rskZVHE6QxU23ghfmFJ7jrwmmRyZsf
Feb 23 02:34:42 ironfish2 subspace-farmer[9893]: 2024-02-22T19:34:42.896275Z INFO subspace_farmer::commands::farm: Multiple L3 cache groups detected l3_cache_groups=2
Feb 23 02:34:42 ironfish2 subspace-farmer[9893]: 2024-02-22T19:34:42.896286Z INFO subspace_farmer::commands::farm: Regrouped CPU cores to match number of farms, more farms may leverage CPU more efficiently farms_count=1
Feb 23 02:34:42 ironfish2 subspace-farmer[9893]: Single disk farm 0:
Feb 23 02:34:42 ironfish2 subspace-farmer[9893]: ID: 01HNJH750BJTFNT9SPFDZN5679
Feb 23 02:34:42 ironfish2 subspace-farmer[9893]: Genesis hash: 0x0c121c75f4ef450f40619e1fca9d1e8e7fbabc42c895bc4790801e85d5a91c34
Feb 23 02:34:42 ironfish2 subspace-farmer[9893]: Public key: 0x2c62fa8f94c99dbda2ce7031adff65f1fb387ae81d18a6ba2b5df7cc2cd9737d
Feb 23 02:34:42 ironfish2 subspace-farmer[9893]: Allocated space: 931.3 GiB (1000.0 GB)
Feb 23 02:34:42 ironfish2 subspace-farmer[9893]: Directory: /home/rabinovitch/subspace-plot/plot0
Feb 23 02:34:42 ironfish2 subspace-farmer[9893]: 2024-02-22T19:34:42.981743Z INFO subspace_farmer::commands::farm: Collecting already plotted pieces (this will take some time)...
Feb 23 02:34:42 ironfish2 subspace-farmer[9893]: 2024-02-22T19:34:42.981756Z INFO subspace_farmer::farmer_cache: Initializing piece cache
Feb 23 02:34:43 ironfish2 subspace-farmer[9893]: 2024-02-22T19:34:43.116328Z INFO subspace_farmer::commands::farm: Finished collecting already plotted pieces successfully
Feb 23 02:34:43 ironfish2 subspace-farmer[9893]: 2024-02-22T19:34:43.116493Z INFO {disk_farm_index=0}: subspace_farmer::single_disk_farm::farming: Subscribing to slot info notifications
Feb 23 02:34:43 ironfish2 subspace-farmer[9893]: 2024-02-22T19:34:43.116515Z INFO {disk_farm_index=0}: subspace_farmer::reward_signing: Subscribing to reward signing notifications
Feb 23 02:34:43 ironfish2 subspace-farmer[9893]: 2024-02-22T19:34:43.116601Z INFO subspace_farmer::commands::farm::dsn: DSN listening on /ip4/127.0.0.1/udp/32533/quic-v1/p2p/12D3KooWNirY8GEysoysW7rskZVHE6QxU23ghfmFJ7jrwmmRyZsf
Feb 23 02:34:43 ironfish2 subspace-farmer[9893]: 2024-02-22T19:34:43.116649Z INFO {disk_farm_index=0}: subspace_farmer::single_disk_farm::plotting: Subscribing to archived segments
Feb 23 02:34:43 ironfish2 subspace-farmer[9893]: 2024-02-22T19:34:43.116750Z INFO subspace_farmer::commands::farm::dsn: DSN listening on /ip4/192.168.0.159/udp/32533/quic-v1/p2p/12D3KooWNirY8GEysoysW7rskZVHE6QxU23ghfmFJ7jrwmmRyZsf
Feb 23 02:34:43 ironfish2 subspace-farmer[9893]: 2024-02-22T19:34:43.116761Z INFO subspace_farmer::commands::farm::dsn: DSN listening on /ip4/172.17.0.1/udp/32533/quic-v1/p2p/12D3KooWNirY8GEysoysW7rskZVHE6QxU23ghfmFJ7jrwmmRyZsf
Feb 23 02:34:43 ironfish2 subspace-farmer[9893]: 2024-02-22T19:34:43.117150Z INFO subspace_farmer::commands::farm::dsn: DSN listening on /ip4/127.0.0.1/tcp/32533/p2p/12D3KooWNirY8GEysoysW7rskZVHE6QxU23ghfmFJ7jrwmmRyZsf
Feb 23 02:34:43 ironfish2 subspace-farmer[9893]: 2024-02-22T19:34:43.117164Z INFO subspace_farmer::commands::farm::dsn: DSN listening on /ip4/192.168.0.159/tcp/32533/p2p/12D3KooWNirY8GEysoysW7rskZVHE6QxU23ghfmFJ7jrwmmRyZsf
Feb 23 02:34:43 ironfish2 subspace-farmer[9893]: 2024-02-22T19:34:43.117173Z INFO subspace_farmer::commands::farm::dsn: DSN listening on /ip4/172.17.0.1/tcp/32533/p2p/12D3KooWNirY8GEysoysW7rskZVHE6QxU23ghfmFJ7jrwmmRyZsf
Feb 23 02:34:43 ironfish2 subspace-farmer[9893]: 2024-02-22T19:34:43.615445Z WARN subspace_farmer::single_disk_farm::piece_cache: Failed to read cache element error=Checksum mismatch offset=2438
Feb 23 02:34:44 ironfish2 subspace-farmer[9893]: 2024-02-22T19:34:44.228797Z WARN subspace_farmer::single_disk_farm::piece_cache: Failed to read cache element error=Checksum mismatch offset=4909
Feb 23 02:34:44 ironfish2 subspace-farmer[9893]: 2024-02-22T19:34:44.766205Z WARN subspace_farmer::single_disk_farm::piece_cache: Failed to read cache element error=Checksum mismatch offset=7134
Feb 23 02:34:45 ironfish2 subspace-farmer[9893]: 2024-02-22T19:34:45.481502Z INFO subspace_farmer::farmer_cache: Synchronizing piece cache
Feb 23 02:37:01 ironfish2 subspace-farmer[9893]: 2024-02-22T19:37:01.813050Z INFO subspace_farmer::farmer_cache: Finished piece cache synchronization
Feb 23 02:37:01 ironfish2 subspace-farmer[9893]: 2024-02-22T19:37:01.908728Z INFO {disk_farm_index=0}: subspace_farmer::single_disk_farm::plotting: Replotting sector (0.00% complete) sector_index=3
Feb 23 02:41:27 ironfish2 subspace-farmer[9893]: 2024-02-22T19:41:27.555545Z INFO {disk_farm_index=0}: subspace_farmer::single_disk_farm::plotting: Replotting sector (0.11% complete) sector_index=4
Feb 23 02:41:32 ironfish2 subspace-farmer[9893]: 2024-02-22T19:41:32.501603Z WARN {disk_farm_index=0}: subspace_farmer::utils::piece_validator: Received invalid piece from peer piece_index=30985 source_peer_id=12D3KooWQ3ni1oT3cwcv89Rwq3ECaCY5jdMjEX28BUAwDv1uHadm
Feb 23 02:42:14 ironfish2 subspace-farmer[9893]: 2024-02-22T19:42:14.889672Z WARN {disk_farm_index=0}: subspace_farmer::utils::piece_validator: Received invalid piece from peer piece_index=1329 source_peer_id=12D3KooWGvmXBuZw7tssB8MRjrY6Jw6yYykx8K5ULLGEwZBmw3Rn
Feb 23 02:42:45 ironfish2 subspace-farmer[9893]: 2024-02-22T19:42:45.700123Z WARN {disk_farm_index=0}: subspace_farmer::utils::piece_validator: Received invalid piece from peer piece_index=48130 source_peer_id=12D3KooWFVQep8q75dd1rS4gVFyvNj6v3oXjDHxh7AZ8CN3snFsS
Feb 23 02:43:16 ironfish2 subspace-farmer[9893]: thread 'replotting-0.9' panicked at crates/subspace-proof-of-space/src/chiapos/table.rs:287:16:
Feb 23 02:43:16 ironfish2 subspace-farmer[9893]: index out of bounds: the len is 15113 but the index is 18446744073709299218
Feb 23 02:43:16 ironfish2 subspace-farmer[9893]: note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
Feb 23 02:43:16 ironfish2 subspace-farmer[9893]: thread 'replotting-0.14' panicked at crates/subspace-proof-of-space/src/chiapos/table.rs:306:49:
Feb 23 02:43:16 ironfish2 subspace-farmer[9893]: index out of bounds: the len is 15113 but the index is 18446744073709299218
Feb 23 02:44:19 ironfish2 subspace-farmer[9893]: 2024-02-22T19:44:19.803473Z WARN {disk_farm_index=0}: subspace_farmer::single_disk_farm::plotting: Failed to send sector index for replotting error=send failed because receiver is gone
I don’t believe it is correct. Chia table is being constructed when this error appears, so the issue is that your computer is unstable, which causes sector creation to fail. Whether there were broken sectors in the plot prior is irrelevant, you need to fix the source of instability instead.
It turned out that one of the memory sticks was out of order.