Failed read s-bucket error=Input/output error (os error 5)

Support
1

Issue Report

Environment

  • Operating System: Ubuntu 20.04.6 LTS
  • Pulsar/Advanced CLI/Docker: gemini-3g-2023-nov-09

Problem

At least one of my farmers suffers from this problem:

Nov 10 18:51:43 lenovo21 subspace-farmer[2301795]: 2023-11-10T15:51:43.892340Z  INFO single_disk_farm{disk_farm_index=1}: subspace_farmer::reward_signing: Successfully signed reward hash 0x8ee9bdc637d681c1fe61fc87f2453fd184d78d87d1cdfd656d25186d68aca9d7
Nov 10 18:51:55 lenovo21 subspace-farmer[2301795]: 2023-11-10T15:51:55.888569Z  WARN subspace_farmer_components::auditing: Failed read s-bucket error=Input/output error (os error 5) sector_index=33 s_bucket_audit_index=37656
Nov 10 18:52:23 lenovo21 subspace-farmer[2301795]: 2023-11-10T15:52:23.569195Z  INFO single_disk_farm{disk_farm_index=0}: subspace_farmer::single_disk_farm::plotting: Plotting sector (36.46% complete) sector_index=660
Nov 10 18:54:28 lenovo21 subspace-farmer[2301795]: 2023-11-10T15:54:28.965530Z  WARN subspace_farmer_components::auditing: Failed read s-bucket error=Input/output error (os error 5) sector_index=33 s_bucket_audit_index=35931
Nov 10 18:54:35 lenovo21 subspace-farmer[2301795]: 2023-11-10T15:54:35.976150Z  WARN subspace_farmer_components::auditing: Failed read s-bucket error=Input/output error (os error 5) sector_index=33 s_bucket_audit_index=16996
Nov 10 18:55:25 lenovo21 subspace-farmer[2301795]: 2023-11-10T15:55:25.981551Z  WARN subspace_farmer_components::auditing: Failed read s-bucket error=Input/output error (os error 5) sector_index=33 s_bucket_audit_index=47055
Nov 10 18:56:32 lenovo21 subspace-farmer[2301795]: 2023-11-10T15:56:32.012879Z  WARN subspace_farmer_components::auditing: Failed read s-bucket error=Input/output error (os error 5) sector_index=33 s_bucket_audit_index=30776
2

Will post my response from Discord here as well:

Looks like an issue with storage: either disk is broken, it was unplugged, or file system error or error with permissions. Try to look at kernel messages for clues (dmesg).

3

Sure.

[Fri Nov 10 22:56:22 2023] blk_update_request: critical medium error, dev nvme0n1, sector 69784680 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0
[Fri Nov 10 23:03:58 2023] blk_update_request: critical medium error, dev nvme0n1, sector 69539152 op 0x0:(READ) flags 0x0 phys_seg 1 prio class 0

Thanks.