More miss reward with Mar 11 release in Windows

dragonP · March 12, 2024, 3:58am

I’ve experienced very high miss reward rate across 16 PC’s, all using Win 11. Below is the benchmark result to start with. Please help to optimize this version.

NVMe benchmark result

`PS C:\Subspace Farmer> .\subspace-farmer3h_11Mar benchmark prove G:\1
Benchmarking prove/plot/rayon/unbuffered/concurrent-chunks
Benchmarking prove/plot/rayon/unbuffered/concurrent-chunks: Warming up for 3.0000 s
Benchmarking prove/plot/rayon/unbuffered/concurrent-chunks: Collecting 10 samples in estimated 7.1303 s (20 iterations)
Benchmarking prove/plot/rayon/unbuffered/concurrent-chunks: Analyzing
prove/plot/rayon/unbuffered/concurrent-chunks
                        time:   [353.43 ms 355.40 ms 357.63 ms]
Benchmarking prove/plot/rayon/unbuffered/whole-sector
Benchmarking prove/plot/rayon/unbuffered/whole-sector: Warming up for 3.0000 s

Warning: Unable to complete 10 samples in 5.0s. You may wish to increase target time to 8.9s.
Benchmarking prove/plot/rayon/unbuffered/whole-sector: Collecting 10 samples in estimated 8.9488 s (10 iterations)
Benchmarking prove/plot/rayon/unbuffered/whole-sector: Analyzing
prove/plot/rayon/unbuffered/whole-sector
                        time:   [888.21 ms 892.33 ms 896.64 ms]
Benchmarking prove/plot/rayon/regular/concurrent-chunks
Benchmarking prove/plot/rayon/regular/concurrent-chunks: Warming up for 3.0000 s
Benchmarking prove/plot/rayon/regular/concurrent-chunks: Collecting 10 samples in estimated 7.7924 s (20 iterations)
Benchmarking prove/plot/rayon/regular/concurrent-chunks: Analyzing
prove/plot/rayon/regular/concurrent-chunks
                        time:   [362.05 ms 366.33 ms 371.77 ms]
Found 1 outliers among 10 measurements (10.00%)
  1 (10.00%) high mild
Benchmarking prove/plot/rayon/regular/whole-sector
Benchmarking prove/plot/rayon/regular/whole-sector: Warming up for 3.0000 s

Warning: Unable to complete 10 samples in 5.0s. You may wish to increase target time to 8.9s.
Benchmarking prove/plot/rayon/regular/whole-sector: Collecting 10 samples in estimated 8.9295 s (10 iterations)
Benchmarking prove/plot/rayon/regular/whole-sector: Analyzing
prove/plot/rayon/regular/whole-sector
                        time:   [878.64 ms 884.00 ms 889.21 ms]`

SATA (directly connecting to mainboard) benchmark result

PS C:\Subspace Farmer> .\subspace-farmer3h_11Mar benchmark prove H:\1
Benchmarking prove/plot/rayon/unbuffered/concurrent-chunks
Benchmarking prove/plot/rayon/unbuffered/concurrent-chunks: Warming up for 3.0000 s

Warning: Unable to complete 10 samples in 5.0s. You may wish to increase target time to 10.6s.
Benchmarking prove/plot/rayon/unbuffered/concurrent-chunks: Collecting 10 samples in estimated 10.557 s (10 iterations)
Benchmarking prove/plot/rayon/unbuffered/concurrent-chunks: Analyzing
prove/plot/rayon/unbuffered/concurrent-chunks
                        time:   [963.75 ms 982.11 ms 1.0018 s]
                        change: [+170.70% +176.34% +182.10%] (p = 0.00 < 0.05)
                        Performance has regressed.
Benchmarking prove/plot/rayon/unbuffered/whole-sector
Benchmarking prove/plot/rayon/unbuffered/whole-sector: Warming up for 3.0000 s

Warning: Unable to complete 10 samples in 5.0s. You may wish to increase target time to 37.8s.
Benchmarking prove/plot/rayon/unbuffered/whole-sector: Collecting 10 samples in estimated 37.760 s (10 iterations)
Benchmarking prove/plot/rayon/unbuffered/whole-sector: Analyzing
prove/plot/rayon/unbuffered/whole-sector
                        time:   [3.3776 s 3.4614 s 3.5353 s]
                        change: [+276.90% +287.91% +296.56%] (p = 0.00 < 0.05)
                        Performance has regressed.
Benchmarking prove/plot/rayon/regular/concurrent-chunks
Benchmarking prove/plot/rayon/regular/concurrent-chunks: Warming up for 3.0000 s

Warning: Unable to complete 10 samples in 5.0s. You may wish to increase target time to 10.9s.
Benchmarking prove/plot/rayon/regular/concurrent-chunks: Collecting 10 samples in estimated 10.852 s (10 iterations)
Benchmarking prove/plot/rayon/regular/concurrent-chunks: Analyzing
prove/plot/rayon/regular/concurrent-chunks
                        time:   [842.04 ms 846.33 ms 850.56 ms]
                        change: [+127.40% +131.03% +134.00%] (p = 0.00 < 0.05)
                        Performance has regressed.
Benchmarking prove/plot/rayon/regular/whole-sector
Benchmarking prove/plot/rayon/regular/whole-sector: Warming up for 3.0000 s

Warning: Unable to complete 10 samples in 5.0s. You may wish to increase target time to 28.8s.
Benchmarking prove/plot/rayon/regular/whole-sector: Collecting 10 samples in estimated 28.799 s (10 iterations)
Benchmarking prove/plot/rayon/regular/whole-sector: Analyzing
prove/plot/rayon/regular/whole-sector
                        time:   [2.8695 s 2.8855 s 2.8994 s]
                        change: [+223.70% +226.41% +228.92%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 1 outliers among 10 measurements (10.00%)
  1 (10.00%) low mild

SATA (via USB gen 1 port) benchmark result

PS C:\Subspace Farmer> .\subspace-farmer3h_11Mar benchmark prove Q:\1
Benchmarking prove/plot/rayon/unbuffered/concurrent-chunks
Benchmarking prove/plot/rayon/unbuffered/concurrent-chunks: Warming up for 3.0000 s

Warning: Unable to complete 10 samples in 5.0s. You may wish to increase target time to 11.3s.
Benchmarking prove/plot/rayon/unbuffered/concurrent-chunks: Collecting 10 samples in estimated 11.285 s (10 iterations)
Benchmarking prove/plot/rayon/unbuffered/concurrent-chunks: Analyzing
prove/plot/rayon/unbuffered/concurrent-chunks
                        time:   [1.1201 s 1.1240 s 1.1278 s]
                        change: [+12.198% +14.445% +16.675%] (p = 0.00 < 0.05)
                        Performance has regressed.
Benchmarking prove/plot/rayon/unbuffered/whole-sector
Benchmarking prove/plot/rayon/unbuffered/whole-sector: Warming up for 3.0000 s

Warning: Unable to complete 10 samples in 5.0s. You may wish to increase target time to 52.4s.
Benchmarking prove/plot/rayon/unbuffered/whole-sector: Collecting 10 samples in estimated 52.380 s (10 iterations)
Benchmarking prove/plot/rayon/unbuffered/whole-sector: Analyzing
prove/plot/rayon/unbuffered/whole-sector
                        time:   [4.9003 s 5.0556 s 5.1711 s]
                        change: [+40.661% +46.057% +51.058%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 2 outliers among 10 measurements (20.00%)
  2 (20.00%) low severe
Benchmarking prove/plot/rayon/regular/concurrent-chunks
Benchmarking prove/plot/rayon/regular/concurrent-chunks: Warming up for 3.0000 s

Warning: Unable to complete 10 samples in 5.0s. You may wish to increase target time to 13.0s.
Benchmarking prove/plot/rayon/regular/concurrent-chunks: Collecting 10 samples in estimated 13.017 s (10 iterations)
Benchmarking prove/plot/rayon/regular/concurrent-chunks: Analyzing
prove/plot/rayon/regular/concurrent-chunks
                        time:   [1.2585 s 1.2625 s 1.2664 s]
                        change: [+48.313% +49.174% +50.075%] (p = 0.00 < 0.05)
                        Performance has regressed.
Benchmarking prove/plot/rayon/regular/whole-sector
Benchmarking prove/plot/rayon/regular/whole-sector: Warming up for 3.0000 s

Warning: Unable to complete 10 samples in 5.0s. You may wish to increase target time to 45.0s.
Benchmarking prove/plot/rayon/regular/whole-sector: Collecting 10 samples in estimated 44.965 s (10 iterations)
Benchmarking prove/plot/rayon/regular/whole-sector: Analyzing
prove/plot/rayon/regular/whole-sector
                        time:   [4.4409 s 4.4571 s 4.4765 s]
                        change: [+53.501% +54.469% +55.540%] (p = 0.00 < 0.05)
                        Performance has regressed.
Found 1 outliers among 10 measurements (10.00%)
  1 (10.00%) high mild

nazar-pc · March 12, 2024, 6:02am

This test build will work exactly the same as mar-11 release, but will print some extra logs: Snapshot build · subspace/subspace@0b7738a · GitHub

Please attach logs from the start of the farmer until it finishes piece cache sync. And use spoilers like in edited first message, that way it is easier to scroll long threads.

dragonP · March 12, 2024, 7:37am

CPU 14900K, 64GB RAM, Win 11. At my first run, I ran with all my disks, I got hung: it couldnt finish its audit for all disks.

I ran 2nd time with the following disks, and result is below. I’ve listed down the disk type and how it’s connected. Please also note that I couldn’t close the farmer instance in the second time and had to hard reboot my PC. I understand this can be ‘as expected’ since I did not let it finish the piece cache sync. However, I can’t leave it to finish piece cache sync, knowing that I’ve taken out 70% of all my disks in the second run. Just to let you know what I experienced.

path=C:\1,size=1600G Disk 0 NVMe Gen 4, M2 port
path=D:\1,size=1900G Disk 1 NVMe Gen 4, via PCIe x16 converter
path=E:\1,size=3900G Disk 2 SSD SATA, directly to mainboard
path=H:\1,size=3900G Disk 3 SSD SATA, directly to mainboard
path=S:\1,size=1900G Disk 4 SSD SATA, via USB 3.2 gen 1

Summary

C:\Subspace Farmer>subspace-farmer-windows-x86_64-skylake-debug-windows.exe farm path=C:\1,size=1600G path=D:\1,size=1900G path=E:\1,size=3900G path=H:\1,size=3900G path=S:\1,size=1900G --farm-during-initial-plotting true --node-rpc-url ws://192.168.2.215:9945 --metrics-endpoints 192.168.2.215:2222 --reward-address stxxxxx
2024-03-12T07:25:01.184377Z  INFO subspace_farmer::commands::farm: Connecting to node RPC url=ws://192.168.2.215:9945
2024-03-12T07:25:01.187103Z  INFO subspace_networking::constructor: DSN instance configured. allow_non_global_addresses_in_dht=false peer_id=12D3KooWR5pmY81H6a7dyWw3ppz1SawSF3PPRhiofrvXWG9Y8JmK protocol_version=/subspace/2/0c121c75f4ef450f40619e1fca9d1e8e7fbabc42c895bc4790801e85d5a91c34
2024-03-12T07:25:01.192419Z  INFO libp2p_swarm: local_peer_id=12D3KooWR5pmY81H6a7dyWw3ppz1SawSF3PPRhiofrvXWG9Y8JmK
2024-03-12T07:25:01.194535Z  INFO subspace_metrics: Metrics server started. endpoints=[192.168.2.215:2222]
2024-03-12T07:25:01.194595Z  INFO actix_server::builder: starting 2 workers
2024-03-12T07:25:01.194874Z  INFO actix_server::server: Tokio runtime found; starting in existing Tokio runtime
2024-03-12T07:25:01.715072Z  INFO {disk_farm_index=0}: subspace_farmer::single_disk_farm::plot_cache: Checking plot cache contents
Testing unbuffered
2024-03-12T07:25:01.732645Z  INFO {disk_farm_index=1}: subspace_farmer::single_disk_farm::plot_cache: Checking plot cache contents
Testing unbuffered
2024-03-12T07:25:01.878556Z  INFO {disk_farm_index=4}: subspace_farmer::single_disk_farm::plot_cache: Checking plot cache contents
Testing unbuffered
2024-03-12T07:25:02.683633Z  INFO {disk_farm_index=3}: subspace_farmer::single_disk_farm::plot_cache: Checking plot cache contents
Testing unbuffered
2024-03-12T07:25:02.842639Z  INFO {disk_farm_index=2}: subspace_farmer::single_disk_farm::plot_cache: Checking plot cache contents
Testing unbuffered
2024-03-12T07:25:02.846891Z  INFO subspace_farmer::single_disk_farm: Benchmarking faster proving method
Chunks in 6.0918645s
Whole sector in 442.3827ms
Chunks in 6.0617926s
Whole sector in 437.2093ms
Chunks in 6.0669184s
Whole sector in 437.3577ms
2024-03-12T07:25:23.091937Z  INFO subspace_farmer::single_disk_farm: Faster proving method found fastest_mode=WholeSector
2024-03-12T07:25:23.092323Z  INFO subspace_farmer::single_disk_farm: Benchmarking faster proving method
Proved in 1.0038553s
Testing regular
Chunks in 14.6054536s
Whole sector in 4.1602457s
Chunks in 14.6727134s
Whole sector in 4.0602971s
Chunks in 14.3128424s
Whole sector in 4.0680942s
2024-03-12T07:26:23.020383Z  INFO subspace_farmer::single_disk_farm: Faster proving method found fastest_mode=WholeSector
2024-03-12T07:26:23.090382Z  INFO subspace_farmer::single_disk_farm: Benchmarking faster proving method
Chunks in 5.0630433s
Whole sector in 272.9312ms
Chunks in 5.0838354s
Whole sector in 274.4275ms
Chunks in 5.1300496s
Whole sector in 275.0331ms
2024-03-12T07:26:39.736962Z  INFO subspace_farmer::single_disk_farm: Faster proving method found fastest_mode=WholeSector
2024-03-12T07:26:39.737302Z  INFO subspace_farmer::single_disk_farm: Benchmarking faster proving method
Proved in 5.3991921s
Testing regular
Chunks in 453.3106ms
Whole sector in 2.0195682s
Chunks in 453.2532ms
Whole sector in 2.0175081s
Chunks in 453.3358ms
Whole sector in 2.0370477s
2024-03-12T07:26:49.255457Z  INFO subspace_farmer::single_disk_farm: Faster proving method found fastest_mode=ConcurrentChunks
2024-03-12T07:26:49.297620Z  INFO subspace_farmer::single_disk_farm: Benchmarking faster proving method
Proved in 915.7803ms
Testing regular
Chunks in 22.0221537s
Whole sector in 3.5396605s
Chunks in 23.0449313s
Proved in 697.0558ms
Testing regular
Whole sector in 19.8215842s
Chunks in 25.388723s
Whole sector in 3.6828729s
2024-03-12T07:28:30.623815Z  INFO subspace_farmer::single_disk_farm: Faster proving method found fastest_mode=WholeSector
2024-03-12T07:28:30.693484Z  INFO subspace_farmer::single_disk_farm: Benchmarking faster proving method
Chunks in 240.818ms
Whole sector in 500.1719ms
Chunks in 178.0603ms
Whole sector in 215.3955ms
Chunks in 176.274ms
Whole sector in 213.1608ms
2024-03-12T07:28:32.948718Z  INFO subspace_farmer::single_disk_farm: Faster proving method found fastest_mode=ConcurrentChunks
2024-03-12T07:28:33.007683Z  INFO subspace_farmer::single_disk_farm: Benchmarking faster proving method
Chunks in 1.7512251s
Whole sector in 4.0470308s
Chunks in 199.6899ms
Whole sector in 308.8812ms
Proved in 948.463ms
Single disk farm 1:
  ID: 01HQ2EPPGT1Y9RC3YA8V3XE8Q8
  Genesis hash: 0x0c121c75f4ef450f40619e1fca9d1e8e7fbabc42c895bc4790801e85d5a91c34
  Public key: 0xd61940eb2af8517c21fddc9632627b2237c4b5a31d8005d1b89dcf25568cc64c
  Allocated space: 1.7 TiB (1.9 TB)
  Directory: D:\1
Chunks in 188.7211ms
Whole sector in 180.9977ms
2024-03-12T07:28:44.000261Z  INFO subspace_farmer::single_disk_farm: Faster proving method found fastest_mode=WholeSector
2024-03-12T07:28:44.008387Z  INFO subspace_farmer::single_disk_farm: Benchmarking faster proving method
Chunks in 251.9533ms
Whole sector in 346.7967ms
Chunks in 181.8841ms
Whole sector in 182.4456ms
Chunks in 182.5721ms
Whole sector in 252.0622ms
2024-03-12T07:28:45.809430Z  INFO subspace_farmer::single_disk_farm: Faster proving method found fastest_mode=ConcurrentChunks
2024-03-12T07:28:45.883282Z  INFO subspace_farmer::single_disk_farm: Benchmarking faster proving method
Chunks in 619.3361ms
Whole sector in 2.0784616s
Chunks in 175.351ms
Whole sector in 173.1971ms
Chunks in 175.5719ms
Whole sector in 229.7904ms
2024-03-12T07:28:51.750072Z  INFO subspace_farmer::single_disk_farm: Faster proving method found fastest_mode=WholeSector
Proved in 486.7208ms
Single disk farm 0:
  ID: 01HPY2B2TCAVA0VHSXH3QK25QK
  Genesis hash: 0x0c121c75f4ef450f40619e1fca9d1e8e7fbabc42c895bc4790801e85d5a91c34
  Public key: 0x90cd2ea172518e6fbbcca49ac5d2be9f011d953d0b68aa5b4bc6b8a64d7f6b7e
  Allocated space: 1.5 TiB (1.6 TB)
  Directory: C:\1
Proved in 1.0777703s
Single disk farm 4:
  ID: 01HQZS918SM73KFEEB68P44DMM
  Genesis hash: 0x0c121c75f4ef450f40619e1fca9d1e8e7fbabc42c895bc4790801e85d5a91c34
  Public key: 0xb25384407db986593c5f5730e7eb5100a7d227a56eb1ea22109e7935c5a72f58
  Allocated space: 1.7 TiB (1.9 TB)
  Directory: S:\1
Proved in 3.6669208s
Testing regular
2024-03-12T07:29:11.565519Z  INFO subspace_farmer::single_disk_farm: Benchmarking faster proving method
Chunks in 2.3103347s
Whole sector in 3.1839691s
Chunks in 170.6605ms
Whole sector in 191.1121ms
Chunks in 172.6613ms
Whole sector in 197.605ms
2024-03-12T07:29:21.658566Z  INFO subspace_farmer::single_disk_farm: Faster proving method found fastest_mode=ConcurrentChunks
Proved in 3.0812864s
Single disk farm 2:
  ID: 01HNKYZ4P7FAQ11R5JF82EMVXE
  Genesis hash: 0x0c121c75f4ef450f40619e1fca9d1e8e7fbabc42c895bc4790801e85d5a91c34
  Public key: 0x6e76c28edbbe8fcf576dac38a435dcdc4f175548abbec55165b1446909740063
  Allocated space: 3.5 TiB (3.9 TB)
  Directory: E:\1
Proved in 1.289327s
Single disk farm 3:
  ID: 01HQ5RJ8DZJS93K0GTC61NBKMM
  Genesis hash: 0x0c121c75f4ef450f40619e1fca9d1e8e7fbabc42c895bc4790801e85d5a91c34
  Public key: 0x90d3aec152cb7886f28a215dac41f22c43a82742e8d0ce453407ecdc9a7ebe05
  Allocated space: 3.5 TiB (3.9 TB)
  Directory: H:\1
2024-03-12T07:29:54.431197Z  INFO subspace_farmer::commands::farm: Collecting already plotted pieces (this will take some time)...
2024-03-12T07:29:54.431296Z  INFO subspace_farmer::farmer_cache: Initializing piece cache
2024-03-12T07:29:55.391713Z  INFO subspace_farmer::commands::farm: Finished collecting already plotted pieces successfully
2024-03-12T07:29:55.397143Z  INFO {disk_farm_index=0}: subspace_farmer::single_disk_farm::farming: Subscribing to slot info notifications
2024-03-12T07:29:55.397426Z  INFO {disk_farm_index=0}: subspace_farmer::reward_signing: Subscribing to reward signing notifications
2024-03-12T07:29:55.397613Z  INFO {disk_farm_index=1}: subspace_farmer::single_disk_farm::farming: Subscribing to slot info notifications
2024-03-12T07:29:55.398512Z  INFO {disk_farm_index=1}: subspace_farmer::reward_signing: Subscribing to reward signing notifications
2024-03-12T07:29:55.398901Z  INFO {disk_farm_index=2}: subspace_farmer::single_disk_farm::farming: Subscribing to slot info notifications
2024-03-12T07:29:55.399726Z  INFO {disk_farm_index=2}: subspace_farmer::reward_signing: Subscribing to reward signing notifications
2024-03-12T07:29:55.400573Z  INFO {disk_farm_index=3}: subspace_farmer::single_disk_farm::farming: Subscribing to slot info notifications
2024-03-12T07:29:55.400696Z  INFO {disk_farm_index=3}: subspace_farmer::reward_signing: Subscribing to reward signing notifications
2024-03-12T07:29:55.401178Z  INFO {disk_farm_index=4}: subspace_farmer::single_disk_farm::farming: Subscribing to slot info notifications
2024-03-12T07:29:55.401402Z  INFO {disk_farm_index=4}: subspace_farmer::reward_signing: Subscribing to reward signing notifications
2024-03-12T07:29:55.403038Z  INFO {disk_farm_index=0}: subspace_farmer::single_disk_farm::plotting: Subscribing to archived segments
2024-03-12T07:29:55.403369Z  INFO {disk_farm_index=1}: subspace_farmer::single_disk_farm::plotting: Subscribing to archived segments
2024-03-12T07:29:55.403497Z  INFO {disk_farm_index=2}: subspace_farmer::single_disk_farm::plotting: Subscribing to archived segments
2024-03-12T07:29:55.404452Z  INFO {disk_farm_index=3}: subspace_farmer::single_disk_farm::plotting: Subscribing to archived segments
2024-03-12T07:29:55.404666Z  INFO {disk_farm_index=4}: subspace_farmer::single_disk_farm::plotting: Subscribing to archived segments
2024-03-12T07:29:55.405253Z  INFO subspace_farmer::commands::farm::dsn: DSN listening on /ip4/192.168.2.215/udp/30533/quic-v1/p2p/12D3KooWR5pmY81H6a7dyWw3ppz1SawSF3PPRhiofrvXWG9Y8JmK
2024-03-12T07:29:55.416106Z  INFO subspace_farmer::commands::farm::dsn: DSN listening on /ip6/::1/udp/30533/quic-v1/p2p/12D3KooWR5pmY81H6a7dyWw3ppz1SawSF3PPRhiofrvXWG9Y8JmK
2024-03-12T07:29:55.416176Z  INFO subspace_farmer::commands::farm::dsn: DSN listening on /ip4/127.0.0.1/udp/30533/quic-v1/p2p/12D3KooWR5pmY81H6a7dyWw3ppz1SawSF3PPRhiofrvXWG9Y8JmK
2024-03-12T07:29:55.443864Z  INFO subspace_farmer::commands::farm::dsn: DSN listening on /ip4/192.168.2.215/tcp/30533/p2p/12D3KooWR5pmY81H6a7dyWw3ppz1SawSF3PPRhiofrvXWG9Y8JmK
2024-03-12T07:29:55.444596Z  INFO subspace_farmer::commands::farm::dsn: DSN listening on /ip6/::1/tcp/30533/p2p/12D3KooWR5pmY81H6a7dyWw3ppz1SawSF3PPRhiofrvXWG9Y8JmK
2024-03-12T07:29:55.444771Z  INFO subspace_farmer::commands::farm::dsn: DSN listening on /ip4/127.0.0.1/tcp/30533/p2p/12D3KooWR5pmY81H6a7dyWw3ppz1SawSF3PPRhiofrvXWG9Y8JmK
2024-03-12T07:30:18.751129Z  INFO subspace_farmer::farmer_cache: Synchronizing piece cache

nazar-pc · March 12, 2024, 8:44am

Wow, those are spectacularly horrible! There is no doubt it will fail proving.

Looks like logs do not include farm ID to show which logs correspond to which SSD, but either way none of the disks should be that bad.

According to your initial proving benches good performance should be achievable, but we may hit something stupid like GetFileInformationByHandle followed by read with FILE_FLAG_NO_BUFFERING is extremely slow - Microsoft Q&A (that is one case I personally discovered, but maybe there are others too that I have not experienced myself).

I’ll prepare another test build to figure out what and where is happening.

P.S. I hate Windows a bit more every day. What a broken OS

nazar-pc · March 12, 2024, 8:59am

Another test build, will print even more information: Snapshot build · subspace/subspace@a270905 · GitHub

NOTE: Files will have the same name.

dragonP · March 12, 2024, 10:31am

This time I run on other PC with same build (last time, that PC also run a node connecting many farmers so hard reboot was a pain to me)

CPU 14900K, 64GB RAM, Win 11

1. path=C:\1,size=1600G Disk 0: NVMe Gen 4 (directly to mainboard)
1. path=D:\1,size=3900G Disk 1: SATA SSD (Samsung QVO, directly to mainboard)
1. path=E:\1,size=3900G Disk 2: SATA SSD (Samsung EVO, directly to mainboard)
1. path=O:\1,size=1900G Disk 4: SATA SSD (X600, via USB 3.2 gen 1)

Summary

C:\Subspace Farmer>subspace-farmer-windows-x86_64-skylake-debug-windows.exe farm path=C:\1,size=1600G path=D:\1,size=3900G path=E:\1,size=3900G path=O:\1,size=1900G --node-rpc-url ws://192.168.2.215:9945 --metrics-endpoints 192.168.2.213:2222 --reward-address st6bf1gCkUszz48jBMcdr6ur1Xs4R2hghiTGNhT7UZoSS5Ue2
2024-03-12T10:18:39.382324Z  INFO subspace_farmer::commands::farm: Connecting to node RPC url=ws://192.168.2.215:9945
2024-03-12T10:18:39.385953Z  INFO subspace_networking::constructor: DSN instance configured. allow_non_global_addresses_in_dht=false peer_id=12D3KooWAmJ7wTwABg55gwFfwzuMyfCFMzuKfc76cxBgPDhGZPMZ protocol_version=/subspace/2/0c121c75f4ef450f40619e1fca9d1e8e7fbabc42c895bc4790801e85d5a91c34
2024-03-12T10:18:39.388298Z  INFO libp2p_swarm: local_peer_id=12D3KooWAmJ7wTwABg55gwFfwzuMyfCFMzuKfc76cxBgPDhGZPMZ
2024-03-12T10:18:39.389789Z  INFO subspace_metrics: Metrics server started. endpoints=[192.168.2.213:2222]
2024-03-12T10:18:39.389839Z  INFO actix_server::builder: starting 2 workers
2024-03-12T10:18:39.389900Z  INFO actix_server::server: Tokio runtime found; starting in existing Tokio runtime
2024-03-12T10:18:41.407475Z  INFO {disk_farm_index=2}: subspace_farmer::single_disk_farm: Bench a
2024-03-12T10:18:41.407559Z  INFO {disk_farm_index=2}: subspace_farmer::single_disk_farm: Benchmarking faster proving method
2024-03-12T10:18:53.048611Z  INFO {disk_farm_index=2}: subspace_farmer::single_disk_farm: Chunks in 9.3139347s
2024-03-12T10:18:55.140929Z  INFO {disk_farm_index=2}: subspace_farmer::single_disk_farm: Whole sector in 2.0922386s
2024-03-12T10:19:04.237780Z  INFO {disk_farm_index=2}: subspace_farmer::single_disk_farm: Chunks in 9.0966724s
2024-03-12T10:19:06.413002Z  INFO {disk_farm_index=2}: subspace_farmer::single_disk_farm: Whole sector in 2.175144s
2024-03-12T10:19:15.407090Z  INFO {disk_farm_index=2}: subspace_farmer::single_disk_farm: Chunks in 8.9940132s
2024-03-12T10:19:17.598760Z  INFO {disk_farm_index=2}: subspace_farmer::single_disk_farm: Whole sector in 2.1915806s
2024-03-12T10:19:17.598927Z  INFO {disk_farm_index=2}: subspace_farmer::single_disk_farm: Faster proving method found fastest_mode=WholeSector
2024-03-12T10:19:17.599172Z  INFO {disk_farm_index=1}: subspace_farmer::single_disk_farm: Bench a
2024-03-12T10:19:17.599376Z  INFO {disk_farm_index=1}: subspace_farmer::single_disk_farm: Benchmarking faster proving method
2024-03-12T10:19:36.271619Z  INFO {disk_farm_index=1}: subspace_farmer::single_disk_farm: Chunks in 16.3888794s
2024-03-12T10:19:38.548724Z  INFO {disk_farm_index=1}: subspace_farmer::single_disk_farm: Whole sector in 2.276965s
2024-03-12T10:19:54.971034Z  INFO {disk_farm_index=1}: subspace_farmer::single_disk_farm: Chunks in 16.4221362s
2024-03-12T10:19:57.214687Z  INFO {disk_farm_index=1}: subspace_farmer::single_disk_farm: Whole sector in 2.2435312s
2024-03-12T10:20:14.077631Z  INFO {disk_farm_index=1}: subspace_farmer::single_disk_farm: Chunks in 16.862873s
2024-03-12T10:20:16.323123Z  INFO {disk_farm_index=1}: subspace_farmer::single_disk_farm: Whole sector in 2.2453997s
2024-03-12T10:20:16.323237Z  INFO {disk_farm_index=1}: subspace_farmer::single_disk_farm: Faster proving method found fastest_mode=WholeSector
2024-03-12T10:20:16.361396Z  INFO {disk_farm_index=0}: subspace_farmer::single_disk_farm: Bench a
2024-03-12T10:20:16.361760Z  INFO {disk_farm_index=0}: subspace_farmer::single_disk_farm: Benchmarking faster proving method
2024-03-12T10:20:21.874687Z  INFO {disk_farm_index=0}: subspace_farmer::single_disk_farm: Chunks in 4.9879853s
2024-03-12T10:20:22.154286Z  INFO {disk_farm_index=0}: subspace_farmer::single_disk_farm: Whole sector in 279.5196ms
2024-03-12T10:20:27.158986Z  INFO {disk_farm_index=0}: subspace_farmer::single_disk_farm: Chunks in 5.0046218s
2024-03-12T10:20:27.436830Z  INFO {disk_farm_index=0}: subspace_farmer::single_disk_farm: Whole sector in 277.7422ms
2024-03-12T10:20:32.477677Z  INFO {disk_farm_index=0}: subspace_farmer::single_disk_farm: Chunks in 5.0407753s
2024-03-12T10:20:32.759021Z  INFO {disk_farm_index=0}: subspace_farmer::single_disk_farm: Whole sector in 281.2501ms
2024-03-12T10:20:32.759136Z  INFO {disk_farm_index=0}: subspace_farmer::single_disk_farm: Faster proving method found fastest_mode=WholeSector
2024-03-12T10:20:32.759275Z  INFO {disk_farm_index=3}: subspace_farmer::single_disk_farm: Bench a
2024-03-12T10:20:32.759354Z  INFO {disk_farm_index=3}: subspace_farmer::single_disk_farm: Benchmarking faster proving method
2024-03-12T10:20:51.681891Z  INFO {disk_farm_index=3}: subspace_farmer::single_disk_farm: Chunks in 14.652019s
2024-03-12T10:20:55.911857Z  INFO {disk_farm_index=3}: subspace_farmer::single_disk_farm: Whole sector in 4.2298674s
2024-03-12T10:21:10.910797Z  INFO {disk_farm_index=3}: subspace_farmer::single_disk_farm: Chunks in 14.9985388s
2024-03-12T10:21:15.167893Z  INFO {disk_farm_index=3}: subspace_farmer::single_disk_farm: Whole sector in 4.2569707s
2024-03-12T10:21:30.364921Z  INFO {disk_farm_index=3}: subspace_farmer::single_disk_farm: Chunks in 15.1967213s
2024-03-12T10:21:34.585884Z  INFO {disk_farm_index=3}: subspace_farmer::single_disk_farm: Whole sector in 4.2208642s
2024-03-12T10:21:34.586101Z  INFO {disk_farm_index=3}: subspace_farmer::single_disk_farm: Faster proving method found fastest_mode=WholeSector
2024-03-12T10:21:34.635664Z  INFO {disk_farm_index=0}: subspace_farmer::single_disk_farm: Bench b
2024-03-12T10:21:34.635742Z  INFO {disk_farm_index=0}: subspace_farmer::single_disk_farm: Benchmarking faster proving method
2024-03-12T10:21:40.157441Z  INFO {disk_farm_index=0}: subspace_farmer::single_disk_farm: Chunks in 5.0035778s
2024-03-12T10:21:40.427902Z  INFO {disk_farm_index=0}: subspace_farmer::single_disk_farm: Whole sector in 270.3855ms
2024-03-12T10:21:45.443956Z  INFO {disk_farm_index=0}: subspace_farmer::single_disk_farm: Chunks in 5.0159768s
2024-03-12T10:21:45.720419Z  INFO {disk_farm_index=0}: subspace_farmer::single_disk_farm: Whole sector in 276.3541ms
2024-03-12T10:21:50.717364Z  INFO {disk_farm_index=0}: subspace_farmer::single_disk_farm: Chunks in 4.9968159s
2024-03-12T10:21:50.988195Z  INFO {disk_farm_index=0}: subspace_farmer::single_disk_farm: Whole sector in 270.7662ms
2024-03-12T10:21:50.988264Z  INFO {disk_farm_index=0}: subspace_farmer::single_disk_farm: Faster proving method found fastest_mode=WholeSector
2024-03-12T10:21:50.988322Z  INFO {disk_farm_index=3}: subspace_farmer::single_disk_farm: Bench b
2024-03-12T10:21:50.988373Z  INFO {disk_farm_index=3}: subspace_farmer::single_disk_farm: Benchmarking faster proving method
2024-03-12T10:22:10.097508Z  INFO {disk_farm_index=3}: subspace_farmer::single_disk_farm: Chunks in 14.8099685s
2024-03-12T10:22:14.268832Z  INFO {disk_farm_index=3}: subspace_farmer::single_disk_farm: Whole sector in 4.1712313s
2024-03-12T10:22:28.997101Z  INFO {disk_farm_index=3}: subspace_farmer::single_disk_farm: Chunks in 14.7281372s
2024-03-12T10:22:33.185560Z  INFO {disk_farm_index=3}: subspace_farmer::single_disk_farm: Whole sector in 4.1883566s
2024-03-12T10:22:47.783528Z  INFO {disk_farm_index=3}: subspace_farmer::single_disk_farm: Chunks in 14.5977317s
2024-03-12T10:22:52.002749Z  INFO {disk_farm_index=3}: subspace_farmer::single_disk_farm: Whole sector in 4.2191311s
2024-03-12T10:22:52.002856Z  INFO {disk_farm_index=3}: subspace_farmer::single_disk_farm: Faster proving method found fastest_mode=WholeSector
2024-03-12T10:22:52.042186Z  INFO {disk_farm_index=2}: subspace_farmer::single_disk_farm: Bench b
2024-03-12T10:22:52.042283Z  INFO {disk_farm_index=2}: subspace_farmer::single_disk_farm: Benchmarking faster proving method
2024-03-12T10:23:03.693592Z  INFO {disk_farm_index=2}: subspace_farmer::single_disk_farm: Chunks in 9.2036908s
2024-03-12T10:23:05.796245Z  INFO {disk_farm_index=2}: subspace_farmer::single_disk_farm: Whole sector in 2.1025519s
2024-03-12T10:23:14.719083Z  INFO {disk_farm_index=2}: subspace_farmer::single_disk_farm: Chunks in 8.9227381s
2024-03-12T10:23:16.821743Z  INFO {disk_farm_index=2}: subspace_farmer::single_disk_farm: Whole sector in 2.1025653s
2024-03-12T10:23:25.739035Z  INFO {disk_farm_index=2}: subspace_farmer::single_disk_farm: Chunks in 8.917215s
2024-03-12T10:23:27.845619Z  INFO {disk_farm_index=2}: subspace_farmer::single_disk_farm: Whole sector in 2.1065023s
2024-03-12T10:23:27.845737Z  INFO {disk_farm_index=2}: subspace_farmer::single_disk_farm: Faster proving method found fastest_mode=WholeSector
2024-03-12T10:23:27.845869Z  INFO {disk_farm_index=1}: subspace_farmer::single_disk_farm: Bench b
2024-03-12T10:23:27.845956Z  INFO {disk_farm_index=1}: subspace_farmer::single_disk_farm: Benchmarking faster proving method
2024-03-12T10:23:45.592868Z  INFO {disk_farm_index=1}: subspace_farmer::single_disk_farm: Chunks in 15.5234734s
2024-03-12T10:23:47.744228Z  INFO {disk_farm_index=1}: subspace_farmer::single_disk_farm: Whole sector in 2.1512368s
2024-03-12T10:24:03.266115Z  INFO {disk_farm_index=1}: subspace_farmer::single_disk_farm: Chunks in 15.5218091s
2024-03-12T10:24:05.454088Z  INFO {disk_farm_index=1}: subspace_farmer::single_disk_farm: Whole sector in 2.1878729s
2024-03-12T10:24:21.310545Z  INFO {disk_farm_index=1}: subspace_farmer::single_disk_farm: Chunks in 15.8563771s
2024-03-12T10:24:23.480007Z  INFO {disk_farm_index=1}: subspace_farmer::single_disk_farm: Whole sector in 2.1693538s
2024-03-12T10:24:23.480091Z  INFO {disk_farm_index=1}: subspace_farmer::single_disk_farm: Faster proving method found fastest_mode=WholeSector
2024-03-12T10:24:23.525398Z  INFO {disk_farm_index=1}: subspace_farmer::single_disk_farm::plot_cache: Checking plot cache contents
2024-03-12T10:24:23.529044Z  INFO {disk_farm_index=2}: subspace_farmer::single_disk_farm::plot_cache: Checking plot cache contents
2024-03-12T10:24:23.529658Z  INFO {disk_farm_index=3}: subspace_farmer::single_disk_farm::plot_cache: Checking plot cache contents
2024-03-12T10:24:23.533350Z  INFO {disk_farm_index=0}: subspace_farmer::single_disk_farm::plot_cache: Checking plot cache contents
2024-03-12T10:24:23.538936Z  INFO {disk_farm_index=3}: subspace_farmer::single_disk_farm: Testing unbuffered
2024-03-12T10:24:23.539056Z  INFO {disk_farm_index=3}: subspace_farmer::single_disk_farm: Benchmarking faster proving method
2024-03-12T10:24:42.261849Z  INFO {disk_farm_index=3}: subspace_farmer::single_disk_farm: Chunks in 14.4188617s
2024-03-12T10:24:46.326240Z  INFO {disk_farm_index=3}: subspace_farmer::single_disk_farm: Whole sector in 4.0643034s
2024-03-12T10:25:00.670124Z  INFO {disk_farm_index=3}: subspace_farmer::single_disk_farm: Chunks in 14.3436895s
2024-03-12T10:25:04.753877Z  INFO {disk_farm_index=3}: subspace_farmer::single_disk_farm: Whole sector in 4.083676s
2024-03-12T10:25:19.466739Z  INFO {disk_farm_index=3}: subspace_farmer::single_disk_farm: Chunks in 14.7127707s
2024-03-12T10:25:23.556626Z  INFO {disk_farm_index=3}: subspace_farmer::single_disk_farm: Whole sector in 4.0897989s
2024-03-12T10:25:23.556747Z  INFO {disk_farm_index=3}: subspace_farmer::single_disk_farm: Faster proving method found fastest_mode=WholeSector
2024-03-12T10:25:23.556884Z  INFO {disk_farm_index=1}: subspace_farmer::single_disk_farm: Testing unbuffered
2024-03-12T10:25:23.556988Z  INFO {disk_farm_index=1}: subspace_farmer::single_disk_farm: Benchmarking faster proving method
2024-03-12T10:25:33.279602Z  INFO subspace_farmer::single_disk_farm: Proved in 5.0207964s
2024-03-12T10:25:41.097619Z  INFO {disk_farm_index=1}: subspace_farmer::single_disk_farm: Chunks in 15.3467963s
2024-03-12T10:25:43.293159Z  INFO {disk_farm_index=1}: subspace_farmer::single_disk_farm: Whole sector in 2.1954471s
2024-03-12T10:25:58.932297Z  INFO {disk_farm_index=1}: subspace_farmer::single_disk_farm: Chunks in 15.6390034s
2024-03-12T10:26:01.100649Z  INFO {disk_farm_index=1}: subspace_farmer::single_disk_farm: Whole sector in 2.1682629s
2024-03-12T10:26:16.890376Z  INFO {disk_farm_index=1}: subspace_farmer::single_disk_farm: Chunks in 15.7896468s
2024-03-12T10:26:19.073413Z  INFO {disk_farm_index=1}: subspace_farmer::single_disk_farm: Whole sector in 2.1829333s
2024-03-12T10:26:19.073541Z  INFO {disk_farm_index=1}: subspace_farmer::single_disk_farm: Faster proving method found fastest_mode=WholeSector
2024-03-12T10:26:19.099260Z  INFO {disk_farm_index=0}: subspace_farmer::single_disk_farm: Testing unbuffered
2024-03-12T10:26:19.099343Z  INFO {disk_farm_index=0}: subspace_farmer::single_disk_farm: Benchmarking faster proving method
2024-03-12T10:26:24.501058Z  INFO {disk_farm_index=0}: subspace_farmer::single_disk_farm: Chunks in 5.0563273s
2024-03-12T10:26:24.778945Z  INFO {disk_farm_index=0}: subspace_farmer::single_disk_farm: Whole sector in 277.8147ms
2024-03-12T10:26:29.823393Z  INFO {disk_farm_index=0}: subspace_farmer::single_disk_farm: Chunks in 5.0443721s
2024-03-12T10:26:30.102284Z  INFO {disk_farm_index=0}: subspace_farmer::single_disk_farm: Whole sector in 278.7801ms
2024-03-12T10:26:35.234357Z  INFO {disk_farm_index=0}: subspace_farmer::single_disk_farm: Chunks in 5.1319968s
2024-03-12T10:26:35.516901Z  INFO {disk_farm_index=0}: subspace_farmer::single_disk_farm: Whole sector in 282.4438ms
2024-03-12T10:26:35.517086Z  INFO {disk_farm_index=0}: subspace_farmer::single_disk_farm: Faster proving method found fastest_mode=WholeSector
2024-03-12T10:26:35.556344Z  INFO {disk_farm_index=2}: subspace_farmer::single_disk_farm: Testing unbuffered
2024-03-12T10:26:35.556560Z  INFO {disk_farm_index=2}: subspace_farmer::single_disk_farm: Benchmarking faster proving method
2024-03-12T10:26:47.148663Z  INFO {disk_farm_index=2}: subspace_farmer::single_disk_farm: Chunks in 9.2099847s
2024-03-12T10:26:47.238857Z  INFO subspace_farmer::single_disk_farm: Proved in 966.9831ms
2024-03-12T10:26:49.244422Z  INFO {disk_farm_index=2}: subspace_farmer::single_disk_farm: Whole sector in 2.0956509s
2024-03-12T10:26:58.009528Z  INFO {disk_farm_index=2}: subspace_farmer::single_disk_farm: Chunks in 8.7650057s
2024-03-12T10:27:00.107281Z  INFO {disk_farm_index=2}: subspace_farmer::single_disk_farm: Whole sector in 2.0976407s
2024-03-12T10:27:08.872629Z  INFO {disk_farm_index=2}: subspace_farmer::single_disk_farm: Chunks in 8.7652205s
2024-03-12T10:27:10.982147Z  INFO {disk_farm_index=2}: subspace_farmer::single_disk_farm: Whole sector in 2.1094283s
2024-03-12T10:27:10.982251Z  INFO {disk_farm_index=2}: subspace_farmer::single_disk_farm: Faster proving method found fastest_mode=WholeSector
2024-03-12T10:27:21.043247Z  INFO subspace_farmer::single_disk_farm: Proved in 2.8832663s
2024-03-12T10:28:06.936044Z  INFO subspace_farmer::single_disk_farm: Proved in 2.8178729s
2024-03-12T10:28:06.940319Z  INFO {disk_farm_index=2}: subspace_farmer::single_disk_farm: Testing regular
2024-03-12T10:28:06.940492Z  INFO {disk_farm_index=2}: subspace_farmer::single_disk_farm: Benchmarking faster proving method
2024-03-12T10:28:09.756080Z  INFO {disk_farm_index=2}: subspace_farmer::single_disk_farm: Chunks in 607.1364ms
2024-03-12T10:28:11.818045Z  INFO {disk_farm_index=2}: subspace_farmer::single_disk_farm: Whole sector in 2.0618939s
2024-03-12T10:28:11.988712Z  INFO {disk_farm_index=2}: subspace_farmer::single_disk_farm: Chunks in 170.4071ms
2024-03-12T10:28:12.131691Z  INFO {disk_farm_index=2}: subspace_farmer::single_disk_farm: Whole sector in 142.8824ms
2024-03-12T10:28:12.303258Z  INFO {disk_farm_index=2}: subspace_farmer::single_disk_farm: Chunks in 171.374ms
2024-03-12T10:28:12.445535Z  INFO {disk_farm_index=2}: subspace_farmer::single_disk_farm: Whole sector in 142.1842ms
2024-03-12T10:28:12.445663Z  INFO {disk_farm_index=2}: subspace_farmer::single_disk_farm: Faster proving method found fastest_mode=WholeSector
2024-03-12T10:28:12.473303Z  INFO {disk_farm_index=3}: subspace_farmer::single_disk_farm: Testing regular
2024-03-12T10:28:12.473436Z  INFO {disk_farm_index=3}: subspace_farmer::single_disk_farm: Benchmarking faster proving method
2024-03-12T10:28:18.272818Z  INFO {disk_farm_index=3}: subspace_farmer::single_disk_farm: Chunks in 1.7364754s
2024-03-12T10:28:22.186059Z  INFO {disk_farm_index=3}: subspace_farmer::single_disk_farm: Whole sector in 3.9131292s
2024-03-12T10:28:22.362852Z  INFO {disk_farm_index=3}: subspace_farmer::single_disk_farm: Chunks in 176.6665ms
2024-03-12T10:28:22.515269Z  INFO {disk_farm_index=3}: subspace_farmer::single_disk_farm: Whole sector in 152.3201ms
2024-03-12T10:28:22.689156Z  INFO {disk_farm_index=3}: subspace_farmer::single_disk_farm: Chunks in 173.7352ms
2024-03-12T10:28:22.839765Z  INFO {disk_farm_index=3}: subspace_farmer::single_disk_farm: Whole sector in 150.5215ms
2024-03-12T10:28:22.839866Z  INFO {disk_farm_index=3}: subspace_farmer::single_disk_farm: Faster proving method found fastest_mode=WholeSector
2024-03-12T10:28:22.857002Z  INFO {disk_farm_index=0}: subspace_farmer::single_disk_farm: Testing regular
2024-03-12T10:28:22.857203Z  INFO {disk_farm_index=0}: subspace_farmer::single_disk_farm: Benchmarking faster proving method
2024-03-12T10:28:23.441672Z  INFO {disk_farm_index=0}: subspace_farmer::single_disk_farm: Chunks in 229.6443ms
2024-03-12T10:28:23.768492Z  INFO {disk_farm_index=0}: subspace_farmer::single_disk_farm: Whole sector in 326.7112ms
2024-03-12T10:28:23.940563Z  INFO {disk_farm_index=0}: subspace_farmer::single_disk_farm: Chunks in 171.9861ms
2024-03-12T10:28:24.120965Z  INFO {disk_farm_index=0}: subspace_farmer::single_disk_farm: Whole sector in 180.3075ms
2024-03-12T10:28:24.297079Z  INFO {disk_farm_index=0}: subspace_farmer::single_disk_farm: Chunks in 175.9057ms
2024-03-12T10:28:24.458115Z  INFO {disk_farm_index=0}: subspace_farmer::single_disk_farm: Whole sector in 160.9356ms
2024-03-12T10:28:24.458237Z  INFO {disk_farm_index=0}: subspace_farmer::single_disk_farm: Faster proving method found fastest_mode=WholeSector
2024-03-12T10:28:24.498598Z  INFO {disk_farm_index=1}: subspace_farmer::single_disk_farm: Testing regular
2024-03-12T10:28:24.498817Z  INFO {disk_farm_index=1}: subspace_farmer::single_disk_farm: Benchmarking faster proving method
2024-03-12T10:28:27.794504Z  INFO {disk_farm_index=1}: subspace_farmer::single_disk_farm: Chunks in 808.0683ms
2024-03-12T10:28:29.907410Z  INFO {disk_farm_index=1}: subspace_farmer::single_disk_farm: Whole sector in 2.1128061s
2024-03-12T10:28:30.100991Z  INFO {disk_farm_index=1}: subspace_farmer::single_disk_farm: Chunks in 193.3263ms
2024-03-12T10:28:30.297003Z  INFO {disk_farm_index=1}: subspace_farmer::single_disk_farm: Whole sector in 195.8764ms
2024-03-12T10:28:30.484286Z  INFO {disk_farm_index=1}: subspace_farmer::single_disk_farm: Chunks in 187.0469ms
2024-03-12T10:28:30.652036Z  INFO {disk_farm_index=1}: subspace_farmer::single_disk_farm: Whole sector in 167.6358ms
2024-03-12T10:28:30.652134Z  INFO {disk_farm_index=1}: subspace_farmer::single_disk_farm: Faster proving method found fastest_mode=WholeSector
2024-03-12T10:28:32.216486Z  INFO subspace_farmer::single_disk_farm: Proved in 5.1719385s
2024-03-12T10:28:37.077220Z  INFO subspace_farmer::single_disk_farm: Proved in 1.1902253s
2024-03-12T10:29:00.262891Z  INFO subspace_farmer::single_disk_farm: Proved in 2.8341227s
2024-03-12T10:29:26.845899Z  INFO subspace_farmer::single_disk_farm: Proved in 3.2065851s
Single disk farm 1:
Single disk farm 2:
  ID: 01HNKYPP3CHGEQ7XFCSFSCZS0Z
  Genesis hash: 0x0c121c75f4ef450f40619e1fca9d1e8e7fbabc42c895bc4790801e85d5a91c34
  Public key: 0xfc87b6672781d3ddd52c3607b522e9e6f4f985bd7fc10d76ae8b9989bfe31232
  ID: 01HNKZ74W9P6Z7TH06HKP6P61H
  Genesis hash: 0x0c121c75f4ef450f40619e1fca9d1e8e7fbabc42c895bc4790801e85d5a91c34
  Public key: 0x082b5ac8684fb093006a7be16fd4ebae277e76df62a3263f2e06cc35aba30669
  Allocated space: 3.5 TiB (3.9 TB)
  Directory: D:\1
Single disk farm 0:
  ID: 01HPXZ5GCW0K2E89MC226KR8K1
  Genesis hash: 0x0c121c75f4ef450f40619e1fca9d1e8e7fbabc42c895bc4790801e85d5a91c34
  Public key: 0x0aabd0848b96a46da9381d65e9d031f630a4dcbaf6f8f3deaa6c6ced97a42219
  Allocated space: 1.5 TiB (1.6 TB)
  Directory: C:\1
  Allocated space: 3.5 TiB (3.9 TB)
  Directory: E:\1
Single disk farm 3:
  ID: 01HRKHY7N9RF2APEJYTNX1C599
  Genesis hash: 0x0c121c75f4ef450f40619e1fca9d1e8e7fbabc42c895bc4790801e85d5a91c34
  Public key: 0xbeb100a7ad39bd22523b700eacde24deae0ea7261c3fae392e25205ff4bd3c75
  Allocated space: 1.7 TiB (1.9 TB)
  Directory: O:\1
2024-03-12T10:29:26.864915Z  INFO subspace_farmer::farmer_cache: Initializing piece cache
2024-03-12T10:29:26.865009Z  INFO subspace_farmer::commands::farm: Collecting already plotted pieces (this will take some time)...
2024-03-12T10:29:27.823288Z  INFO subspace_farmer::commands::farm: Finished collecting already plotted pieces successfully
2024-03-12T10:29:27.825205Z  INFO {disk_farm_index=0}: subspace_farmer::single_disk_farm::farming: Subscribing to slot info notifications
2024-03-12T10:29:27.826119Z  INFO {disk_farm_index=0}: subspace_farmer::reward_signing: Subscribing to reward signing notifications
2024-03-12T10:29:27.826653Z  INFO {disk_farm_index=1}: subspace_farmer::single_disk_farm::farming: Subscribing to slot info notifications
2024-03-12T10:29:27.826850Z  INFO {disk_farm_index=1}: subspace_farmer::reward_signing: Subscribing to reward signing notifications
2024-03-12T10:29:27.826980Z  INFO {disk_farm_index=2}: subspace_farmer::single_disk_farm::farming: Subscribing to slot info notifications
2024-03-12T10:29:27.827090Z  INFO {disk_farm_index=2}: subspace_farmer::reward_signing: Subscribing to reward signing notifications
2024-03-12T10:29:27.827449Z  INFO {disk_farm_index=3}: subspace_farmer::single_disk_farm::farming: Subscribing to slot info notifications
2024-03-12T10:29:27.827515Z  INFO {disk_farm_index=3}: subspace_farmer::reward_signing: Subscribing to reward signing notifications
2024-03-12T10:29:27.828426Z  INFO {disk_farm_index=0}: subspace_farmer::single_disk_farm::plotting: Subscribing to archived segments
2024-03-12T10:29:27.828642Z  INFO {disk_farm_index=1}: subspace_farmer::single_disk_farm::plotting: Subscribing to archived segments
2024-03-12T10:29:27.828743Z  INFO {disk_farm_index=2}: subspace_farmer::single_disk_farm::plotting: Subscribing to archived segments
2024-03-12T10:29:27.828821Z  INFO {disk_farm_index=3}: subspace_farmer::single_disk_farm::plotting: Subscribing to archived segments
2024-03-12T10:29:27.833031Z  INFO subspace_farmer::commands::farm::dsn: DSN listening on /ip4/192.168.2.213/udp/30533/quic-v1/p2p/12D3KooWAmJ7wTwABg55gwFfwzuMyfCFMzuKfc76cxBgPDhGZPMZ
2024-03-12T10:29:27.837842Z  INFO subspace_farmer::commands::farm::dsn: DSN listening on /ip6/::1/udp/30533/quic-v1/p2p/12D3KooWAmJ7wTwABg55gwFfwzuMyfCFMzuKfc76cxBgPDhGZPMZ
2024-03-12T10:29:27.837908Z  INFO subspace_farmer::commands::farm::dsn: DSN listening on /ip4/127.0.0.1/udp/30533/quic-v1/p2p/12D3KooWAmJ7wTwABg55gwFfwzuMyfCFMzuKfc76cxBgPDhGZPMZ
2024-03-12T10:29:27.839624Z  INFO subspace_farmer::commands::farm::dsn: DSN listening on /ip4/192.168.2.213/tcp/30533/p2p/12D3KooWAmJ7wTwABg55gwFfwzuMyfCFMzuKfc76cxBgPDhGZPMZ
2024-03-12T10:29:27.844441Z  INFO subspace_farmer::commands::farm::dsn: DSN listening on /ip6/::1/tcp/30533/p2p/12D3KooWAmJ7wTwABg55gwFfwzuMyfCFMzuKfc76cxBgPDhGZPMZ
2024-03-12T10:29:27.844866Z  INFO subspace_farmer::commands::farm::dsn: DSN listening on /ip4/127.0.0.1/tcp/30533/p2p/12D3KooWAmJ7wTwABg55gwFfwzuMyfCFMzuKfc76cxBgPDhGZPMZ
2024-03-12T10:29:42.591966Z  INFO subspace_farmer::farmer_cache: Synchronizing piece cache
2024-03-12T10:29:42.672056Z  INFO {disk_farm_index=1}: subspace_farmer::single_disk_farm::plotting: Plotting sector (75.22% complete) sector_index=2747
2024-03-12T10:29:42.675863Z  INFO {disk_farm_index=3}: subspace_farmer::single_disk_farm::plotting: Plotting sector (7.36% complete) sector_index=131
2024-03-12T10:29:42.676259Z  INFO {disk_farm_index=2}: subspace_farmer::single_disk_farm::plotting: Plotting sector (75.16% complete) sector_index=2745

nazar-pc · March 12, 2024, 10:51am

So these issues seem to happen right after file opening, which probably means they should be reproducible with benchmarks.

Can you run proving benchmark on D:\1 again?

dragonP · March 12, 2024, 11:29am

It’s weird, it takes more RAM just to run this benchmark than the whole farming of many disks.

A later prove arpproach

And below is the benchmark result

Summary

PS C:\Subspace Farmer> .\subspace-farmer-windows-x86_64-skylake-debug-windows.exe benchmark prove D:\1
Benchmarking prove/plot/rayon/unbuffered/concurrent-chunks
Benchmarking prove/plot/rayon/unbuffered/concurrent-chunks: Warming up for 3.0000 s

Warning: Unable to complete 10 samples in 5.0s. You may wish to increase target time to 7.2s.
Benchmarking prove/plot/rayon/unbuffered/concurrent-chunks: Collecting 10 samples in estimated 7.1526 s (10 iterations)
Benchmarking prove/plot/rayon/unbuffered/concurrent-chunks: Analyzing
prove/plot/rayon/unbuffered/concurrent-chunks
                        time:   [715.68 ms 723.31 ms 730.84 ms]
Benchmarking prove/plot/rayon/unbuffered/whole-sector
Benchmarking prove/plot/rayon/unbuffered/whole-sector: Warming up for 3.0000 s

Warning: Unable to complete 10 samples in 5.0s. You may wish to increase target time to 27.9s.
Benchmarking prove/plot/rayon/unbuffered/whole-sector: Collecting 10 samples in estimated 27.919 s (10 iterations)
Benchmarking prove/plot/rayon/unbuffered/whole-sector: Analyzing
prove/plot/rayon/unbuffered/whole-sector
                        time:   [2.7211 s 2.7637 s 2.8058 s]
Benchmarking prove/plot/rayon/regular/concurrent-chunks
Benchmarking prove/plot/rayon/regular/concurrent-chunks: Warming up for 3.0000 s

Warning: Unable to complete 10 samples in 5.0s. You may wish to increase target time to 12.2s.
Benchmarking prove/plot/rayon/regular/concurrent-chunks: Collecting 10 samples in estimated 12.199 s (10 iterations)
Benchmarking prove/plot/rayon/regular/concurrent-chunks: Analyzing
prove/plot/rayon/regular/concurrent-chunks
                        time:   [925.65 ms 953.98 ms 984.19 ms]
Benchmarking prove/plot/rayon/regular/whole-sector
Benchmarking prove/plot/rayon/regular/whole-sector: Warming up for 3.0000 s

Warning: Unable to complete 10 samples in 5.0s. You may wish to increase target time to 32.4s.
Benchmarking prove/plot/rayon/regular/whole-sector: Collecting 10 samples in estimated 32.383 s (10 iterations)
Benchmarking prove/plot/rayon/regular/whole-sector: Analyzing
prove/plot/rayon/regular/whole-sector
                        time:   [3.0834 s 3.1117 s 3.1367 s]

nazar-pc · March 12, 2024, 11:45am

Okay, this is one of those cases that I have no explanation for. Can you try to start farmer with just one farm, namely this D:\1 and see if internal benches are similarly ridiculous? I’m wondering if there is something related to number of opened files or something.

dragonP · March 12, 2024, 11:57am

Here you are

Summary

C:\Subspace Farmer>subspace-farmer-windows-x86_64-skylake-debug-windows.exe farm path=D:\1,size=3900G --node-rpc-url ws://192.168.2.215:9945 --metrics-endpoints 192.168.2.213:2222 --reward-address st6xxxxxxxxxx
2024-03-12T11:53:42.645362Z  INFO subspace_farmer::commands::farm: Connecting to node RPC url=ws://192.168.2.215:9945
2024-03-12T11:53:42.667502Z  INFO subspace_networking::constructor: DSN instance configured. allow_non_global_addresses_in_dht=false peer_id=12D3KooWMYVm4znwQ2b3yEZNar49bDcdqfoqziqCAmjm96wnfPLu protocol_version=/subspace/2/0c121c75f4ef450f40619e1fca9d1e8e7fbabc42c895bc4790801e85d5a91c34
2024-03-12T11:53:42.675457Z  INFO libp2p_swarm: local_peer_id=12D3KooWMYVm4znwQ2b3yEZNar49bDcdqfoqziqCAmjm96wnfPLu
2024-03-12T11:53:42.679961Z  INFO subspace_metrics: Metrics server started. endpoints=[192.168.2.213:2222]
2024-03-12T11:53:42.680372Z  INFO actix_server::builder: starting 2 workers
2024-03-12T11:53:42.680875Z  INFO actix_server::server: Tokio runtime found; starting in existing Tokio runtime
2024-03-12T11:53:44.730599Z  INFO {disk_farm_index=0}: subspace_farmer::single_disk_farm: Bench a
2024-03-12T11:53:44.730785Z  INFO {disk_farm_index=0}: subspace_farmer::single_disk_farm: Benchmarking faster proving method
2024-03-12T11:53:47.995394Z  INFO {disk_farm_index=0}: subspace_farmer::single_disk_farm: Chunks in 881.1216ms
2024-03-12T11:53:50.076541Z  INFO {disk_farm_index=0}: subspace_farmer::single_disk_farm: Whole sector in 2.0810406s
2024-03-12T11:53:50.955333Z  INFO {disk_farm_index=0}: subspace_farmer::single_disk_farm: Chunks in 878.7267ms
2024-03-12T11:53:53.052347Z  INFO {disk_farm_index=0}: subspace_farmer::single_disk_farm: Whole sector in 2.0969421s
2024-03-12T11:53:53.927147Z  INFO {disk_farm_index=0}: subspace_farmer::single_disk_farm: Chunks in 874.7411ms
2024-03-12T11:53:56.020246Z  INFO {disk_farm_index=0}: subspace_farmer::single_disk_farm: Whole sector in 2.0930412s
2024-03-12T11:53:56.020306Z  INFO {disk_farm_index=0}: subspace_farmer::single_disk_farm: Faster proving method found fastest_mode=ConcurrentChunks
2024-03-12T11:53:56.032291Z  INFO {disk_farm_index=0}: subspace_farmer::single_disk_farm: Bench b
2024-03-12T11:53:56.032379Z  INFO {disk_farm_index=0}: subspace_farmer::single_disk_farm: Benchmarking faster proving method
2024-03-12T11:53:59.250638Z  INFO {disk_farm_index=0}: subspace_farmer::single_disk_farm: Chunks in 879.0477ms
2024-03-12T11:54:01.354109Z  INFO {disk_farm_index=0}: subspace_farmer::single_disk_farm: Whole sector in 2.1033995s
2024-03-12T11:54:02.238123Z  INFO {disk_farm_index=0}: subspace_farmer::single_disk_farm: Chunks in 883.9147ms
2024-03-12T11:54:04.311306Z  INFO {disk_farm_index=0}: subspace_farmer::single_disk_farm: Whole sector in 2.0731097s
2024-03-12T11:54:05.190616Z  INFO {disk_farm_index=0}: subspace_farmer::single_disk_farm: Chunks in 879.1822ms
2024-03-12T11:54:07.295285Z  INFO {disk_farm_index=0}: subspace_farmer::single_disk_farm: Whole sector in 2.1046111s
2024-03-12T11:54:07.295467Z  INFO {disk_farm_index=0}: subspace_farmer::single_disk_farm: Faster proving method found fastest_mode=ConcurrentChunks
2024-03-12T11:54:07.332322Z  INFO {disk_farm_index=0}: subspace_farmer::single_disk_farm::plot_cache: Checking plot cache contents
2024-03-12T11:54:07.344714Z  INFO {disk_farm_index=0}: subspace_farmer::single_disk_farm: Testing unbuffered
2024-03-12T11:54:07.344852Z  INFO {disk_farm_index=0}: subspace_farmer::single_disk_farm: Benchmarking faster proving method
2024-03-12T11:54:10.389886Z  INFO {disk_farm_index=0}: subspace_farmer::single_disk_farm: Chunks in 879.8558ms
2024-03-12T11:54:12.525452Z  INFO {disk_farm_index=0}: subspace_farmer::single_disk_farm: Whole sector in 2.1354964s
2024-03-12T11:54:13.406761Z  INFO {disk_farm_index=0}: subspace_farmer::single_disk_farm: Chunks in 881.0925ms
2024-03-12T11:54:15.547893Z  INFO {disk_farm_index=0}: subspace_farmer::single_disk_farm: Whole sector in 2.141058s
2024-03-12T11:54:16.433126Z  INFO {disk_farm_index=0}: subspace_farmer::single_disk_farm: Chunks in 885.1283ms
2024-03-12T11:54:18.572767Z  INFO {disk_farm_index=0}: subspace_farmer::single_disk_farm: Whole sector in 2.1395353s
2024-03-12T11:54:18.572847Z  INFO {disk_farm_index=0}: subspace_farmer::single_disk_farm: Faster proving method found fastest_mode=ConcurrentChunks
2024-03-12T11:55:24.013585Z  INFO subspace_farmer::single_disk_farm: Proved in 812.0091ms
2024-03-12T11:55:24.019639Z  INFO {disk_farm_index=0}: subspace_farmer::single_disk_farm: Testing regular
2024-03-12T11:55:24.020107Z  INFO {disk_farm_index=0}: subspace_farmer::single_disk_farm: Benchmarking faster proving method
2024-03-12T11:55:27.167394Z  INFO {disk_farm_index=0}: subspace_farmer::single_disk_farm: Chunks in 822.5193ms
2024-03-12T11:55:29.284550Z  INFO {disk_farm_index=0}: subspace_farmer::single_disk_farm: Whole sector in 2.1170764s
2024-03-12T11:55:29.460250Z  INFO {disk_farm_index=0}: subspace_farmer::single_disk_farm: Chunks in 175.4651ms
2024-03-12T11:55:29.634802Z  INFO {disk_farm_index=0}: subspace_farmer::single_disk_farm: Whole sector in 174.4606ms
2024-03-12T11:55:29.813870Z  INFO {disk_farm_index=0}: subspace_farmer::single_disk_farm: Chunks in 178.9478ms
2024-03-12T11:55:30.021300Z  INFO {disk_farm_index=0}: subspace_farmer::single_disk_farm: Whole sector in 207.3368ms
2024-03-12T11:55:30.021607Z  INFO {disk_farm_index=0}: subspace_farmer::single_disk_farm: Faster proving method found fastest_mode=WholeSector
2024-03-12T11:56:31.520928Z  INFO subspace_farmer::single_disk_farm: Proved in 2.9585972s
Single disk farm 0:
  ID: 01HNKZ74W9P6Z7TH06HKP6P61H
  Genesis hash: 0x0c121c75f4ef450f40619e1fca9d1e8e7fbabc42c895bc4790801e85d5a91c34
  Public key: 0x082b5ac8684fb093006a7be16fd4ebae277e76df62a3263f2e06cc35aba30669
  Allocated space: 3.5 TiB (3.9 TB)
  Directory: D:\1
2024-03-12T11:56:31.540828Z  INFO subspace_farmer::commands::farm: Collecting already plotted pieces (this will take some time)...
2024-03-12T11:56:31.540920Z  INFO subspace_farmer::farmer_cache: Initializing piece cache
2024-03-12T11:56:31.997637Z  INFO subspace_farmer::commands::farm: Finished collecting already plotted pieces successfully
2024-03-12T11:56:32.006045Z  INFO {disk_farm_index=0}: subspace_farmer::single_disk_farm::farming: Subscribing to slot info notifications
2024-03-12T11:56:32.006178Z  INFO {disk_farm_index=0}: subspace_farmer::reward_signing: Subscribing to reward signing notifications
2024-03-12T11:56:32.006560Z  INFO {disk_farm_index=0}: subspace_farmer::single_disk_farm::plotting: Subscribing to archived segments
2024-03-12T11:56:32.008582Z  INFO subspace_farmer::commands::farm::dsn: DSN listening on /ip4/192.168.2.213/udp/30533/quic-v1/p2p/12D3KooWMYVm4znwQ2b3yEZNar49bDcdqfoqziqCAmjm96wnfPLu
2024-03-12T11:56:32.009976Z  INFO subspace_farmer::commands::farm::dsn: DSN listening on /ip6/::1/udp/30533/quic-v1/p2p/12D3KooWMYVm4znwQ2b3yEZNar49bDcdqfoqziqCAmjm96wnfPLu
2024-03-12T11:56:32.010029Z  INFO subspace_farmer::commands::farm::dsn: DSN listening on /ip4/127.0.0.1/udp/30533/quic-v1/p2p/12D3KooWMYVm4znwQ2b3yEZNar49bDcdqfoqziqCAmjm96wnfPLu
2024-03-12T11:56:32.013204Z  INFO subspace_farmer::commands::farm::dsn: DSN listening on /ip4/192.168.2.213/tcp/30533/p2p/12D3KooWMYVm4znwQ2b3yEZNar49bDcdqfoqziqCAmjm96wnfPLu
2024-03-12T11:56:32.014257Z  INFO subspace_farmer::commands::farm::dsn: DSN listening on /ip6/::1/tcp/30533/p2p/12D3KooWMYVm4znwQ2b3yEZNar49bDcdqfoqziqCAmjm96wnfPLu
2024-03-12T11:56:32.014441Z  INFO subspace_farmer::commands::farm::dsn: DSN listening on /ip4/127.0.0.1/tcp/30533/p2p/12D3KooWMYVm4znwQ2b3yEZNar49bDcdqfoqziqCAmjm96wnfPLu
2024-03-12T11:56:49.195407Z  INFO subspace_farmer::farmer_cache: Synchronizing piece cache
2024-03-12T11:56:49.283328Z  INFO {disk_farm_index=0}: subspace_farmer::single_disk_farm::plotting: Plotting sector (75.30% complete) sector_index=2750

nazar-pc · March 12, 2024, 11:59am

So this works great! Now try adding farms back and check when you see many seconds for this particular farm. There must be a pattern somewhere.

I’m curious whether it is a particular farm or number of farms or something else that causes these epic slowdowns.

dragonP · March 12, 2024, 1:47pm

Sorry, what kind of pattern are you looking for? In task manager or just in the farmer’s log?

nazar-pc · March 12, 2024, 1:51pm

In logs. Chunks in was 22 seconds for D:\1 before with multiple farms and now it is suddenly 881ms. Subsequent runs are even better because SSD clearly cached it.

I’m wondering what could possibly cause such a slow down. Try to start adding farms and see what numbers you get.

dragonP · March 12, 2024, 2:36pm

This time when I’ve added disks and ran benchmark again, the farmer crashed again and again. Anyway, please find below the logs of 2-3 times in a row.

P/S: It seems this build pick up the plot randomly.

Summary

C:\Subspace Farmer>subspace-farmer-windows-x86_64-skylake-debug-windows.exe farm path=C:\1,size=1600G path=D:\1,size=3900G path=E:\1,size=3900G path=O:\1,size=1900G --node-rpc-url ws://192.168.2.215:9945 --metrics-endpoints 192.168.2.213:2222 --reward-address stxxxxx
2024-03-12T14:03:13.638207Z  INFO subspace_farmer::commands::farm: Connecting to node RPC url=ws://192.168.2.215:9945
2024-03-12T14:03:13.647405Z  INFO subspace_networking::constructor: DSN instance configured. allow_non_global_addresses_in_dht=false peer_id=12D3KooWAmJ7wTwABg55gwFfwzuMyfCFMzuKfc76cxBgPDhGZPMZ protocol_version=/subspace/2/0c121c75f4ef450f40619e1fca9d1e8e7fbabc42c895bc4790801e85d5a91c34
2024-03-12T14:03:13.653439Z  INFO libp2p_swarm: local_peer_id=12D3KooWAmJ7wTwABg55gwFfwzuMyfCFMzuKfc76cxBgPDhGZPMZ
2024-03-12T14:03:13.655242Z  INFO subspace_metrics: Metrics server started. endpoints=[192.168.2.213:2222]
2024-03-12T14:03:13.655288Z  INFO actix_server::builder: starting 2 workers
2024-03-12T14:03:13.655342Z  INFO actix_server::server: Tokio runtime found; starting in existing Tokio runtime
2024-03-12T14:03:16.173335Z  INFO {disk_farm_index=1}: subspace_farmer::single_disk_farm: Bench a
2024-03-12T14:03:16.173481Z  INFO {disk_farm_index=1}: subspace_farmer::single_disk_farm: Benchmarking faster proving method
2024-03-12T14:03:19.398603Z  INFO {disk_farm_index=1}: subspace_farmer::single_disk_farm: Chunks in 880.6305ms
2024-03-12T14:03:21.487159Z  INFO {disk_farm_index=1}: subspace_farmer::single_disk_farm: Whole sector in 2.080326s
2024-03-12T14:03:22.369192Z  INFO {disk_farm_index=1}: subspace_farmer::single_disk_farm: Chunks in 881.9705ms
2024-03-12T14:03:24.452232Z  INFO {disk_farm_index=1}: subspace_farmer::single_disk_farm: Whole sector in 2.0829639s
2024-03-12T14:03:25.334550Z  INFO {disk_farm_index=1}: subspace_farmer::single_disk_farm: Chunks in 882.184ms
2024-03-12T14:03:27.453361Z  INFO {disk_farm_index=1}: subspace_farmer::single_disk_farm: Whole sector in 2.1187422s
2024-03-12T14:03:27.453443Z  INFO {disk_farm_index=1}: subspace_farmer::single_disk_farm: Faster proving method found fastest_mode=ConcurrentChunks
2024-03-12T14:03:27.453611Z  INFO {disk_farm_index=3}: subspace_farmer::single_disk_farm: Bench a
2024-03-12T14:03:27.453691Z  INFO {disk_farm_index=3}: subspace_farmer::single_disk_farm: Benchmarking faster proving method
2024-03-12T14:03:46.859111Z  INFO {disk_farm_index=3}: subspace_farmer::single_disk_farm: Chunks in 15.3145016s
2024-03-12T14:03:50.873223Z  INFO {disk_farm_index=3}: subspace_farmer::single_disk_farm: Whole sector in 4.0140463s
2024-03-12T14:04:05.215871Z  INFO {disk_farm_index=3}: subspace_farmer::single_disk_farm: Chunks in 14.3425095s
2024-03-12T14:04:09.188914Z  INFO {disk_farm_index=3}: subspace_farmer::single_disk_farm: Whole sector in 3.9729631s
2024-03-12T14:04:23.487786Z  INFO {disk_farm_index=3}: subspace_farmer::single_disk_farm: Chunks in 14.2987693s
2024-03-12T14:04:27.449997Z  INFO {disk_farm_index=3}: subspace_farmer::single_disk_farm: Whole sector in 3.9621371s
2024-03-12T14:04:27.450070Z  INFO {disk_farm_index=3}: subspace_farmer::single_disk_farm: Faster proving method found fastest_mode=WholeSector
2024-03-12T14:04:27.532916Z  INFO {disk_farm_index=0}: subspace_farmer::single_disk_farm: Bench a
2024-03-12T14:04:27.533186Z  INFO {disk_farm_index=0}: subspace_farmer::single_disk_farm: Benchmarking faster proving method
2024-03-12T14:04:33.002577Z  INFO {disk_farm_index=0}: subspace_farmer::single_disk_farm: Chunks in 4.9204185s
2024-03-12T14:04:33.293529Z  INFO {disk_farm_index=0}: subspace_farmer::single_disk_farm: Whole sector in 290.8583ms
2024-03-12T14:04:39.069698Z  INFO {disk_farm_index=0}: subspace_farmer::single_disk_farm: Chunks in 5.7761026s
2024-03-12T14:04:39.357464Z  INFO {disk_farm_index=0}: subspace_farmer::single_disk_farm: Whole sector in 287.6135ms
2024-03-12T14:04:44.419114Z  INFO {disk_farm_index=0}: subspace_farmer::single_disk_farm: Chunks in 5.0615278s
2024-03-12T14:04:44.687246Z  INFO {disk_farm_index=0}: subspace_farmer::single_disk_farm: Whole sector in 268.0608ms
2024-03-12T14:04:44.687306Z  INFO {disk_farm_index=0}: subspace_farmer::single_disk_farm: Faster proving method found fastest_mode=WholeSector
2024-03-12T14:04:44.687434Z  INFO {disk_farm_index=2}: subspace_farmer::single_disk_farm: Bench a
2024-03-12T14:04:44.687511Z  INFO {disk_farm_index=2}: subspace_farmer::single_disk_farm: Benchmarking faster proving method
2024-03-12T14:04:47.217811Z  INFO {disk_farm_index=2}: subspace_farmer::single_disk_farm: Chunks in 447.8227ms
2024-03-12T14:04:49.238237Z  INFO {disk_farm_index=2}: subspace_farmer::single_disk_farm: Whole sector in 2.0203438s
2024-03-12T14:04:49.667367Z  INFO {disk_farm_index=2}: subspace_farmer::single_disk_farm: Chunks in 428.9043ms
2024-03-12T14:04:51.690851Z  INFO {disk_farm_index=2}: subspace_farmer::single_disk_farm: Whole sector in 2.0234097s
2024-03-12T14:04:52.120164Z  INFO {disk_farm_index=2}: subspace_farmer::single_disk_farm: Chunks in 429.177ms
2024-03-12T14:04:54.147024Z  INFO {disk_farm_index=2}: subspace_farmer::single_disk_farm: Whole sector in 2.0267761s
2024-03-12T14:04:54.147157Z  INFO {disk_farm_index=2}: subspace_farmer::single_disk_farm: Faster proving method found fastest_mode=ConcurrentChunks
2024-03-12T14:04:54.213764Z  INFO {disk_farm_index=3}: subspace_farmer::single_disk_farm: Bench b
2024-03-12T14:04:54.213838Z  INFO {disk_farm_index=3}: subspace_farmer::single_disk_farm: Benchmarking faster proving method
2024-03-12T14:05:12.947213Z  INFO {disk_farm_index=3}: subspace_farmer::single_disk_farm: Chunks in 14.497822s
2024-03-12T14:05:16.923339Z  INFO {disk_farm_index=3}: subspace_farmer::single_disk_farm: Whole sector in 3.9758904s
2024-03-12T14:05:31.481441Z  INFO {disk_farm_index=3}: subspace_farmer::single_disk_farm: Chunks in 14.5579749s
2024-03-12T14:05:35.461230Z  INFO {disk_farm_index=3}: subspace_farmer::single_disk_farm: Whole sector in 3.9796964s
2024-03-12T14:05:49.807080Z  INFO {disk_farm_index=3}: subspace_farmer::single_disk_farm: Chunks in 14.3457243s
2024-03-12T14:05:53.773144Z  INFO {disk_farm_index=3}: subspace_farmer::single_disk_farm: Whole sector in 3.9659937s
2024-03-12T14:05:53.773295Z  INFO {disk_farm_index=3}: subspace_farmer::single_disk_farm: Faster proving method found fastest_mode=WholeSector
2024-03-12T14:05:53.773381Z  INFO {disk_farm_index=2}: subspace_farmer::single_disk_farm: Bench b
2024-03-12T14:05:53.773473Z  INFO {disk_farm_index=2}: subspace_farmer::single_disk_farm: Benchmarking faster proving method
2024-03-12T14:05:56.279096Z  INFO {disk_farm_index=2}: subspace_farmer::single_disk_farm: Chunks in 429.6047ms
2024-03-12T14:05:58.308724Z  INFO {disk_farm_index=2}: subspace_farmer::single_disk_farm: Whole sector in 2.0295251s
2024-03-12T14:05:58.737500Z  INFO {disk_farm_index=2}: subspace_farmer::single_disk_farm: Chunks in 428.6895ms
2024-03-12T14:06:00.760064Z  INFO {disk_farm_index=2}: subspace_farmer::single_disk_farm: Whole sector in 2.0224889s
2024-03-12T14:06:01.187840Z  INFO {disk_farm_index=2}: subspace_farmer::single_disk_farm: Chunks in 427.6938ms
2024-03-12T14:06:03.223571Z  INFO {disk_farm_index=2}: subspace_farmer::single_disk_farm: Whole sector in 2.0356497s
2024-03-12T14:06:03.223653Z  INFO {disk_farm_index=2}: subspace_farmer::single_disk_farm: Faster proving method found fastest_mode=ConcurrentChunks
2024-03-12T14:06:03.270451Z  INFO {disk_farm_index=0}: subspace_farmer::single_disk_farm: Bench b
2024-03-12T14:06:03.270558Z  INFO {disk_farm_index=0}: subspace_farmer::single_disk_farm: Benchmarking faster proving method
2024-03-12T14:06:08.684203Z  INFO {disk_farm_index=0}: subspace_farmer::single_disk_farm: Chunks in 4.8793092s
2024-03-12T14:06:08.951392Z  INFO {disk_farm_index=0}: subspace_farmer::single_disk_farm: Whole sector in 267.0603ms
2024-03-12T14:06:13.828794Z  INFO {disk_farm_index=0}: subspace_farmer::single_disk_farm: Chunks in 4.877271s
2024-03-12T14:06:14.097023Z  INFO {disk_farm_index=0}: subspace_farmer::single_disk_farm: Whole sector in 268.1592ms
2024-03-12T14:06:18.964227Z  INFO {disk_farm_index=0}: subspace_farmer::single_disk_farm: Chunks in 4.8670816s
2024-03-12T14:06:19.232787Z  INFO {disk_farm_index=0}: subspace_farmer::single_disk_farm: Whole sector in 268.46ms
2024-03-12T14:06:19.232875Z  INFO {disk_farm_index=0}: subspace_farmer::single_disk_farm: Faster proving method found fastest_mode=WholeSector
2024-03-12T14:06:19.233036Z  INFO {disk_farm_index=1}: subspace_farmer::single_disk_farm: Bench b
2024-03-12T14:06:19.233116Z  INFO {disk_farm_index=1}: subspace_farmer::single_disk_farm: Benchmarking faster proving method
2024-03-12T14:06:22.267118Z  INFO {disk_farm_index=1}: subspace_farmer::single_disk_farm: Chunks in 881.0996ms
2024-03-12T14:06:24.360926Z  INFO {disk_farm_index=1}: subspace_farmer::single_disk_farm: Whole sector in 2.0937384s
2024-03-12T14:06:25.242933Z  INFO {disk_farm_index=1}: subspace_farmer::single_disk_farm: Chunks in 881.8633ms
2024-03-12T14:06:27.332240Z  INFO {disk_farm_index=1}: subspace_farmer::single_disk_farm: Whole sector in 2.089221s
2024-03-12T14:06:28.213053Z  INFO {disk_farm_index=1}: subspace_farmer::single_disk_farm: Chunks in 880.6499ms
2024-03-12T14:06:30.303081Z  INFO {disk_farm_index=1}: subspace_farmer::single_disk_farm: Whole sector in 2.0899422s
2024-03-12T14:06:30.303159Z  INFO {disk_farm_index=1}: subspace_farmer::single_disk_farm: Faster proving method found fastest_mode=ConcurrentChunks
2024-03-12T14:06:30.355733Z  INFO {disk_farm_index=1}: subspace_farmer::single_disk_farm::plot_cache: Checking plot cache contents
2024-03-12T14:06:30.357623Z  INFO {disk_farm_index=2}: subspace_farmer::single_disk_farm::plot_cache: Checking plot cache contents
2024-03-12T14:06:30.358135Z  INFO {disk_farm_index=3}: subspace_farmer::single_disk_farm::plot_cache: Checking plot cache contents
2024-03-12T14:06:30.361143Z  INFO {disk_farm_index=0}: subspace_farmer::single_disk_farm::plot_cache: Checking plot cache contents
2024-03-12T14:06:30.367418Z  INFO {disk_farm_index=3}: subspace_farmer::single_disk_farm: Testing unbuffered
2024-03-12T14:06:30.367471Z  INFO {disk_farm_index=3}: subspace_farmer::single_disk_farm: Benchmarking faster proving method
2024-03-12T14:06:48.965860Z  INFO {disk_farm_index=3}: subspace_farmer::single_disk_farm: Chunks in 14.3739166s
2024-03-12T14:06:52.949682Z  INFO {disk_farm_index=3}: subspace_farmer::single_disk_farm: Whole sector in 3.9837475s
2024-03-12T14:07:07.385242Z  INFO {disk_farm_index=3}: subspace_farmer::single_disk_farm: Chunks in 14.4354767s
2024-03-12T14:07:11.364032Z  INFO {disk_farm_index=3}: subspace_farmer::single_disk_farm: Whole sector in 3.9787048s
2024-03-12T14:07:25.872369Z  INFO {disk_farm_index=3}: subspace_farmer::single_disk_farm: Chunks in 14.5081701s
2024-03-12T14:07:29.856423Z  INFO {disk_farm_index=3}: subspace_farmer::single_disk_farm: Whole sector in 3.9839663s
2024-03-12T14:07:29.856598Z  INFO {disk_farm_index=3}: subspace_farmer::single_disk_farm: Faster proving method found fastest_mode=WholeSector
2024-03-12T14:07:29.856735Z  INFO {disk_farm_index=2}: subspace_farmer::single_disk_farm: Testing unbuffered
2024-03-12T14:07:29.857000Z  INFO {disk_farm_index=2}: subspace_farmer::single_disk_farm: Benchmarking faster proving method
2024-03-12T14:07:32.363309Z  INFO {disk_farm_index=2}: subspace_farmer::single_disk_farm: Chunks in 428.2666ms
2024-03-12T14:07:34.383704Z  INFO {disk_farm_index=2}: subspace_farmer::single_disk_farm: Whole sector in 2.0203083s
2024-03-12T14:07:34.811769Z  INFO {disk_farm_index=2}: subspace_farmer::single_disk_farm: Chunks in 427.9358ms
2024-03-12T14:07:36.852733Z  INFO {disk_farm_index=2}: subspace_farmer::single_disk_farm: Whole sector in 2.0408887s
2024-03-12T14:07:37.279422Z  INFO {disk_farm_index=2}: subspace_farmer::single_disk_farm: Chunks in 426.6133ms
2024-03-12T14:07:39.305179Z  INFO {disk_farm_index=2}: subspace_farmer::single_disk_farm: Whole sector in 2.0256824s
2024-03-12T14:07:39.305257Z  INFO {disk_farm_index=2}: subspace_farmer::single_disk_farm: Faster proving method found fastest_mode=ConcurrentChunks
2024-03-12T14:07:39.352064Z  INFO {disk_farm_index=1}: subspace_farmer::single_disk_farm: Testing unbuffered
2024-03-12T14:07:39.352235Z  INFO {disk_farm_index=1}: subspace_farmer::single_disk_farm: Benchmarking faster proving method
2024-03-12T14:07:40.106544Z  INFO subspace_farmer::single_disk_farm: Proved in 5.1552277s
2024-03-12T14:07:42.594803Z  INFO {disk_farm_index=1}: subspace_farmer::single_disk_farm: Chunks in 881.3332ms
2024-03-12T14:07:44.672355Z  INFO {disk_farm_index=1}: subspace_farmer::single_disk_farm: Whole sector in 2.0774573s
2024-03-12T14:07:45.553588Z  INFO {disk_farm_index=1}: subspace_farmer::single_disk_farm: Chunks in 881.0584ms
2024-03-12T14:07:47.642073Z  INFO {disk_farm_index=1}: subspace_farmer::single_disk_farm: Whole sector in 2.0883966s
2024-03-12T14:07:48.521545Z  INFO {disk_farm_index=1}: subspace_farmer::single_disk_farm: Chunks in 879.3716ms
2024-03-12T14:07:50.594983Z  INFO {disk_farm_index=1}: subspace_farmer::single_disk_farm: Whole sector in 2.0733406s
2024-03-12T14:07:50.595053Z  INFO {disk_farm_index=1}: subspace_farmer::single_disk_farm: Faster proving method found fastest_mode=ConcurrentChunks
2024-03-12T14:07:50.641427Z  INFO {disk_farm_index=0}: subspace_farmer::single_disk_farm: Testing unbuffered
2024-03-12T14:07:50.641616Z  INFO {disk_farm_index=0}: subspace_farmer::single_disk_farm: Benchmarking faster proving method
2024-03-12T14:07:56.170424Z  INFO {disk_farm_index=0}: subspace_farmer::single_disk_farm: Chunks in 4.9624925s
2024-03-12T14:07:56.444345Z  INFO {disk_farm_index=0}: subspace_farmer::single_disk_farm: Whole sector in 273.808ms
2024-03-12T14:08:01.394123Z  INFO {disk_farm_index=0}: subspace_farmer::single_disk_farm: Chunks in 4.9496934s
2024-03-12T14:08:01.665002Z  INFO {disk_farm_index=0}: subspace_farmer::single_disk_farm: Whole sector in 270.7919ms
2024-03-12T14:08:06.595197Z  INFO {disk_farm_index=0}: subspace_farmer::single_disk_farm: Chunks in 4.9301148s
2024-03-12T14:08:06.868722Z  INFO {disk_farm_index=0}: subspace_farmer::single_disk_farm: Whole sector in 273.4297ms
2024-03-12T14:08:06.868845Z  INFO {disk_farm_index=0}: subspace_farmer::single_disk_farm: Faster proving method found fastest_mode=WholeSector
2024-03-12T14:08:18.424695Z  INFO subspace_farmer::single_disk_farm: Proved in 1.0102868s
2024-03-12T14:08:33.843945Z  INFO subspace_farmer::single_disk_farm: Proved in 506.9568ms
2024-03-12T14:08:48.114835Z  INFO subspace_farmer::single_disk_farm: Proved in 1.3128304s
2024-03-12T14:08:48.297785Z  INFO {disk_farm_index=1}: subspace_farmer::single_disk_farm: Testing regular
2024-03-12T14:08:48.298127Z  INFO {disk_farm_index=1}: subspace_farmer::single_disk_farm: Benchmarking faster proving method
2024-03-12T14:08:51.303656Z  INFO {disk_farm_index=1}: subspace_farmer::single_disk_farm: Chunks in 812.7438ms
2024-03-12T14:08:53.372332Z  INFO {disk_farm_index=1}: subspace_farmer::single_disk_farm: Whole sector in 2.0685962s
2024-03-12T14:08:53.546104Z  INFO {disk_farm_index=1}: subspace_farmer::single_disk_farm: Chunks in 173.5309ms
2024-03-12T14:08:53.693304Z  INFO {disk_farm_index=1}: subspace_farmer::single_disk_farm: Whole sector in 147.1073ms
2024-03-12T14:08:53.870622Z  INFO {disk_farm_index=1}: subspace_farmer::single_disk_farm: Chunks in 177.1854ms
2024-03-12T14:08:54.027101Z  INFO {disk_farm_index=1}: subspace_farmer::single_disk_farm: Whole sector in 156.3891ms
2024-03-12T14:08:54.027320Z  INFO {disk_farm_index=1}: subspace_farmer::single_disk_farm: Faster proving method found fastest_mode=WholeSector
2024-03-12T14:08:54.057558Z  INFO {disk_farm_index=3}: subspace_farmer::single_disk_farm: Testing regular
2024-03-12T14:08:54.057698Z  INFO {disk_farm_index=3}: subspace_farmer::single_disk_farm: Benchmarking faster proving method
2024-03-12T14:08:59.819909Z  INFO {disk_farm_index=3}: subspace_farmer::single_disk_farm: Chunks in 1.7374677s
2024-03-12T14:09:03.749520Z  INFO {disk_farm_index=3}: subspace_farmer::single_disk_farm: Whole sector in 3.9294847s
2024-03-12T14:09:03.926465Z  INFO {disk_farm_index=3}: subspace_farmer::single_disk_farm: Chunks in 176.8457ms
2024-03-12T14:09:04.082234Z  INFO {disk_farm_index=3}: subspace_farmer::single_disk_farm: Whole sector in 155.6756ms
2024-03-12T14:09:04.258548Z  INFO {disk_farm_index=3}: subspace_farmer::single_disk_farm: Chunks in 176.109ms
2024-03-12T14:09:04.413370Z  INFO {disk_farm_index=3}: subspace_farmer::single_disk_farm: Whole sector in 154.7297ms
2024-03-12T14:09:04.413465Z  INFO {disk_farm_index=3}: subspace_farmer::single_disk_farm: Faster proving method found fastest_mode=WholeSector
2024-03-12T14:09:04.413575Z  INFO {disk_farm_index=2}: subspace_farmer::single_disk_farm: Testing regular
2024-03-12T14:09:04.414039Z  INFO {disk_farm_index=2}: subspace_farmer::single_disk_farm: Benchmarking faster proving method
2024-03-12T14:09:07.113482Z  INFO {disk_farm_index=2}: subspace_farmer::single_disk_farm: Chunks in 610.764ms
2024-03-12T14:09:09.256244Z  INFO {disk_farm_index=2}: subspace_farmer::single_disk_farm: Whole sector in 2.1426667s
2024-03-12T14:09:09.461216Z  INFO {disk_farm_index=2}: subspace_farmer::single_disk_farm: Chunks in 204.7313ms
2024-03-12T14:09:09.726756Z  INFO {disk_farm_index=2}: subspace_farmer::single_disk_farm: Whole sector in 265.4105ms
2024-03-12T14:09:09.933350Z  INFO {disk_farm_index=2}: subspace_farmer::single_disk_farm: Chunks in 206.451ms
2024-03-12T14:09:10.135965Z  INFO {disk_farm_index=2}: subspace_farmer::single_disk_farm: Whole sector in 202.4988ms
2024-03-12T14:09:10.136205Z  INFO {disk_farm_index=2}: subspace_farmer::single_disk_farm: Faster proving method found fastest_mode=WholeSector
2024-03-12T14:09:10.149267Z  INFO {disk_farm_index=0}: subspace_farmer::single_disk_farm: Testing regular
2024-03-12T14:09:10.149385Z  INFO {disk_farm_index=0}: subspace_farmer::single_disk_farm: Benchmarking faster proving method
2024-03-12T14:09:10.837466Z  INFO {disk_farm_index=0}: subspace_farmer::single_disk_farm: Chunks in 277.223ms
2024-03-12T14:09:10.873846Z  INFO subspace_farmer::single_disk_farm: Proved in 1.6556731s
2024-03-12T14:09:11.192735Z  INFO {disk_farm_index=0}: subspace_farmer::single_disk_farm: Whole sector in 355.1508ms
2024-03-12T14:09:11.389541Z  INFO {disk_farm_index=0}: subspace_farmer::single_disk_farm: Chunks in 196.5851ms
2024-03-12T14:09:11.601839Z  INFO {disk_farm_index=0}: subspace_farmer::single_disk_farm: Whole sector in 212.2016ms
2024-03-12T14:09:11.796254Z  INFO {disk_farm_index=0}: subspace_farmer::single_disk_farm: Chunks in 194.2308ms
2024-03-12T14:09:11.974225Z  INFO {disk_farm_index=0}: subspace_farmer::single_disk_farm: Whole sector in 177.8558ms
2024-03-12T14:09:11.974315Z  INFO {disk_farm_index=0}: subspace_farmer::single_disk_farm: Faster proving method found fastest_mode=WholeSector
2024-03-12T14:09:25.396277Z  INFO subspace_farmer::single_disk_farm: Proved in 1.2768809s
memory allocation of 8192000 bytes failed

C:\Subspace Farmer>goto loop

C:\Subspace Farmer>subspace-farmer-windows-x86_64-skylake-debug-windows.exe farm path=C:\1,size=1600G path=D:\1,size=3900G path=E:\1,size=3900G path=O:\1,size=1900G --node-rpc-url ws://192.168.2.215:9945 --metrics-endpoints 192.168.2.213:2222 --reward-address st6bf1gCkUszz48jBMcdr6ur1Xs4R2hghiTGNhT7UZoSS5Ue2
2024-03-12T14:10:07.023269Z  INFO subspace_farmer::commands::farm: Connecting to node RPC url=ws://192.168.2.215:9945
2024-03-12T14:10:07.026356Z  INFO subspace_networking::constructor: DSN instance configured. allow_non_global_addresses_in_dht=false peer_id=12D3KooWAmJ7wTwABg55gwFfwzuMyfCFMzuKfc76cxBgPDhGZPMZ protocol_version=/subspace/2/0c121c75f4ef450f40619e1fca9d1e8e7fbabc42c895bc4790801e85d5a91c34
2024-03-12T14:10:07.028778Z  INFO libp2p_swarm: local_peer_id=12D3KooWAmJ7wTwABg55gwFfwzuMyfCFMzuKfc76cxBgPDhGZPMZ
2024-03-12T14:10:07.030280Z  INFO subspace_metrics: Metrics server started. endpoints=[192.168.2.213:2222]
2024-03-12T14:10:07.030337Z  INFO actix_server::builder: starting 2 workers
2024-03-12T14:10:07.030404Z  INFO actix_server::server: Tokio runtime found; starting in existing Tokio runtime
2024-03-12T14:10:09.015285Z  INFO {disk_farm_index=0}: subspace_farmer::single_disk_farm: Bench a
2024-03-12T14:10:09.015382Z  INFO {disk_farm_index=0}: subspace_farmer::single_disk_farm: Benchmarking faster proving method
2024-03-12T14:10:14.703160Z  INFO {disk_farm_index=0}: subspace_farmer::single_disk_farm: Chunks in 5.1397951s
2024-03-12T14:10:14.986246Z  INFO {disk_farm_index=0}: subspace_farmer::single_disk_farm: Whole sector in 283.0034ms
2024-03-12T14:10:20.380151Z  INFO {disk_farm_index=0}: subspace_farmer::single_disk_farm: Chunks in 5.3938336s
2024-03-12T14:10:20.667038Z  INFO {disk_farm_index=0}: subspace_farmer::single_disk_farm: Whole sector in 286.8014ms
2024-03-12T14:10:26.017035Z  INFO {disk_farm_index=0}: subspace_farmer::single_disk_farm: Chunks in 5.349927s
2024-03-12T14:10:26.301171Z  INFO {disk_farm_index=0}: subspace_farmer::single_disk_farm: Whole sector in 284.0156ms
2024-03-12T14:10:26.301238Z  INFO {disk_farm_index=0}: subspace_farmer::single_disk_farm: Faster proving method found fastest_mode=WholeSector
2024-03-12T14:10:26.301424Z  INFO {disk_farm_index=2}: subspace_farmer::single_disk_farm: Bench a
2024-03-12T14:10:26.301566Z  INFO {disk_farm_index=2}: subspace_farmer::single_disk_farm: Benchmarking faster proving method
2024-03-12T14:10:38.151055Z  INFO {disk_farm_index=2}: subspace_farmer::single_disk_farm: Chunks in 9.5946151s
2024-03-12T14:10:40.264467Z  INFO {disk_farm_index=2}: subspace_farmer::single_disk_farm: Whole sector in 2.1133206s
2024-03-12T14:10:49.464524Z  INFO {disk_farm_index=2}: subspace_farmer::single_disk_farm: Chunks in 9.1999832s
2024-03-12T14:10:51.581062Z  INFO {disk_farm_index=2}: subspace_farmer::single_disk_farm: Whole sector in 2.1164416s
2024-03-12T14:11:00.643672Z  INFO {disk_farm_index=2}: subspace_farmer::single_disk_farm: Chunks in 9.0625347s
2024-03-12T14:11:02.747688Z  INFO {disk_farm_index=2}: subspace_farmer::single_disk_farm: Whole sector in 2.1039177s
2024-03-12T14:11:02.747866Z  INFO {disk_farm_index=2}: subspace_farmer::single_disk_farm: Faster proving method found fastest_mode=WholeSector
2024-03-12T14:11:02.794210Z  INFO {disk_farm_index=3}: subspace_farmer::single_disk_farm: Bench a
2024-03-12T14:11:02.794467Z  INFO {disk_farm_index=3}: subspace_farmer::single_disk_farm: Benchmarking faster proving method
2024-03-12T14:11:21.719501Z  INFO {disk_farm_index=3}: subspace_farmer::single_disk_farm: Chunks in 14.6046465s
2024-03-12T14:11:25.770849Z  INFO {disk_farm_index=3}: subspace_farmer::single_disk_farm: Whole sector in 4.0512529s
2024-03-12T14:11:40.158124Z  INFO {disk_farm_index=3}: subspace_farmer::single_disk_farm: Chunks in 14.3869015s
2024-03-12T14:11:44.199028Z  INFO {disk_farm_index=3}: subspace_farmer::single_disk_farm: Whole sector in 4.040815s
2024-03-12T14:11:58.591724Z  INFO {disk_farm_index=3}: subspace_farmer::single_disk_farm: Chunks in 14.3926138s
2024-03-12T14:12:02.654761Z  INFO {disk_farm_index=3}: subspace_farmer::single_disk_farm: Whole sector in 4.0629431s
2024-03-12T14:12:02.655014Z  INFO {disk_farm_index=3}: subspace_farmer::single_disk_farm: Faster proving method found fastest_mode=WholeSector
2024-03-12T14:12:02.655495Z  INFO {disk_farm_index=1}: subspace_farmer::single_disk_farm: Bench a
2024-03-12T14:12:02.655757Z  INFO {disk_farm_index=1}: subspace_farmer::single_disk_farm: Benchmarking faster proving method
2024-03-12T14:12:20.624546Z  INFO {disk_farm_index=1}: subspace_farmer::single_disk_farm: Chunks in 15.7226837s
2024-03-12T14:12:22.807349Z  INFO {disk_farm_index=1}: subspace_farmer::single_disk_farm: Whole sector in 2.1827151s
2024-03-12T14:12:38.491776Z  INFO {disk_farm_index=1}: subspace_farmer::single_disk_farm: Chunks in 15.6842457s
2024-03-12T14:12:40.642702Z  INFO {disk_farm_index=1}: subspace_farmer::single_disk_farm: Whole sector in 2.1508293s
2024-03-12T14:12:56.165592Z  INFO {disk_farm_index=1}: subspace_farmer::single_disk_farm: Chunks in 15.5228153s
2024-03-12T14:12:58.330928Z  INFO {disk_farm_index=1}: subspace_farmer::single_disk_farm: Whole sector in 2.1652459s
2024-03-12T14:12:58.331027Z  INFO {disk_farm_index=1}: subspace_farmer::single_disk_farm: Faster proving method found fastest_mode=WholeSector
2024-03-12T14:12:58.384425Z  INFO {disk_farm_index=3}: subspace_farmer::single_disk_farm: Bench b
2024-03-12T14:12:58.384507Z  INFO {disk_farm_index=3}: subspace_farmer::single_disk_farm: Benchmarking faster proving method
2024-03-12T14:13:17.406816Z  INFO {disk_farm_index=3}: subspace_farmer::single_disk_farm: Chunks in 14.7190027s
2024-03-12T14:13:21.476096Z  INFO {disk_farm_index=3}: subspace_farmer::single_disk_farm: Whole sector in 4.0691887s
2024-03-12T14:13:35.918873Z  INFO {disk_farm_index=3}: subspace_farmer::single_disk_farm: Chunks in 14.4425964s
2024-03-12T14:13:39.972011Z  INFO {disk_farm_index=3}: subspace_farmer::single_disk_farm: Whole sector in 4.0530299s
2024-03-12T14:13:54.395380Z  INFO {disk_farm_index=3}: subspace_farmer::single_disk_farm: Chunks in 14.4232862s
2024-03-12T14:13:58.446167Z  INFO {disk_farm_index=3}: subspace_farmer::single_disk_farm: Whole sector in 4.0506702s
2024-03-12T14:13:58.446340Z  INFO {disk_farm_index=3}: subspace_farmer::single_disk_farm: Faster proving method found fastest_mode=WholeSector
2024-03-12T14:13:58.446469Z  INFO {disk_farm_index=2}: subspace_farmer::single_disk_farm: Bench b
2024-03-12T14:13:58.446593Z  INFO {disk_farm_index=2}: subspace_farmer::single_disk_farm: Benchmarking faster proving method
2024-03-12T14:14:09.902614Z  INFO {disk_farm_index=2}: subspace_farmer::single_disk_farm: Chunks in 9.2714296s
2024-03-12T14:14:12.018652Z  INFO {disk_farm_index=2}: subspace_farmer::single_disk_farm: Whole sector in 2.1159617s
2024-03-12T14:14:21.280463Z  INFO {disk_farm_index=2}: subspace_farmer::single_disk_farm: Chunks in 9.2617293s
2024-03-12T14:14:23.396769Z  INFO {disk_farm_index=2}: subspace_farmer::single_disk_farm: Whole sector in 2.1162018s
2024-03-12T14:14:32.653317Z  INFO {disk_farm_index=2}: subspace_farmer::single_disk_farm: Chunks in 9.2564712s
2024-03-12T14:14:34.765444Z  INFO {disk_farm_index=2}: subspace_farmer::single_disk_farm: Whole sector in 2.1120355s
2024-03-12T14:14:34.765526Z  INFO {disk_farm_index=2}: subspace_farmer::single_disk_farm: Faster proving method found fastest_mode=WholeSector
2024-03-12T14:14:34.807551Z  INFO {disk_farm_index=0}: subspace_farmer::single_disk_farm: Bench b
2024-03-12T14:14:34.807699Z  INFO {disk_farm_index=0}: subspace_farmer::single_disk_farm: Benchmarking faster proving method
2024-03-12T14:14:40.327941Z  INFO {disk_farm_index=0}: subspace_farmer::single_disk_farm: Chunks in 4.9806338s
2024-03-12T14:14:40.606741Z  INFO {disk_farm_index=0}: subspace_farmer::single_disk_farm: Whole sector in 278.7123ms
2024-03-12T14:14:45.615157Z  INFO {disk_farm_index=0}: subspace_farmer::single_disk_farm: Chunks in 5.0083208s
2024-03-12T14:14:45.888930Z  INFO {disk_farm_index=0}: subspace_farmer::single_disk_farm: Whole sector in 273.6761ms
2024-03-12T14:14:50.891706Z  INFO {disk_farm_index=0}: subspace_farmer::single_disk_farm: Chunks in 5.0026525s
2024-03-12T14:14:51.164353Z  INFO {disk_farm_index=0}: subspace_farmer::single_disk_farm: Whole sector in 272.565ms
2024-03-12T14:14:51.164425Z  INFO {disk_farm_index=0}: subspace_farmer::single_disk_farm: Faster proving method found fastest_mode=WholeSector
2024-03-12T14:14:51.164564Z  INFO {disk_farm_index=1}: subspace_farmer::single_disk_farm: Bench b
2024-03-12T14:14:51.164646Z  INFO {disk_farm_index=1}: subspace_farmer::single_disk_farm: Benchmarking faster proving method
2024-03-12T14:15:08.869620Z  INFO {disk_farm_index=1}: subspace_farmer::single_disk_farm: Chunks in 15.4767487s
2024-03-12T14:15:11.031035Z  INFO {disk_farm_index=1}: subspace_farmer::single_disk_farm: Whole sector in 2.161332s
2024-03-12T14:15:26.332754Z  INFO {disk_farm_index=1}: subspace_farmer::single_disk_farm: Chunks in 15.3014865s
2024-03-12T14:15:28.499649Z  INFO {disk_farm_index=1}: subspace_farmer::single_disk_farm: Whole sector in 2.1668203s
2024-03-12T14:15:43.902094Z  INFO {disk_farm_index=1}: subspace_farmer::single_disk_farm: Chunks in 15.4023588s
2024-03-12T14:15:46.070194Z  INFO {disk_farm_index=1}: subspace_farmer::single_disk_farm: Whole sector in 2.167978s
2024-03-12T14:15:46.070259Z  INFO {disk_farm_index=1}: subspace_farmer::single_disk_farm: Faster proving method found fastest_mode=WholeSector
2024-03-12T14:15:46.110398Z  INFO {disk_farm_index=1}: subspace_farmer::single_disk_farm::plot_cache: Checking plot cache contents
2024-03-12T14:15:46.114316Z  INFO {disk_farm_index=2}: subspace_farmer::single_disk_farm::plot_cache: Checking plot cache contents
2024-03-12T14:15:46.114879Z  INFO {disk_farm_index=3}: subspace_farmer::single_disk_farm::plot_cache: Checking plot cache contents
2024-03-12T14:15:46.118254Z  INFO {disk_farm_index=0}: subspace_farmer::single_disk_farm::plot_cache: Checking plot cache contents
2024-03-12T14:15:46.129372Z  INFO {disk_farm_index=3}: subspace_farmer::single_disk_farm: Testing unbuffered
2024-03-12T14:15:46.129462Z  INFO {disk_farm_index=3}: subspace_farmer::single_disk_farm: Benchmarking faster proving method
2024-03-12T14:16:04.795710Z  INFO {disk_farm_index=3}: subspace_farmer::single_disk_farm: Chunks in 14.3802773s
2024-03-12T14:16:08.836702Z  INFO {disk_farm_index=3}: subspace_farmer::single_disk_farm: Whole sector in 4.0408659s
2024-03-12T14:16:23.284012Z  INFO {disk_farm_index=3}: subspace_farmer::single_disk_farm: Chunks in 14.4470646s
2024-03-12T14:16:27.336976Z  INFO {disk_farm_index=3}: subspace_farmer::single_disk_farm: Whole sector in 4.0528772s
2024-03-12T14:16:41.739262Z  INFO {disk_farm_index=3}: subspace_farmer::single_disk_farm: Chunks in 14.4022163s
2024-03-12T14:16:45.769175Z  INFO {disk_farm_index=3}: subspace_farmer::single_disk_farm: Whole sector in 4.0298203s
2024-03-12T14:16:45.769457Z  INFO {disk_farm_index=3}: subspace_farmer::single_disk_farm: Faster proving method found fastest_mode=WholeSector
2024-03-12T14:16:45.769813Z  INFO {disk_farm_index=2}: subspace_farmer::single_disk_farm: Testing unbuffered
2024-03-12T14:16:45.770044Z  INFO {disk_farm_index=2}: subspace_farmer::single_disk_farm: Benchmarking faster proving method
2024-03-12T14:16:55.913035Z  INFO subspace_farmer::single_disk_farm: Proved in 5.0945354s
2024-03-12T14:16:56.713747Z  INFO {disk_farm_index=2}: subspace_farmer::single_disk_farm: Chunks in 8.7800415s
2024-03-12T14:16:58.839431Z  INFO {disk_farm_index=2}: subspace_farmer::single_disk_farm: Whole sector in 2.1256026s
2024-03-12T14:17:07.841314Z  INFO {disk_farm_index=2}: subspace_farmer::single_disk_farm: Chunks in 9.0018089s
2024-03-12T14:17:09.957468Z  INFO {disk_farm_index=2}: subspace_farmer::single_disk_farm: Whole sector in 2.1160696s
2024-03-12T14:17:19.021840Z  INFO {disk_farm_index=2}: subspace_farmer::single_disk_farm: Chunks in 9.0642287s
2024-03-12T14:17:21.138513Z  INFO {disk_farm_index=2}: subspace_farmer::single_disk_farm: Whole sector in 2.1165807s
2024-03-12T14:17:21.138718Z  INFO {disk_farm_index=2}: subspace_farmer::single_disk_farm: Faster proving method found fastest_mode=WholeSector

nazar-pc · March 12, 2024, 3:19pm

That is not what I needed. Since one farm works, I’d like you to add farms back one at a time and observe if timing in self-benchmarks changes drastically and when.

micar · March 12, 2024, 4:06pm

I have the same issue after I moved from mar-04 to mar-11.
Config with running 2 farm instances on the same PC:
(All SSD are NVME on motherboard)

.\subspace-farmer-windows-x86_64-skylake-gemini-3h-2024-mar-11.exe farm path=J:\Subspace\farms_1,size=1000GB path=J:\Subspace\farms_2,size=1000GB
path=K:\Subspace\farms_3,size=1000GB path=K:\Subspace\farms_4,size=1000GB
MISSED ALL REWARDS
.\subspace-farmer-windows-x86_64-skylake-gemini-3h-2024-mar-11.exe farm --farm-during-initial-plotting true --record-encoding-concurrency 5 path=L:\Subspace\farms_5,size=1000GB
Plotting sector - 70%
REWARDS ACUIRED OK

mystery3000 · March 12, 2024, 6:55pm

I’m also still having issues with misses with my Windows 10 farm (Ryzen 5800X/64GB RAM since mar-08 build and mar-11 build haven’t fixed it.

I have several drives but even if I downscale farm to a single good drive (reasonably good NVME drive in PCIE 3.0 M.2 port, benchmarks to 3GB/s), I’m still having too high solve times on mar-11 build.

This is how this drive benchmarks on feb-19 build:

Summary

.\subspace-farmer-windows-x86_64-skylake-gemini-3h-2024-feb-19.exe benchmark prove J:\Subspace
Benchmarking prove/plot/rayon
Benchmarking prove/plot/rayon: Warming up for 3.0000 s

Warning: Unable to complete 10 samples in 5.0s. You may wish to increase target time to 7.2s.
Benchmarking prove/plot/rayon: Collecting 10 samples in estimated 7.1641 s (10 iterations)
Benchmarking prove/plot/rayon: Analyzing
prove/plot/rayon time: [706.48 ms 713.24 ms 720.35 ms]
change: [+4.1517% +12.781% +23.231%] (p = 0.01 < 0.05)
Performance has regressed.

and this is how performance is on mar-11 build’s benchmark:

Summary

.\subspace-farmer-windows-x86_64-skylake-gemini-3h-2024-mar-11.exe benchmark prove J:\Subspace
Benchmarking prove/plot/rayon/unbuffered/concurrent-chunks
Benchmarking prove/plot/rayon/unbuffered/concurrent-chunks: Warming up for 3.0000 s

Warning: Unable to complete 10 samples in 5.0s. You may wish to increase target time to 49.7s.
Benchmarking prove/plot/rayon/unbuffered/concurrent-chunks: Collecting 10 samples in estimated 49.718 s (10 iterations)
Benchmarking prove/plot/rayon/unbuffered/concurrent-chunks: Analyzing
prove/plot/rayon/unbuffered/concurrent-chunks
time: [5.3241 s 5.3361 s 5.3531 s]
change: [+3.5395% +9.5651% +15.923%] (p = 0.01 < 0.05)
Performance has regressed.
Found 1 outliers among 10 measurements (10.00%)
1 (10.00%) high severe
Benchmarking prove/plot/rayon/unbuffered/whole-sector
Benchmarking prove/plot/rayon/unbuffered/whole-sector: Warming up for 3.0000 s

Warning: Unable to complete 10 samples in 5.0s. You may wish to increase target time to 30.5s.
Benchmarking prove/plot/rayon/unbuffered/whole-sector: Collecting 10 samples in estimated 30.513 s (10 iterations)
Benchmarking prove/plot/rayon/unbuffered/whole-sector: Analyzing
prove/plot/rayon/unbuffered/whole-sector
time: [3.1588 s 3.4091 s 3.6082 s]
change: [-19.816% -12.334% -6.1830%] (p = 0.00 < 0.05)
Performance has improved.
Found 1 outliers among 10 measurements (10.00%)
1 (10.00%) low severe
Benchmarking prove/plot/rayon/regular/concurrent-chunks
Benchmarking prove/plot/rayon/regular/concurrent-chunks: Warming up for 3.0000 s

Warning: Unable to complete 10 samples in 5.0s. You may wish to increase target time to 11.8s.
Benchmarking prove/plot/rayon/regular/concurrent-chunks: Collecting 10 samples in estimated 11.832 s (10 iterations)
Benchmarking prove/plot/rayon/regular/concurrent-chunks: Analyzing
prove/plot/rayon/regular/concurrent-chunks
time: [719.08 ms 728.02 ms 737.79 ms]
change: [-38.296% -37.194% -36.074%] (p = 0.00 < 0.05)
Performance has improved.
Found 2 outliers among 10 measurements (20.00%)
2 (20.00%) high mild
Benchmarking prove/plot/rayon/regular/whole-sector
Benchmarking prove/plot/rayon/regular/whole-sector: Warming up for 3.0000 s

Warning: Unable to complete 10 samples in 5.0s. You may wish to increase target time to 19.8s.
Benchmarking prove/plot/rayon/regular/whole-sector: Collecting 10 samples in estimated 19.756 s (10 iterations)
Benchmarking prove/plot/rayon/regular/whole-sector: Analyzing
prove/plot/rayon/regular/whole-sector
time: [1.7672 s 1.8115 s 1.8626 s]
change: [-43.501% -41.432% -39.409%] (p = 0.00 < 0.05)
Performance has improved.
Found 1 outliers among 10 measurements (10.00%)
1 (10.00%) high mild

There doesn’t seem to be any obvious bottleneck happening; SSD isn’t that busy, CPU load is very moderate, plenty of free RAM. I’m running about 30-40% miss rate on mar-11 build, it’d be great if something could be done to fix that, I don’t want to sit through all 3G testnet on old build.

dragonP · March 12, 2024, 11:23pm

Even though I didn’t do what you wanted. I’ve added 3 more instead of 1 by 1.

But do you think it is necessary? Because when I added 3 at once, it didn’t work. And please remember I’m farming about 15 disks on this machine.

Sorry but I think we have enough data to confirm that it doesn’t work if we farm multiple disks with your testing build (please don’t take ‘multiple’ is only 2).

I won’t spend more time to test something that we know. The target is to farm effectively as many disks as possible, not just 4. Even though, it already doesn’t work with only 4 disks.

nazar-pc · March 13, 2024, 12:27am

I appreciate the testing, but it is frustrating when I ask to test a very specific thing and you do something else that you want on purpose again.

Yes, we know that with 4 disks performance is bad, in fact we knew that from the very beginning when we were chatting on Discord, so there is zero new information added from that latest test, just distraction with more messages that need to be read and interpreted.

What I need is to isolate an issue, which is why I asked to start adding farms and see if internal benchmarks change. Depending on what we get as the result we may find out that it is two specific disks together that cause issues or total number of disks that cause issues or something else entirely.

I think it is quite clear that if it works with one farm, it should also work with 4 farms, and it is hard to find reasonable explanation for 25x slowdown just from adding 3 other farms that are not doing anything at the time of that benchmark, they just “exist”. So we need to find something unreasonable that Windows is doing here and that requires experimentation. And I most certainly don’t know what stupid thing Windows does this time or what bug we have in the code that causes this.

I don’t see it on my Windows machine with 4 farms, so unless someone is willing to follow instructions and test what I need, not what they want to find the root cause, we may be stuck in this situation for a very long time.

I can explain every single theory and the whole though process behind every test, but it is very technical, time consuming and damaging to productivity. I just need you to help me by doing exactly the tests I requested and nothing else to minimize the communication noise.

@micar @mystery3000 we know some of it already, please read this thread carefully, download test build and provide the information I requested.

dragonP · March 13, 2024, 1:11am

You said “add farms back”. Next time, you need to provide your requirement in more detail so I can do exactly what you want.

OK, I will add 1 by 1 and share the result soon.

I really don’t get why you can farm normally with 4 farms on your windows machine, unless they are all NVMe directly mounted to the mainboard. Do NOT think my SATA SSD is bad, they are not. I can farm with very low miss reward on Feb 19 release. And I mention many times that it is Z790 chipset and 14900K. It can easily beat most build available in the market.

You know your code, we, farmers, know our hardware.

Topic		Replies	Views
How about my benchmark score Support	6	201	March 31, 2024
No rewards for 5 days, Benchmark Audit looks OK Support	7	119	March 26, 2024
Worse sector speeds & misses than before, dual Epyc 7532, Windows 10, Mar 29 release Support	17	190	April 2, 2024
Is NFS benchmark score OK for reward? Support	2	109	March 25, 2024
More rewards on smaller plots Testing rewards , gemini-3f	37	1616	September 11, 2023

More miss reward with Mar 11 release in Windows

Related Topics