Audit benchmark result - explanation please

Support
1

I have the following audit benchmark result in one of my PC, please if someone from team explain us how to interpret the result.

I have 3 questions to start with please.

  1. What is difference between single and rayon? Why the benchmark run both?
  2. What is the threshold to conclude the audit is too slow?
  3. Can we calculate the max plot size based on the result below? How?
> PS H:\> .\subspace-farmer3g_9Nov benchmark audit D:\1
> Benchmarking audit/plot/single
> Benchmarking audit/plot/single: Warming up for 3.0000 s
> Benchmarking audit/plot/single: Collecting 10 samples in estimated 6.3563 s (20 iterations)
> Benchmarking audit/plot/single: Analyzing
> audit/plot/single       time:   [309.49 ms 313.77 ms 318.15 ms]
>                         thrpt:  [5794.8 GiB/s 5875.6 GiB/s 5957.0 GiB/s]
>                  change:
>                         time:   [-0.2475% +2.4795% +5.9062%] (p = 0.45 > 0.05)
>                         thrpt:  [-5.5768% -2.4195% +0.2481%]
>                         No change in performance detected.
> Benchmarking audit/plot/rayon
> Benchmarking audit/plot/rayon: Warming up for 3.0000 s
> Benchmarking audit/plot/rayon: Collecting 10 samples in estimated 5.4454 s (165 iterations)
> Benchmarking audit/plot/rayon: Analyzing
> audit/plot/rayon        time:   [32.175 ms 32.622 ms 33.215 ms]
>                         thrpt:  [ 55506 GiB/s  56514 GiB/s  57300 GiB/s]
>                  change:
>                         time:   [-9.9736% -7.9958% -6.2526%] (p = 0.00 < 0.05)
>                         thrpt:  [+6.6696% +8.6907% +11.079%]
>                         Performance has improved.
2

Two implementation of auditing, there were more. We were using this to figure out how each behaves on different operating systems and drives to make sure we are using the fastest implementation by default on all platforms.

  • single opens the file once and uses that for reading from multiple threads.
  • rayon opens files as many times as there are farming threads and for each thread uses that dedicates file handle. rayon is the library we use for multi-threading stuff.

On Linux both perform similarly with small edge to rayon, Windows is problematic and rayon is MUCH faster there (your machine is likely much faster than old-ish machine I used for benches there, so you see even bigger performance gap). This allowed us to get rid of memory-mapped I/O and get even better performance on Windows and get some speed-up on other OSs too.

rayon is currently the default, there might be other options added in the future.

You should be able to audit well within 1s window because farmer receives a new challenge every second (1s is our slot time).

Generally you look at throughput and that will tell you how big should that SSD be for you to not be able to audit it quickly enough. Both CPU and disk performance impacts that, so if you add a second disk of the same size, the time will not necessarily increase 2x, it will likely have much less impact. It still gives you an idea about performance.

If you have high-end NVMe 4.0 or 5.0 SSD (or RAM disk) you can use this as an auditing CPU benchmark since SSD will no longer be your bottleneck.

For example this benchmark uncovered some CPU-related performance bottlenecks in the protocol and we have already improved auditing on the protocol level in Gemini 3g, 3g is more energy efficient than 3f in that sense and further research is ongoing, so there might be more improvements.