# Audit benchmark result - explanation please

**URL:** <https://forum.autonomys.xyz/t/audit-benchmark-result-explanation-please/2149>\
**Category:** Support\
**Created:** [November 11, 2023, 12:11pm UTC](https://forum.autonomys.xyz/t/audit-benchmark-result-explanation-please/2149 "2023-11-11T12:11:09Z")\
**Posts on this page:** 2\
**Page:** 1

<div class="post-metadata">

**Author:** ![dragonP](https://yyz1.discourse-cdn.com/flex011/user_avatar/forum.autonomys.xyz/dragonp/32/716_2.png) [@dragonP](https://forum.autonomys.xyz/u/dragonP)\
**Post date:** [November 11, 2023, 12:11pm UTC](https://forum.autonomys.xyz/t/audit-benchmark-result-explanation-please/2149/1 "2023-11-11T12:11:09Z")

</div>

I have the following audit benchmark result in one of my PC, please if someone from team explain us how to interpret the result.

I have 3 questions to start with please.

1. What is difference between single and rayon? Why the benchmark run both?
2. What is the threshold to conclude the audit is too slow?
3. Can we calculate the max plot size based on the result below? How?

```auto
> PS H:\> .\subspace-farmer3g_9Nov benchmark audit D:\1
> Benchmarking audit/plot/single
> Benchmarking audit/plot/single: Warming up for 3.0000 s
> Benchmarking audit/plot/single: Collecting 10 samples in estimated 6.3563 s (20 iterations)
> Benchmarking audit/plot/single: Analyzing
> audit/plot/single time: [309.49 ms 313.77 ms 318.15 ms]
> thrpt: [5794.8 GiB/s 5875.6 GiB/s 5957.0 GiB/s]
> change:
> time: [-0.2475% +2.4795% +5.9062%] (p = 0.45 > 0.05)
> thrpt: [-5.5768% -2.4195% +0.2481%]
> No change in performance detected.
> Benchmarking audit/plot/rayon
> Benchmarking audit/plot/rayon: Warming up for 3.0000 s
> Benchmarking audit/plot/rayon: Collecting 10 samples in estimated 5.4454 s (165 iterations)
> Benchmarking audit/plot/rayon: Analyzing
> audit/plot/rayon time: [32.175 ms 32.622 ms 33.215 ms]
> thrpt: [55506 GiB/s 56514 GiB/s 57300 GiB/s]
> change:
> time: [-9.9736% -7.9958% -6.2526%] (p = 0.00 < 0.05)
> thrpt: [+6.6696% +8.6907% +11.079%]
> Performance has improved.

```

---

<div class="post-metadata">

**Author:** ![nazar-pc](https://yyz1.discourse-cdn.com/flex011/user_avatar/forum.autonomys.xyz/nazar-pc/32/6_2.png) [@nazar-pc](https://forum.autonomys.xyz/u/nazar-pc)\
**Post date:** [November 11, 2023, 10:16pm UTC](https://forum.autonomys.xyz/t/audit-benchmark-result-explanation-please/2149/2 "2023-11-11T22:16:14Z")

</div>

> [@dragonP](#):
>
> What is difference between single and rayon? Why the benchmark run both?

Two implementation of auditing, there were more. We were using this to figure out how each behaves on different operating systems and drives to make sure we are using the fastest implementation by default on all platforms.

- `single` opens the file once and uses that for reading from multiple threads.
- `rayon` opens files as many times as there are farming threads and for each thread uses that dedicates file handle. [`rayon`](https://github.com/rayon-rs/rayon) is the library we use for multi-threading stuff.

On Linux both perform similarly with small edge to `rayon`, Windows is problematic and `rayon` is [MUCH faster there](https://github.com/subspace/subspace/pull/2151) (your machine is likely much faster than old-ish machine I used for benches there, so you see even bigger performance gap). This allowed us to get rid of memory-mapped I/O and get even better performance on Windows and get some speed-up on other OSs too.

`rayon` is currently the default, there might be other options added in the future.

> [@dragonP](#):
>
> What is the threshold to conclude the audit is too slow?

You should be able to audit well within 1s window because farmer receives a new challenge every second (1s is our slot time).

> [@dragonP](#):
>
> Can we calculate the max plot size based on the result below? How?

Generally you look at throughput and that will tell you how big should that SSD be for you to not be able to audit it quickly enough. Both CPU and disk performance impacts that, so if you add a second disk of the same size, the time will not necessarily increase 2x, it will likely have much less impact. It still gives you an idea about performance.

If you have high-end NVMe 4.0 or 5.0 SSD (or RAM disk) you can use this as an auditing CPU benchmark since SSD will no longer be your bottleneck.

For example this benchmark uncovered some [CPU-related performance bottlenecks](https://forum.autonomys.xyz/t/extreme-audit-algorithm-inefficiency-cpu-bound/2040) in the protocol and we have already improved auditing on the protocol level in Gemini 3g, 3g is more energy efficient than 3f in that sense and further research is ongoing, so there might be more improvements.