In 3f, there’re a lot of factors that impacts to plot speed: CPU (high), internet (high), RAM (medium), SSD (medium to low). As we experienced, sometimes it took hours to plot 1 sector (in such scenario, there’re many many ‘recovering missing pieces’). But thing has totally been changed in 3g. I notice the only key and main factor is the CPU, internet plays almost no role at all since there is no much internet use during plotting.
So can someone from team explain what exactly plotting process does in 3g? And what is the main factors determining the plot speed so we, farmers, can decide on our hardware configuration and investment?
Hey. I can share my experience with you so far. I rented dedicated AX41-NVMe from hetzher which has amd ryzen 5 3600 (6 cores 12 threads) with an addidional ssd in total 1.5T NVMe SSDs (3 x SAMSUNG MZVL2512HCJQ-00B00).
First I setup SSDs with a RAID 0 setup ( combined them as one disk) but plotting was very slow such:
Then I decided to split disks and create three plots (300G-450G-450G). After that plotting seems faster than before but for me Its slower than I expected.
2023-11-03T17:13:43.124025Z INFO single_disk_farm{disk_farm_index=1}: subspace_farmer::single_disk_farm::plotting: Plotting sector (6.31% complete) sector_index=26
2023-11-03T17:19:46.298027Z INFO single_disk_farm{disk_farm_index=2}: subspace_farmer::single_disk_farm::plotting: Plotting sector (6.55% complete) sector_index=27
2023-11-03T17:20:59.712050Z WARN quinn_proto::connection: discarding possible duplicate packet
2023-11-03T17:20:59.712066Z WARN quinn_proto::connection: discarding possible duplicate packet
2023-11-03T17:21:01.711222Z WARN quinn_proto::connection: discarding possible duplicate packet
2023-11-03T17:21:01.711286Z WARN quinn_proto::connection: discarding possible duplicate packet
2023-11-03T17:25:56.466486Z INFO single_disk_farm{disk_farm_index=0}: subspace_farmer::single_disk_farm::plotting: Plotting sector (9.64% complete) sector_index=27
2023-11-03T17:29:59.632269Z WARN quinn_proto::connection: discarding possible duplicate packet
2023-11-03T17:29:59.632294Z WARN quinn_proto::connection: discarding possible duplicate packet
2023-11-03T17:29:59.632301Z WARN quinn_proto::connection: discarding possible duplicate packet
2023-11-03T17:29:59.632914Z WARN quinn_proto::connection: discarding possible duplicate packet
2023-11-03T17:29:59.632923Z WARN quinn_proto::connection: discarding possible duplicate packet
2023-11-03T17:29:59.632961Z WARN quinn_proto::connection: discarding possible duplicate packet
2023-11-03T17:29:59.632990Z WARN quinn_proto::connection: discarding possible duplicate packet
2023-11-03T17:29:59.633242Z WARN quinn_proto::connection: discarding possible duplicate packet
2023-11-03T17:29:59.633247Z WARN quinn_proto::connection: discarding possible duplicate packet
2023-11-03T17:29:59.925309Z WARN quinn_proto::connection: discarding possible duplicate packet
2023-11-03T17:29:59.926492Z WARN quinn_proto::connection: discarding possible duplicate packet
2023-11-03T17:29:59.926517Z WARN quinn_proto::connection: discarding possible duplicate packet
2023-11-03T17:29:59.926523Z WARN quinn_proto::connection: discarding possible duplicate packet
2023-11-03T17:29:59.926800Z WARN quinn_proto::connection: discarding possible duplicate packet
2023-11-03T17:31:00.926272Z WARN quinn_proto::connection: discarding possible duplicate packet
2023-11-03T17:31:00.926290Z WARN quinn_proto::connection: discarding possible duplicate packet
2023-11-03T17:31:00.926567Z WARN quinn_proto::connection: discarding possible duplicate packet
2023-11-03T17:31:00.926577Z WARN quinn_proto::connection: discarding possible duplicate packet
2023-11-03T17:31:00.926585Z WARN quinn_proto::connection: discarding possible duplicate packet
2023-11-03T17:31:01.215765Z WARN quinn_proto::connection: discarding possible duplicate packet
2023-11-03T17:31:01.217529Z WARN quinn_proto::connection: discarding possible duplicate packet
2023-11-03T17:31:01.217550Z WARN quinn_proto::connection: discarding possible duplicate packet
2023-11-03T17:31:01.217559Z WARN quinn_proto::connection: discarding possible duplicate packet
2023-11-03T17:31:01.217565Z WARN quinn_proto::connection: discarding possible duplicate packet
2023-11-03T17:32:04.597468Z INFO single_disk_farm{disk_farm_index=1}: subspace_farmer::single_disk_farm::plotting: Plotting sector (6.55% complete) sector_index=27
2023-11-03T17:38:08.417133Z INFO single_disk_farm{disk_farm_index=2}: subspace_farmer::single_disk_farm::plotting: Plotting sector (6.80% complete) sector_index=28
2023-11-03T17:44:18.697033Z INFO single_disk_farm{disk_farm_index=0}: subspace_farmer::single_disk_farm::plotting: Plotting sector (10.00% complete) sector_index=28
2023-11-03T17:45:58.322527Z WARN quinn_proto::connection: discarding possible duplicate packet
2023-11-03T17:45:58.322604Z WARN quinn_proto::connection: discarding possible duplicate packet
2023-11-03T17:45:58.322629Z WARN quinn_proto::connection: discarding possible duplicate packet
2023-11-03T17:45:58.322655Z WARN quinn_proto::connection: discarding possible duplicate packet
2023-11-03T17:45:58.322680Z WARN quinn_proto::connection: discarding possible duplicate packet
2023-11-03T17:45:58.612409Z WARN quinn_proto::connection: discarding possible duplicate packet
2023-11-03T17:50:26.738356Z INFO single_disk_farm{disk_farm_index=1}: subspace_farmer::single_disk_farm::plotting: Plotting sector (6.80% complete) sector_index=28
I think CPU is a good one but with this speed I estimated that plotting will be finish in 6-7 days. I will keep observe and provide more feedback.
You can plot even on HDD (I have done that when I needed to create a large plot before sending it to a remote machine for benchmarking, plotted 3TB without issues). Disk doesn’t play almost any role for plotting (farming is a different story of course), if you see the opposite - it is a coincidence.
CPU and memory (mostly its latency) are the main contributing factors. Networking will be a factor when network gets large, but for significantly sized farms it will never be an issue due to local cache hits that became much more successful due to changes introduced in 3g farmer as I described during community call.
With CPUs and memory it is not always as easy as getting more cores, some processors are slower/faster than others, generally getting the biggest consumer chip will be the most efficient, I did not see many tests of different memory configs yet though.
My 13900K running 5.5GHz on performance cores and 4.3GHz on efficiency cores creates a sector in ~2m 15s using all cores and not much slower when I limit it to use 20 or 16 cores only, others reported similar numbers on Discord IIRC.
The team should consider multi-threaded CPUs, especially the EPYC series. The EPYC series has the slowest running speed, even slower than the E5 series. In my tests, the speed of the EPYC 7V13 was not as fast as that of the E5-2699AV4. This should not be the case, as the EPYC surpasses the E5 in both clock speed and core count.
All modern CPUs are multi-threaded and implementation is able to utilize them nicely.
As to Epyc and Xeon processors, they are not just multi-threaded, they have NUMA and sometimes multiple sockets.
Feel free to contribute improvements for those systems, we’ll review them, but this is not the highest priority right now.
