Per CPU core performance gap might not be so large, but since plotting is multi-threaded the difference in how cores are interacting and how memory is shared can have big impact.
I’d recommend you to enable max NUMA configuration for your CPU and run one farmer per NUMA node to see if there is a difference there. You don’t have to wipe if you do that, individual farms can be re-connected to different farmers any time, only piece cache will be affected when you move farm between different farmers, it’ll have to be updated every time you change things up.