# Slow Plotting with GPU **URL:** \ **Category:** Support\ **Created:** [October 9, 2024, 2:02pm UTC](https://forum.autonomys.xyz/t/slow-plotting-with-gpu/4483 "2024-10-09T14:02:26Z")\ **Posts on this page:** 14\ **Page:** 1
**Author:** ![repost](https://avatars.discourse-cdn.com/v4/letter/r/b487fb/32.png) [@repost](https://forum.autonomys.xyz/u/repost)\ **Post date:** [October 9, 2024, 2:02pm UTC](https://forum.autonomys.xyz/t/slow-plotting-with-gpu/4483/1 "2024-10-09T14:02:26Z")
## Issue Report ### Environment **SERVER 1** **Components** : Node (on 1TB SSD), Plotter, Cache (110GiB on 1TB SSD), Controller, NATs (all Docker) **Hardware** : 10G NIC connected to 10G switch. RTX 3070. i7 12700KF **OS** : Ubuntu 24.04 server **GPU Drivers:** 560 **SERVER 2** **Components** : Plotter, Controller, Cache (group1) (all Docker) **Hardware** : 10G NIC connected to 10G switch. 2 x RTX 4090. Threadripper 3975WX **OS** : Ubuntu 24.04 server **GPU Drivers:** 560 **SERVER 3** **Components:** Farmer (Docker) **Hardware:** 2.5G NIC connected to 10G switch @ 2.5G. ARM CPU. 4x4TB NVMEs **OS** : Debian GNU/Linux 12 (bookworm) ### Problem I am seeing very slow plotting with the RTX 3070. I am unsure if this is related to the network or my setup. Until recently I had seen very fast plot times with GPU. There are no errors on the Plotter or Farmer. The network is not saturated, I do see spikes periodically when there is a plot. Resources on both servers remain low (less than 20% CPU and RAM utilization). I am happy to attach any relevant logs or screen shots. Here is a snippet of my plots to see times: ```auto 2024-10-09T14:00:42.862646Z INFO {farm_index=3}:{sector_index=10}: subspace_farmer::single_disk_farm::plotting: Plotting sector (0.28% complete) 2024-10-09T14:00:13.124579Z INFO {farm_index=1}:{sector_index=9}: subspace_farmer::single_disk_farm::plotting: Plotting sector (0.25% complete) 2024-10-09T14:00:12.171474Z INFO {farm_index=1}:{sector_index=8}: subspace_farmer::single_disk_farm::plotting: Plotting sector (0.22% complete) 2024-10-09T14:00:11.912956Z INFO {farm_index=1}:{sector_index=7}: subspace_farmer::single_disk_farm::plotting: Plotting sector (0.19% complete) 2024-10-09T14:00:11.683526Z INFO {farm_index=1}:{sector_index=6}: subspace_farmer::single_disk_farm::plotting: Plotting sector (0.17% complete) 2024-10-09T14:00:11.396150Z INFO {farm_index=1}:{sector_index=5}: subspace_farmer::single_disk_farm::plotting: Plotting sector (0.14% complete) 2024-10-09T13:57:24.187192Z INFO {farm_index=0}:{sector_index=11}: subspace_farmer::single_disk_farm::plotting: Plotting sector (0.31% complete) 2024-10-09T13:57:06.113683Z INFO {farm_index=0}:{sector_index=10}: subspace_farmer::single_disk_farm::plotting: Plotting sector (0.28% complete) 2024-10-09T13:54:50.512231Z INFO {farm_index=0}:{sector_index=9}: subspace_farmer::single_disk_farm::plotting: Plotting sector (0.25% complete) 2024-10-09T13:54:48.515731Z INFO {farm_index=0}:{sector_index=8}: subspace_farmer::single_disk_farm::plotting: Plotting sector (0.22% complete) 2024-10-09T13:54:41.028735Z INFO {farm_index=0}:{sector_index=7}: subspace_farmer::single_disk_farm::plotting: Plotting sector (0.19% complete) 2024-10-09T13:54:40.760937Z INFO {farm_index=0}:{sector_index=6}: subspace_farmer::single_disk_farm::plotting: Plotting sector (0.17% complete) 2024-10-09T13:54:40.758386Z INFO {farm_index=0}:{sector_index=5}: subspace_farmer::single_disk_farm::plotting: Plotting sector (0.14% complete) 2024-10-09T13:52:22.453069Z INFO {farm_index=0}:{sector_index=4}: subspace_farmer::single_disk_farm::plotting: Plotting sector (0.11% complete) 2024-10-09T13:52:07.213289Z INFO {farm_index=0}:{sector_index=3}: subspace_farmer::single_disk_farm::plotting: Plotting sector (0.08% complete) ``` ---
**Author:** ![duanyz\_aiyo](https://yyz1.discourse-cdn.com/flex011/user_avatar/forum.autonomys.xyz/duanyz_aiyo/32/2587_2.png) [@duanyz\_aiyo](https://forum.autonomys.xyz/u/duanyz_aiyo)\ **Post date:** [October 10, 2024, 2:24am UTC](https://forum.autonomys.xyz/t/slow-plotting-with-gpu/4483/2 "2024-10-10T02:24:16Z")
I have the same issue; the GPU utilization in the cluster version is unstable and the utilization is low. ---
**Author:** ![nazar-pc](https://yyz1.discourse-cdn.com/flex011/user_avatar/forum.autonomys.xyz/nazar-pc/32/6_2.png) [@nazar-pc](https://forum.autonomys.xyz/u/nazar-pc)\ **Post date:** [October 10, 2024, 5:58pm UTC](https://forum.autonomys.xyz/t/slow-plotting-with-gpu/4483/3 "2024-10-10T17:58:54Z")
The first step would be to pull metrics on the plotter and see what it is spending the most time doing ---
**Author:** ![repost](https://avatars.discourse-cdn.com/v4/letter/r/b487fb/32.png) [@repost](https://forum.autonomys.xyz/u/repost)\ **Post date:** [October 10, 2024, 6:39pm UTC](https://forum.autonomys.xyz/t/slow-plotting-with-gpu/4483/4 "2024-10-10T18:39:05Z")
I will pull Prometheus metrics soon, do not have time right now. But do have node\_exporter up. But assuming you want the other metrics. Plotter At a 1 hour resolution ![image](https://canada1.discourse-cdn.com/flex011/uploads/subspace/original/2X/e/e44124da798d9d78fa24253a2557eb4d0cd80177.png) At a 12 hour resolution ![image](https://canada1.discourse-cdn.com/flex011/uploads/subspace/original/2X/e/e027e607efd44df6c9e179186bcc895ac99b6848.png) The Farmer: At a 1 hour resolution ![image](https://canada1.discourse-cdn.com/flex011/uploads/subspace/original/2X/6/6c0d13eeeba389fd39a8943d5ffcb5e1b3fef35b.png) At a 12 hour resolution ![image](https://canada1.discourse-cdn.com/flex011/uploads/subspace/original/2X/1/14589acc6f6eba76a47d95e295bf54636a8ac357.png) Farmer Disk metrics ![image](https://canada1.discourse-cdn.com/flex011/uploads/subspace/original/2X/f/f9dd7231c2da804b619a072497a96ed329b9ee8f.jpeg) ---
**Author:** ![nazar-pc](https://yyz1.discourse-cdn.com/flex011/user_avatar/forum.autonomys.xyz/nazar-pc/32/6_2.png) [@nazar-pc](https://forum.autonomys.xyz/u/nazar-pc)\ **Post date:** [October 10, 2024, 7:52pm UTC](https://forum.autonomys.xyz/t/slow-plotting-with-gpu/4483/5 "2024-10-10T19:52:36Z")
Application metrics contain breakdown what plotter is spending time on. For example it might be slow to download pieces or slow to encode or slow to send the sector back. Similarly you can pull plotting metrics from farmer’s point of view, which is a different perspective and can also provide additional insights. ---
**Author:** ![repost](https://avatars.discourse-cdn.com/v4/letter/r/b487fb/32.png) [@repost](https://forum.autonomys.xyz/u/repost)\ **Post date:** [October 10, 2024, 11:46pm UTC](https://forum.autonomys.xyz/t/slow-plotting-with-gpu/4483/6 "2024-10-10T23:46:18Z")
![image](https://canada1.discourse-cdn.com/flex011/uploads/subspace/original/2X/9/9913bff7a7dcfe5a9f3fd49da6b4af4ead40d6ae.png) The long download time appears to be the issue, can you advise what might cause this? ---
**Author:** ![nazar-pc](https://yyz1.discourse-cdn.com/flex011/user_avatar/forum.autonomys.xyz/nazar-pc/32/6_2.png) [@nazar-pc](https://forum.autonomys.xyz/u/nazar-pc)\ **Post date:** [October 11, 2024, 1:23am UTC](https://forum.autonomys.xyz/t/slow-plotting-with-gpu/4483/7 "2024-10-11T01:23:59Z")
This is what some other folks were observing as well, for example in [Comparison of Single vs. Multiple GPU Instances](https://forum.autonomys.xyz/t/comparison-of-single-vs-multiple-gpu-instances/4424) The workaround there was to run multiple instances to absorb these delays. I have not yet invested much time into figuring out what is going on there, but one day I will. ---
**Author:** ![repost](https://avatars.discourse-cdn.com/v4/letter/r/b487fb/32.png) [@repost](https://forum.autonomys.xyz/u/repost)\ **Post date:** [October 11, 2024, 2:05am UTC](https://forum.autonomys.xyz/t/slow-plotting-with-gpu/4483/8 "2024-10-11T02:05:12Z")
Am I understanding correctly that the Cache located on the Plotter is supposed to prevent these sectors from needing to be downloaded? It seems like the Cache is not being used at all if this is the case. Is it worth trying without a cluster to see if that is the problem? If I can only plot at 2.5 minute sectors and some are getting 3 second sectors that would be greatly disappointing. ---
**Author:** ![nazar-pc](https://yyz1.discourse-cdn.com/flex011/user_avatar/forum.autonomys.xyz/nazar-pc/32/6_2.png) [@nazar-pc](https://forum.autonomys.xyz/u/nazar-pc)\ **Post date:** [October 11, 2024, 2:19am UTC](https://forum.autonomys.xyz/t/slow-plotting-with-gpu/4483/9 "2024-10-11T02:19:19Z")
That would be a reasonable assumption, you can check both controller and cache metrics to check what is going on with cache. Controller should print logs when it sees and looses cache instance. ---
**Author:** ![repost](https://avatars.discourse-cdn.com/v4/letter/r/b487fb/32.png) [@repost](https://forum.autonomys.xyz/u/repost)\ **Post date:** [October 11, 2024, 3:43am UTC](https://forum.autonomys.xyz/t/slow-plotting-with-gpu/4483/10 "2024-10-11T03:43:25Z")
The existing Grafana Dashboard does not work for the cluster. So I rebuilding. These are the plotting and cache info for SERVER 1 ![image](https://canada1.discourse-cdn.com/flex011/uploads/subspace/original/2X/4/4c0ef613eb6b910bbef2a98ce365a97abb9843fd.png) There are no logs for the controller or cache that would indicate any misses or issues. ---
**Author:** ![repost](https://avatars.discourse-cdn.com/v4/letter/r/b487fb/32.png) [@repost](https://forum.autonomys.xyz/u/repost)\ **Post date:** [October 11, 2024, 4:09am UTC](https://forum.autonomys.xyz/t/slow-plotting-with-gpu/4483/11 "2024-10-11T04:09:40Z")
![image](https://canada1.discourse-cdn.com/flex011/uploads/subspace/original/2X/8/8450a12337444f3da0822112981f6bda9e53d1c7.png) ---
**Author:** ![nazar-pc](https://yyz1.discourse-cdn.com/flex011/user_avatar/forum.autonomys.xyz/nazar-pc/32/6_2.png) [@nazar-pc](https://forum.autonomys.xyz/u/nazar-pc)\ **Post date:** [October 13, 2024, 8:52am UTC](https://forum.autonomys.xyz/t/slow-plotting-with-gpu/4483/13 "2024-10-13T08:52:00Z")
I meant simply text output of the `/metrics` endpoint, not screenshots ---
**Author:** ![nazar-pc](https://yyz1.discourse-cdn.com/flex011/user_avatar/forum.autonomys.xyz/nazar-pc/32/6_2.png) [@nazar-pc](https://forum.autonomys.xyz/u/nazar-pc)\ **Post date:** [October 24, 2024, 9:23pm UTC](https://forum.autonomys.xyz/t/slow-plotting-with-gpu/4483/14 "2024-10-24T21:23:07Z")
I think this should be addressed in [Release taurus-2024-oct-24 · autonomys/subspace · GitHub](https://github.com/autonomys/subspace/releases/tag/taurus-2024-oct-24), would be nice to get some feedback ---
**Author:** ![Casey\_Bjorn](https://yyz1.discourse-cdn.com/flex011/user_avatar/forum.autonomys.xyz/casey_bjorn/32/3239_2.png) [@Casey\_Bjorn](https://forum.autonomys.xyz/u/Casey_Bjorn)\ **Post date:** [January 13, 2025, 2:48pm UTC](https://forum.autonomys.xyz/t/slow-plotting-with-gpu/4483/15 "2025-01-13T14:48:28Z")
Beautiful dashboards @repost ! Will you please share it? I would be VERY much obliged 🙌 🙏 🤘