Piece cache sync is too slow

Support
1

Issue Report

Piece cache sync is too slow.

Environment

  • Operating System: Ubuntu 18.04
  • Pulsar/Advanced CLI/Docker: Advanced CLI

Problem

I deployed a 10 server subspace cluster with 2024/06/11 release.
The cache size is 200GiB.
The controller log shows that piece cache syncing takes 24 hours for 5% progress.
Is the speed too slow? How to speed it up?

[Paste any errors or relevant logs here]

2024-06-13T07:38:19.878779Z  INFO subspace_farmer::farmer_cache: Piece cache sync 9.46% complete
2024-06-13T08:09:53.132734Z  INFO subspace_farmer::farmer_cache: Piece cache sync 9.59% complete
2024-06-13T08:37:52.865935Z  INFO subspace_farmer::farmer_cache: Piece cache sync 9.72% complete
2024-06-13T09:04:03.116352Z  INFO subspace_farmer::farmer_cache: Piece cache sync 9.86% complete
2024-06-13T09:27:26.389633Z  INFO subspace_farmer::farmer_cache: Piece cache sync 9.99% complete
2024-06-13T09:46:48.258689Z  INFO subspace_farmer::farmer_cache: Piece cache sync 10.12% complete
2024-06-13T10:34:46.528711Z  INFO subspace_farmer::farmer_cache: Piece cache sync 10.26% complete
2024-06-13T11:14:10.923322Z  INFO subspace_farmer::farmer_cache: Piece cache sync 10.39% complete
2024-06-13T11:57:30.635926Z  INFO subspace_farmer::farmer_cache: Piece cache sync 10.52% complete
2024-06-13T12:32:08.474655Z  INFO subspace_farmer::farmer_cache: Piece cache sync 10.66% complete
2024-06-13T13:02:27.059904Z  INFO subspace_farmer::farmer_cache: Piece cache sync 10.79% complete
2024-06-13T13:04:03.564823Z  WARN subspace_farmer::commands::cluster::controller::farms: Farm expired and removed farm_index=2 farm_id=01J064A903CJZD18FRXTQYV7G5
2024-06-13T13:04:03.564850Z  WARN subspace_farmer::commands::cluster::controller::farms: Farm expired and removed farm_index=5 farm_id=01J064A906CPNBJXTP9C7Y51MY
2024-06-13T13:04:03.564854Z  WARN subspace_farmer::commands::cluster::controller::farms: Farm expired and removed farm_index=0 farm_id=
2024-06-13T08:37:52.865935Z  INFO subspace_farmer::farmer_cache: Piece cache sync 9.72% complete
2024-06-13T09:04:03.116352Z  INFO subspace_farmer::farmer_cache: Piece cache sync 9.86% complete
2024-06-13T09:27:26.389633Z  INFO subspace_farmer::farmer_cache: Piece cache sync 9.99% complete
2024-06-13T09:46:48.258689Z  INFO subspace_farmer::farmer_cache: Piece cache sync 10.12% complete
2024-06-13T10:34:46.528711Z  INFO subspace_farmer::farmer_cache: Piece cache sync 10.26% complete
2024-06-13T11:14:10.923322Z  INFO subspace_farmer::farmer_cache: Piece cache sync 10.39% complete
2024-06-13T11:57:30.635926Z  INFO subspace_farmer::farmer_cache: Piece cache sync 10.52% complete
2024-06-13T12:32:08.474655Z  INFO subspace_farmer::farmer_cache: Piece cache sync 10.66% complete
2024-06-13T13:02:27.059904Z  INFO subspace_farmer::farmer_cache: Piece cache sync 10.79% complete
2024-06-13T13:04:03.564823Z  WARN subspace_farmer::commands::cluster::controller::farms: Farm expired and removed farm_index=2 farm_id=01J064A903CJZD18FRXTQYV7G5
2024-06-13T13:04:03.564850Z  WARN subspace_farmer::commands::cluster::controller::farms: Farm expired and removed farm_index=5 farm_id=01J064A906CPNBJXTP9C7Y51MY
2024-06-13T13:04:03.564854Z  WARN subspace_farmer::commands::cluster::controller::farms: Farm expired and removed farm_index=0 farm_id=01J064A906XK3GT42E4FA676XF
2024-06-13T13:04:03.564857Z  WARN subspace_farmer::commands::cluster::controller::farms: Farm expired and removed farm_index=3 farm_id=01J064A903WYHBQF5V70XK9TV3
2024-06-13T13:04:03.564860Z  WARN subspace_farmer::commands::cluster::controller::farms: Farm expired and removed farm_index=4 farm_id=01J064A906MATW2K6J975NHP5S
2024-06-13T13:04:03.564864Z  WARN subspace_farmer::commands::cluster::controller::farms: Farm expired and removed farm_index=1 farm_id=01J064A906WE1RTTQQ9F2N1W4V
2024-06-13T13:04:03.564898Z  INFO subspace_farmer::commands::cluster::controller::farms: Farm exited successfully farm_index=2
2024-06-13T13:04:03.564933Z  INFO subspace_farmer::commands::cluster::controller::farms: Farm exited successfully farm_index=5
2024-06-13T13:04:03.564961Z  INFO subspace_farmer::commands::cluster::controller::farms: Farm exited successfully farm_index=0
2024-06-13T13:04:03.564988Z  INFO subspace_farmer::commands::cluster::controller::farms: Farm exited successfully farm_index=3
2024-06-13T13:04:03.565007Z  INFO subspace_farmer::commands::cluster::controller::farms: Farm exited successfully farm_index=4
2024-06-13T13:04:03.565032Z  INFO subspace_farmer::commands::cluster::controller::farms: Farm exited successfully farm_index=1
2024-06-13T13:39:58.571016Z  INFO subspace_farmer::farmer_cache: Piece cache sync 10.92% complete
2024-06-13T14:36:58.053910Z  INFO subspace_farmer::farmer_cache: Piece cache sync 11.06% complete
2024-06-13T15:12:43.938461Z  INFO subspace_farmer::farmer_cache: Piece cache sync 11.19% complete
2024-06-13T15:52:53.459220Z  INFO subspace_farmer::farmer_cache: Piece cache sync 11.32% complete
2024-06-13T16:30:00.001075Z  INFO subspace_farmer::farmer_cache: Piece cache sync 11.46% complete
2024-06-13T16:58:46.347461Z  INFO subspace_farmer::farmer_cache: Piece cache sync 11.59% complete
2024-06-13T17:59:40.801031Z  INFO subspace_farmer::farmer_cache: Piece cache sync 11.72% complete
2024-06-13T19:01:22.987925Z  INFO subspace_farmer::farmer_cache: Piece cache sync 11.86% complete
2024-06-13T19:37:19.062833Z  INFO subspace_farmer::farmer_cache: Piece cache sync 11.99% complete
2024-06-13T20:15:46.777411Z  INFO subspace_farmer::farmer_cache: Piece cache sync 12.12% complete
2024-06-13T20:49:32.379214Z  INFO subspace_farmer::farmer_cache: Piece cache sync 12.25% complete
2024-06-13T21:34:37.984973Z  INFO subspace_farmer::farmer_cache: Piece cache sync 12.39% complete
2024-06-13T22:27:20.909348Z  INFO subspace_farmer::farmer_cache: Piece cache sync 12.52% complete
2024-06-13T23:06:58.345258Z  INFO subspace_farmer::farmer_cache: Piece cache sync 12.65% complete
2024-06-13T23:40:38.829110Z  INFO subspace_farmer::farmer_cache: Piece cache sync 12.79% complete
2024-06-14T00:17:30.706025Z  INFO subspace_farmer::farmer_cache: Piece cache sync 12.92% complete
2024-06-14T00:38:57.218294Z  INFO subspace_farmer::farmer_cache: Piece cache sync 13.05% complete
2024-06-14T01:31:31.054347Z  INFO subspace_farmer::farmer_cache: Piece cache sync 13.19% complete
2024-06-14T02:23:57.190653Z  INFO subspace_farmer::farmer_cache: Piece cache sync 13.32% complete
2024-06-14T03:00:51.853912Z  INFO subspace_farmer::farmer_cache: Piece cache sync 13.45% complete
2024-06-14T03:42:08.672372Z  INFO subspace_farmer::farmer_cache: Piece cache sync 13.59% complete
2024-06-14T04:19:34.032250Z  INFO subspace_farmer::farmer_cache: Piece cache sync 13.72% complete
2024-06-14T05:01:22.233127Z  INFO subspace_farmer::farmer_cache: Piece cache sync 13.85% complete
2024-06-14T05:53:27.902126Z  INFO subspace_farmer::farmer_cache: Piece cache sync 13.99% complete
2024-06-14T06:30:02.251212Z  INFO subspace_farmer::farmer_cache: Piece cache sync 14.12% complete
2024-06-14T07:08:17.734351Z  INFO subspace_farmer::farmer_cache: Piece cache sync 14.25% complete
2024-06-14T07:40:12.295223Z  INFO subspace_farmer::farmer_cache: Piece cache sync 14.39% complete
2024-06-14T08:03:34.459146Z  INFO subspace_farmer::farmer_cache: Piece cache sync 14.52% complete
2024-06-14T08:36:11.819615Z  INFO subspace_farmer::farmer_cache: Piece cache sync 14.65% complete
2024-06-14T09:16:52.072922Z  INFO subspace_farmer::farmer_cache: Piece cache sync 14.79% complete
2024-06-14T09:49:15.915560Z  INFO subspace_farmer::farmer_cache: Piece cache sync 14.92% complete
2024-06-14T10:21:37.143789Z  INFO subspace_farmer::farmer_cache: Piece cache sync 15.05% complete
2

Was it significantly faster in previous releases?

To speed things up you can try to increase the number of outgoing connections using CLI options (try to double it from default), but overall it is heavily dependant on your Internet connection to the rest of the network.

3

If I restart the cluster processes for CLI option change, will it do piece sync from 0% again? or resume from the interrupt point?

4

With cluster or not it will resume from where it ended, even though remaining progress percentage will be re-scaled and will start from 0% again.