# Node out of sync

**URL:** <https://forum.autonomys.xyz/t/node-out-of-sync/4164>\
**Category:** Support\
**Created:** [June 9, 2024, 3:02pm UTC](https://forum.autonomys.xyz/t/node-out-of-sync/4164 "2024-06-09T15:02:26Z")\
**Posts on this page:** 15\
**Page:** 1

<div class="post-metadata">

**Author:** ![duanyz\_aiyo](https://yyz1.discourse-cdn.com/flex011/user_avatar/forum.autonomys.xyz/duanyz_aiyo/32/2587_2.png) [@duanyz\_aiyo](https://forum.autonomys.xyz/u/duanyz_aiyo)\
**Post date:** [June 9, 2024, 3:02pm UTC](https://forum.autonomys.xyz/t/node-out-of-sync/4164/1 "2024-06-09T15:02:26Z")

</div>

## Issue Report

### Environment

- **Operating System: Ubuntu 22.04.3 LTS**
- **Advanced CLI**

### Problem

When the following error appears in the node logs, it will no longer stay in sync

```auto
`2024-06-09T14:54:11.405124Z ERROR sc_utils::mpsc: The number of unprocessed messages in channel `subspace_acknowledgement` exceeded 100.
The channel was created at:
   0: sc_utils::mpsc::tracing_unbounded
   1: sc_consensus_subspace::archiver::send_archived_segment_notification::{{closure}}.124193
   2: subspace_service::new_full::{{closure}}::{{closure}}.124192
   3: <futures_util::future::future::Map<Fut,F> as core::future::future::Future>::poll
   4: <tracing_futures::Instrumented<T> as core::future::future::Future>::poll
   5: tokio::runtime::task::raw::poll
   6: std::sys_common::backtrace::__rust_begin_short_backtrace
   7: core::ops::function::FnOnce::call_once{{vtable.shim}}
   8: std::sys::pal::unix::thread::Thread::new::thread_start
   9: <unknown>
  10: <unknown>

                                         Last message was sent from:
   0: <sc_consensus_subspace_rpc::SubspaceRpc<Block,Client,SO,AS> as sc_consensus_subspace_rpc::SubspaceRpcApiServer>::acknowledge_archived_segment_header::{{closure}}
   1: jsonrpsee_core::server::rpc_module::RpcModule<Context>::register_async_method::{{closure}}::{{closure}}
   2: tokio::runtime::task::core::Core<T,S>::poll
   3: tokio::runtime::task::raw::poll
   4: tokio::runtime::scheduler::multi_thread::worker::Context::run_task
   5: tokio::runtime::scheduler::multi_thread::worker::run
   6: tokio::runtime::task::raw::poll
   7: std::sys_common::backtrace::__rust_begin_short_backtrace
   8: core::ops::function::FnOnce::call_once{{vtable.shim}}
   9: std::sys::pal::unix::thread::Thread::new::thread_start
  10: <unknown>
  11: <unknown>`

```

---

<div class="post-metadata">

**Author:** ![nazar-pc](https://yyz1.discourse-cdn.com/flex011/user_avatar/forum.autonomys.xyz/nazar-pc/32/6_2.png) [@nazar-pc](https://forum.autonomys.xyz/u/nazar-pc)\
**Post date:** [June 10, 2024, 12:02am UTC](https://forum.autonomys.xyz/t/node-out-of-sync/4164/2 "2024-06-10T00:02:23Z")

</div>

How many farms do you have connected to the node? Any other details you can provide about the setup?

---

<div class="post-metadata">

**Author:** ![duanyz\_aiyo](https://yyz1.discourse-cdn.com/flex011/user_avatar/forum.autonomys.xyz/duanyz_aiyo/32/2587_2.png) [@duanyz\_aiyo](https://forum.autonomys.xyz/u/duanyz_aiyo)\
**Post date:** [June 10, 2024, 12:39pm UTC](https://forum.autonomys.xyz/t/node-out-of-sync/4164/3 "2024-06-10T12:39:22Z")

</div>

There are 80 farmers connected to this node. Is it because there are too many farmers connected? How many farmers should a single node handle?

---

<div class="post-metadata">

**Author:** ![nazar-pc](https://yyz1.discourse-cdn.com/flex011/user_avatar/forum.autonomys.xyz/nazar-pc/32/6_2.png) [@nazar-pc](https://forum.autonomys.xyz/u/nazar-pc)\
**Post date:** [June 10, 2024, 1:23pm UTC](https://forum.autonomys.xyz/t/node-out-of-sync/4164/4 "2024-06-10T13:23:19Z")

</div>

Hypothetically there are no hard limits and above message should be non-blocking, I will take a closer look at what might be the root cause of it. You may also want to consider newly added farming custer as a more scalable option for such use case, especially with upcoming release that contains a bunch of improvements.

---

<div class="post-metadata">

**Author:** ![nazar-pc](https://yyz1.discourse-cdn.com/flex011/user_avatar/forum.autonomys.xyz/nazar-pc/32/6_2.png) [@nazar-pc](https://forum.autonomys.xyz/u/nazar-pc)\
**Post date:** [June 10, 2024, 1:24pm UTC](https://forum.autonomys.xyz/t/node-out-of-sync/4164/5 "2024-06-10T13:24:03Z")

</div>

BTW you mentioned 80 farmers, but how many farms is it?

---

<div class="post-metadata">

**Author:** ![duanyz\_aiyo](https://yyz1.discourse-cdn.com/flex011/user_avatar/forum.autonomys.xyz/duanyz_aiyo/32/2587_2.png) [@duanyz\_aiyo](https://forum.autonomys.xyz/u/duanyz_aiyo)\
**Post date:** [June 10, 2024, 1:57pm UTC](https://forum.autonomys.xyz/t/node-out-of-sync/4164/6 "2024-06-10T13:57:36Z")

</div>

Sorry, I meant there are 80 devices connected to this node, each with 8 farmer directories, making it a total of 640 farmers

---

<div class="post-metadata">

**Author:** ![nazar-pc](https://yyz1.discourse-cdn.com/flex011/user_avatar/forum.autonomys.xyz/nazar-pc/32/6_2.png) [@nazar-pc](https://forum.autonomys.xyz/u/nazar-pc)\
**Post date:** [June 10, 2024, 2:17pm UTC](https://forum.autonomys.xyz/t/node-out-of-sync/4164/7 "2024-06-10T14:17:21Z")

</div>

I see, that is quite a large setup. Can you provide all node logs since it started rather than just a single error, please? There might be additional information in there as well.

---

<div class="post-metadata">

**Author:** ![duanyz\_aiyo](https://yyz1.discourse-cdn.com/flex011/user_avatar/forum.autonomys.xyz/duanyz_aiyo/32/2587_2.png) [@duanyz\_aiyo](https://forum.autonomys.xyz/u/duanyz_aiyo)\
**Post date:** [June 10, 2024, 2:34pm UTC](https://forum.autonomys.xyz/t/node-out-of-sync/4164/8 "2024-06-10T14:34:14Z")

</div>

Currently, the node logs have been cleared, so I can’t provide them at the moment. This error has been occurring daily recently. When it happens again, I will provide the complete logs.

---

<div class="post-metadata">

**Author:** ![nazar-pc](https://yyz1.discourse-cdn.com/flex011/user_avatar/forum.autonomys.xyz/nazar-pc/32/6_2.png) [@nazar-pc](https://forum.autonomys.xyz/u/nazar-pc)\
**Post date:** [June 10, 2024, 3:36pm UTC](https://forum.autonomys.xyz/t/node-out-of-sync/4164/9 "2024-06-10T15:36:24Z")

</div>

I think this might happen if one of the farmers crashed and disconnected after new segment of archival history was created. Upcoming release will increase performance and may help with this, while farming cluster is even less likely to end up in this situation due to its architecture.

---

<div class="post-metadata">

**Author:** ![nazar-pc](https://yyz1.discourse-cdn.com/flex011/user_avatar/forum.autonomys.xyz/nazar-pc/32/6_2.png) [@nazar-pc](https://forum.autonomys.xyz/u/nazar-pc)\
**Post date:** [June 10, 2024, 3:55pm UTC](https://forum.autonomys.xyz/t/node-out-of-sync/4164/10 "2024-06-10T15:55:03Z")

</div>

[Add segment acknowledgement timeout by nazar-pc · Pull Request #2835 · subspace/subspace · GitHub](https://github.com/subspace/subspace/pull/2835) will prevent node from getting stuck completely at least, though the recommendation is still to switch to farming cluster and add some redundancy with more nodes.

---

<div class="post-metadata">

**Author:** ![duanyz\_aiyo](https://yyz1.discourse-cdn.com/flex011/user_avatar/forum.autonomys.xyz/duanyz_aiyo/32/2587_2.png) [@duanyz\_aiyo](https://forum.autonomys.xyz/u/duanyz_aiyo)\
**Post date:** [June 11, 2024, 9:19am UTC](https://forum.autonomys.xyz/t/node-out-of-sync/4164/11 "2024-06-11T09:19:47Z")

</div>

My two nodes have encountered this error again. The complete log file is as follows

[node-01](http://qp.duanyz.net:8088/dl/ssc-node-88.log)  
[node-02](http://qp.duanyz.net:8088/dl/ssc-node-99.log)

---

<div class="post-metadata">

**Author:** ![nazar-pc](https://yyz1.discourse-cdn.com/flex011/user_avatar/forum.autonomys.xyz/nazar-pc/32/6_2.png) [@nazar-pc](https://forum.autonomys.xyz/u/nazar-pc)\
**Post date:** [June 11, 2024, 9:21am UTC](https://forum.autonomys.xyz/t/node-out-of-sync/4164/12 "2024-06-11T09:21:47Z")

</div>

Right, new segment was just archived, so it happens every time. BTW I see you have change number of peers. Not sure why, it should not be necessary and can even be harmful in certain cases.

---

<div class="post-metadata">

**Author:** ![duanyz\_aiyo](https://yyz1.discourse-cdn.com/flex011/user_avatar/forum.autonomys.xyz/duanyz_aiyo/32/2587_2.png) [@duanyz\_aiyo](https://forum.autonomys.xyz/u/duanyz_aiyo)\
**Post date:** [June 11, 2024, 9:32am UTC](https://forum.autonomys.xyz/t/node-out-of-sync/4164/13 "2024-06-11T09:32:51Z")

</div>

Excuse me, what does “new segment was just archived” mean, and how can I avoid this error? The frequent issue of nodes being out of sync

---

<div class="post-metadata">

**Author:** ![nazar-pc](https://yyz1.discourse-cdn.com/flex011/user_avatar/forum.autonomys.xyz/nazar-pc/32/6_2.png) [@nazar-pc](https://forum.autonomys.xyz/u/nazar-pc)\
**Post date:** [June 11, 2024, 9:35am UTC](https://forum.autonomys.xyz/t/node-out-of-sync/4164/14 "2024-06-11T09:35:06Z")

</div>

I already gave you a few options how to avoid above (more nodes, farming cluster), next release will also improve situation for such cases. Farmer was supposed to run with a node on the same machine. 80 farmers connected to the same node was never the intended setup even though I know many do it.

---

<div class="post-metadata">

**Author:** ![duanyz\_aiyo](https://yyz1.discourse-cdn.com/flex011/user_avatar/forum.autonomys.xyz/duanyz_aiyo/32/2587_2.png) [@duanyz\_aiyo](https://forum.autonomys.xyz/u/duanyz_aiyo)\
**Post date:** [June 11, 2024, 9:37am UTC](https://forum.autonomys.xyz/t/node-out-of-sync/4164/15 "2024-06-11T09:37:01Z")

</div>

Okay, I plan to replace all programs with the farmer cluster soon.