Thread '<unknown>' has overflowed its stack

Support Incentivized Testnet
1

2023-09-08T06:22:16.725163Z INFO single_disk_farm{disk_farm_index=2}: subspace_farmer::single_disk_farm::plotting: Sector plotted successfully (16.59%) sector_index=1164

thread ‘’ has overflowed its stack
fatal runtime error: stack overflow
./farmer.sh: line 34: 2975492 Aborted (core dumped) $cmd

Not sure what happened here,unfortunately don’t have any more data. There was no mention in dmesg norwas there more information in the stdout of the farmer. this was Centos9 and sept-11 compiled

2

This is not good for sure, but hard to help without even a thread name. Is it reproducible for you?

3

I have not seen it again, and unfortunately the single colons that might include a name, were empty, just in case I didn’t delete the thread name thread (thread ‘’ has overflowed its stack) There was no mention of OOM reaper or similar in dmesg.

4

Was this with a custom build where you have tuned some parameters or the official one?

5

this is on sep-11 and built myself,

Blockquote
export RUSTFLAGS=“-C target-cpu=native -C llvm-args=-cost-kind=latency -C opt-level=3”
export CFLAGS=“-march=native -mtune=native -O3 -pipe -fstack-protector-strong”
export CXXFLAGS=“-march=native -mtune=native -O3 -pipe -fstack-protector-strong”

Blockquote
cargo build
–profile production
–bin subspace-node
–bin subspace-farmer
–target x86_64-unknown-linux-gnu -Z build-std \

Blockquote
$ clang --version
clang version 15.0.7 (Fedora 15.0.7-2.fc37)
Target: x86_64-redhat-linux-gnu
Thread model: posix
InstalledDir: /usr/bin

6

If code wasn’t modified then should still be applicable to official builds

7

I have only modified one value as a last resort to see if it made a difference, not generally modifying the code, that modification as it didn’t have any positive effect is undone.

8

Well, maybe that value did cause something to increase in size and overflow the stack. It is often one line change that breaks things.

9

This error happened on a setup without any code changes applied. The code changes I applied on the box where farming (vote/sign) just stops, when plotting gets active.

10

Xorinox faced theissue again in Discord, this time with a thread #. Here is his message

2023-10-10T22:06:49.927011Z  INFO single_disk_farm{disk_farm_index=2}: subspace_farmer::single_disk_farm::plotting: Sector plotted successfully (89.90%) sector_index=1556
2023-10-10T22:08:27.941504Z  INFO single_disk_farm{disk_farm_index=2}: subspace_farmer::single_disk_farm::plotting: Sector plotted successfully (89.95%) sector_index=1557
2023-10-10T22:10:05.424077Z  INFO single_disk_farm{disk_farm_index=2}: subspace_farmer::single_disk_farm::plotting: Sector plotted successfully (90.01%) sector_index=1558

thread 'plotting#22' has overflowed its stack
fatal runtime error: stack overflow

Unfortunately don't have more data, oct-6, didn't see mem issues on machine, Fedora v37,5950x, 32 G DDR4
11

Thanks, I think this is related to stack overflow in nested ThreadPool.scope · Issue #751 · rayon-rs/rayon · GitHub upstream.

Would be nice to find an easily reproducible example, but overall I think this will not be applicable to the next version of the network due to plotting not doing encoding for more than one plot at a time by default (it is less efficient and SIGNIFICANTLY slower even though appears to use CPU cores more fully).

12

Make plotting/replotting thread pools distinct for farms to avoid stack overflow, still protected by semaphores by nazar-pc · Pull Request #2167 · subspace/subspace · GitHub will work around this stack overflow fully.

13

Will these changes be made to the pulsar? The user is facing the same problem

14

Yes, this fix will be included in Pulsar as well. Though if user faces it with Pulsar that doesn’t support multiple farms yet IIRC then they might have a different issue and they should report it as such.