buffer size overflow error while ingesting large size document.

[tahseen2k8]

this is the error which i can see on sonic server side.

(ERROR) - closing channel thread because of buffer overflow
thread 'sonic-channel-client' panicked at 'buffer overflow (24168/20002 bytes)', /home/xxx/.cargo/registry/src/github.aichem.org-1ecc6299db9ec823/sonic-server-1.3.5/src/channel/handle.rs:149:29
note: run with `RUST_BACKTRACE=1` environment variable to display a backtrace
(DEBUG) - running a tasker tick...

can someone guide me how large document/ text i can put in one go and what are the config vars which i can tweak to make this working . TIA

[valeriansaliou]

Which Sonic Channel library are you using to ingest documents? If you're using a library that has no backpressure control or splitting feature, then Sonic may run behind (ie. you're pushing faster than Sonic can ingest), and therefore it is aborting the channel for safety reasons.

[tahseen2k8]

@valeriansaliou thanks for your message, I have tried only single message to ingest . though its a large html of whole page just for testing .
I have tried both below mentioned libraries.
https://github.com/xmonader/python-sonic-client
https://github.com/AlongWY/pysonic

Also I have tried with not only custom installed sonic but also with Docker image

[tahseen2k8]

in addition I have 1 TB of text which i want to ingest and doc size vary from 1 to other any recommendation how to ingest them super fast without getting this kind of issues .

[valeriansaliou]

I am not sure the mentionned libraries have backpressure control nor support for long text splitting, you may ask their authors. Can you try w/ https://github.com/valeriansaliou/node-sonic-channel which implements both? That's the one we use in production at Crisp w/o a hitch.