[MMM-20372] Make pool_maxsize configurable.

[nickolai-dr]

Should address:

WARNING urllib3.connectionpool: Connection pool is full, discarding connection: datarobot-otel-collector. Connection pool size: 10

in high perf scenarios.

[s-gavrenkov]

What about

total_concurrent_requests = 500
num_workers = self._params.get("processes")

pool_maxsize = ceil(total_concurrent_requests / num_workers)
for 8 cores it is ~60

[s-gavrenkov]

# Custom adapter with default timeout
class TimeoutHTTPAdapter(requests.adapters.HTTPAdapter):
    def __init__(self, *args, timeout=5, **kwargs):
        self.timeout = timeout
        super().__init__(*args, **kwargs)

    def send(self, request, **kwargs):
        kwargs["timeout"] = kwargs.get("timeout", self.timeout)
        return super().send(request, **kwargs)

adapter = TimeoutHTTPAdapter(pool_maxsize=pool_maxsize, timeout=5)

[s-gavrenkov]

probably need timeout

[s-gavrenkov]

or

class TimeoutSession(requests.Session):
    def __init__(self, timeout=(5, 15)):
        super().__init__()
        self._timeout = timeout

    def request(self, *args, **kwargs):
        kwargs.setdefault("timeout", self._timeout)
        return super().request(*args, **kwargs)

# Example usage:
pool_maxsize = 10  # or any desired value
session = TimeoutSession(timeout=(5, 15))
adapter = HTTPAdapter(pool_maxsize=pool_maxsize)

[nickolai-dr]

Exporter already has timeout config that they pass into session, can be configured by env var:
OTEL_EXPORTER_OTLP_TRACES_TIMEOUT
see
https://github.com/open-telemetry/opentelemetry-python/blob/31289bd1722fcb8b09d9b7a36a7e00d5cf7e17a3/exporter/opentelemetry-exporter-otlp-proto-http/src/opentelemetry/exporter/otlp/proto/http/trace_exporter/__init__.py#L111-L115

[nickolai-dr]

Default timeout is 10 sec.

[s-gavrenkov]

Need to check timeouts.

[s-gavrenkov]

need to check timeout.