Issue when use a proxy connection to cluster

[franciscopalos]

Hi team.

Currently we are using the following service in docker-compose to extract data from a cluster:

rally:
    depends_on: 
      - es01
    image: elastic/rally:2.11.1
    container_name: esrally
    command: | 
      create-track --track=alertas-indices 
      --target-hosts=myhost.es.psc.europe-west4.gcp.elastic-cloud.com:443
      --indices="index_returned" --output-path=~/tracks
      --client-options="use_ssl:true,verify_certs:false,api_key:'MyAPIKEY',timeout:None"
   
    environment:
      - ALL_PROXY=http://USER:PASS@myproxy:8080
      - HTTP_PROXY=http://USER:PASS@myproxy:8080
      - HTTPS_PROXY=http://USER:PASS@myproxy:8080
    ulimits:
      memlock:
        soft: -1
        hard: -1
    volumes:
      - ./tracks:/rally/tracks
      - ./myrally:/rally/.rally
    networks:
      - elastic  

We get the following error:

2025-03-12 07:09:46,912 -not-actor-/PID:1 esrally.rally INFO OS [uname_result(system='Linux', node='03f4ffac532d', release='5.15.167.4-microsoft-standard-WSL2', version='#1 SMP Tue Nov 5 00:21:55 UTC 2024', machine='x86_64')]
2025-03-12 07:09:46,913 -not-actor-/PID:1 esrally.rally INFO Python [namespace(name='cpython', cache_tag='cpython-312', version=sys.version_info(major=3, minor=12, micro=3, releaselevel='final', serial=0), hexversion=51119088, _multiarch='x86_64-linux-gnu')]
2025-03-12 07:09:46,914 -not-actor-/PID:1 esrally.rally INFO Rally version [2.11.1]
2025-03-12 07:09:46,915 -not-actor-/PID:1 esrally.utils.net INFO Connecting via proxy URL [http://USER:PASS@myproxy:8080] to the Internet (picked up from the environment variable [http_proxy]).
2025-03-12 07:09:46,916 -not-actor-/PID:1 esrally.utils.net INFO Connecting via proxy URL [http://USER:PASS@myproxy:8080] to the Internet (picked up from the environment variable [https_proxy]).
2025-03-12 07:09:46,917 -not-actor-/PID:1 esrally.rally INFO Cleaning track dependency directory [/rally/.rally/libs]...
2025-03-12 07:09:46,918 -not-actor-/PID:1 esrally.client.factory INFO Creating ES client connected to [{'host': 'myhost.es.psc.europe-west4.gcp.elastic-cloud.com', 'port': 443}] with options [{'timeout': None, 'use_ssl': True, 'verify_certs': False, 'api_key': '*****'}]
2025-03-12 07:09:46,927 -not-actor-/PID:1 esrally.client.factory WARNING User has enabled SSL but disabled certificate verification. This is dangerous but may be ok for a benchmark.
2025-03-12 07:09:47,185 -not-actor-/PID:1 elastic_transport.node DEBUG > GET / HTTP/?.?
> Accept: application/json
> Authorization: ApiKey <hidden>
> Connection: keep-alive
> User-Agent: elasticsearch-py/8.6.1 (Python/3.12.3; elastic-transport/8.4.1)
> X-Elastic-Client-Meta: es=8.6.1,py=3.12.3,t=8.4.1,ur=1.26.19
elastic_transport.ConnectionError: Connection error caused by: NewConnectionError(<urllib3.connection.HTTPSConnection object at 0x7f5709f31a60>: Failed to establish a new connection: [Errno -2] Name or service not known)
2025-03-12 07:09:47,187 -not-actor-/PID:1 elastic_transport.transport INFO GET https://myhost.es.psc.europe-west4.gcp.elastic-cloud.com:443/ [status:N/A duration:0.157s]
2025-03-12 07:09:47,187 -not-actor-/PID:1 elastic_transport.node_pool WARNING Node <Urllib3HttpNode(https://myhost.es.psc.europe-west4.gcp.elastic-cloud.com:443)> has failed for 1 times in a row, putting on 1 second timeout
2025-03-12 07:09:47,188 -not-actor-/PID:1 elastic_transport.transport WARNING Retrying request after failure (attempt 0 of 3)
Traceback (most recent call last):
  File "/rally/venv/lib/python3.12/site-packages/elastic_transport/_transport.py", line 328, in perform_request
    meta, raw_data = node.perform_request(
                     ^^^^^^^^^^^^^^^^^^^^^
  File "/rally/venv/lib/python3.12/site-packages/elastic_transport/_node/_http_urllib3.py", line 197, in perform_request
    raise err from None
elastic_transport.ConnectionError: Connection error caused by: NewConnectionError(<urllib3.connection.HTTPSConnection object at 0x7f5709f31a60>: Failed to establish a new connection: [Errno -2] Name or service not known)
2025-03-12 07:09:47,201 -not-actor-/PID:1 elastic_transport.node DEBUG > GET / HTTP/?.?
> Accept: application/json
> Authorization: ApiKey <hidden>
> Connection: keep-alive
> User-Agent: elasticsearch-py/8.6.1 (Python/3.12.3; elastic-transport/8.4.1)
> X-Elastic-Client-Meta: es=8.6.1,py=3.12.3,t=8.4.1,ur=1.26.19
elastic_transport.ConnectionError: Connection error caused by: NewConnectionError(<urllib3.connection.HTTPSConnection object at 0x7f5709f31c40>: Failed to establish a new connection: [Errno -2] Name or service not known)
2025-03-12 07:09:47,202 -not-actor-/PID:1 elastic_transport.transport INFO GET https://myhost.es.psc.europe-west4.gcp.elastic-cloud.com:443/ [status:N/A duration:0.011s]
2025-03-12 07:09:47,202 -not-actor-/PID:1 elastic_transport.node_pool WARNING Node <Urllib3HttpNode(https://myhost.es.psc.europe-west4.gcp.elastic-cloud.com:443)> has failed for 2 times in a row, putting on 2 second timeout
2025-03-12 07:09:47,203 -not-actor-/PID:1 elastic_transport.transport WARNING Retrying request after failure (attempt 1 of 3)
Traceback (most recent call last):
  File "/rally/venv/lib/python3.12/site-packages/elastic_transport/_transport.py", line 328, in perform_request
    meta, raw_data = node.perform_request(
                     ^^^^^^^^^^^^^^^^^^^^^
  File "/rally/venv/lib/python3.12/site-packages/elastic_transport/_node/_http_urllib3.py", line 197, in perform_request
    raise err from None
elastic_transport.ConnectionError: Connection error caused by: NewConnectionError(<urllib3.connection.HTTPSConnection object at 0x7f5709f31c40>: Failed to establish a new connection: [Errno -2] Name or service not known)
2025-03-12 07:09:47,218 -not-actor-/PID:1 elastic_transport.node DEBUG > GET / HTTP/?.?
> Accept: application/json
> Authorization: ApiKey <hidden>
> Connection: keep-alive
> User-Agent: elasticsearch-py/8.6.1 (Python/3.12.3; elastic-transport/8.4.1)
> X-Elastic-Client-Meta: es=8.6.1,py=3.12.3,t=8.4.1,ur=1.26.19
elastic_transport.ConnectionError: Connection error caused by: NewConnectionError(<urllib3.connection.HTTPSConnection object at 0x7f5709f31f40>: Failed to establish a new connection: [Errno -2] Name or service not known)
2025-03-12 07:09:47,219 -not-actor-/PID:1 elastic_transport.transport INFO GET https://myhost.es.psc.europe-west4.gcp.elastic-cloud.com:443/ [status:N/A duration:0.016s]
2025-03-12 07:09:47,220 -not-actor-/PID:1 elastic_transport.node_pool WARNING Node <Urllib3HttpNode(https://myhost.es.psc.europe-west4.gcp.elastic-cloud.com:443)> has failed for 3 times in a row, putting on 4 second timeout
2025-03-12 07:09:47,220 -not-actor-/PID:1 elastic_transport.transport WARNING Retrying request after failure (attempt 2 of 3)
Traceback (most recent call last):
  File "/rally/venv/lib/python3.12/site-packages/elastic_transport/_transport.py", line 328, in perform_request
    meta, raw_data = node.perform_request(
                     ^^^^^^^^^^^^^^^^^^^^^
  File "/rally/venv/lib/python3.12/site-packages/elastic_transport/_node/_http_urllib3.py", line 197, in perform_request
    raise err from None
elastic_transport.ConnectionError: Connection error caused by: NewConnectionError(<urllib3.connection.HTTPSConnection object at 0x7f5709f31f40>: Failed to establish a new connection: [Errno -2] Name or service not known)
2025-03-12 07:09:47,233 -not-actor-/PID:1 elastic_transport.node DEBUG > GET / HTTP/?.?
> Accept: application/json
> Authorization: ApiKey <hidden>
> Connection: keep-alive
> User-Agent: elasticsearch-py/8.6.1 (Python/3.12.3; elastic-transport/8.4.1)
> X-Elastic-Client-Meta: es=8.6.1,py=3.12.3,t=8.4.1,ur=1.26.19
elastic_transport.ConnectionError: Connection error caused by: NewConnectionError(<urllib3.connection.HTTPSConnection object at 0x7f5709f320f0>: Failed to establish a new connection: [Errno -2] Name or service not known)
2025-03-12 07:09:47,234 -not-actor-/PID:1 elastic_transport.transport INFO GET https://myhost.es.psc.europe-west4.gcp.elastic-cloud.com:443/ [status:N/A duration:0.013s]
2025-03-12 07:09:47,235 -not-actor-/PID:1 elastic_transport.node_pool WARNING Node <Urllib3HttpNode(https://myhost.es.psc.europe-west4.gcp.elastic-cloud.com:443)> has failed for 4 times in a row, putting on 8 second timeout
2025-03-12 07:09:47,236 -not-actor-/PID:1 esrally.rally ERROR A fatal error occurred while running subcommand [create-track].
Traceback (most recent call last):
  File "/rally/venv/lib/python3.12/site-packages/esrally/rally.py", line 1211, in dispatch_sub_command
    tracker.create_track(cfg)
  File "/rally/venv/lib/python3.12/site-packages/esrally/tracker/tracker.py", line 82, in create_track
    distribution_flavor, distribution_version, _, _ = factory.cluster_distribution_version(target_hosts, client_options)
                                                      ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/rally/venv/lib/python3.12/site-packages/esrally/client/factory.py", line 365, in cluster_distribution_version
    version = es.info()["version"]
              ^^^^^^^^^
  File "/rally/venv/lib/python3.12/site-packages/elasticsearch/_sync/client/utils.py", line 414, in wrapped
    return api(*args, **kwargs)
           ^^^^^^^^^^^^^^^^^^^^
  File "/rally/venv/lib/python3.12/site-packages/elasticsearch/_sync/client/__init__.py", line 2296, in info
    return self.perform_request(  # type: ignore[return-value]
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/rally/venv/lib/python3.12/site-packages/esrally/client/synchronous.py", line 168, in perform_request
    info = self.transport.perform_request(method="GET", target="/", headers=request_headers)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/rally/venv/lib/python3.12/site-packages/elastic_transport/_transport.py", line 328, in perform_request
    meta, raw_data = node.perform_request(
                     ^^^^^^^^^^^^^^^^^^^^^
  File "/rally/venv/lib/python3.12/site-packages/elastic_transport/_node/_http_urllib3.py", line 197, in perform_request
    raise err from None
elastic_transport.ConnectionError: Connection error caused by: ConnectionError(Connection error caused by: NewConnectionError(<urllib3.connection.HTTPSConnection object at 0x7f5709f31a60>: Failed to establish a new connection: [Errno -2] Name or service not known))

We have checked that proxy connection is working fine. I check the same rally version installed on my host with the same configuration (host linux with http_proxy and https_proxy defined) and the result is the same.

It is possible that there is an issue about the use of proxy connections?

Thanks & Kind Regards

[gareth-ellis]

Hello - thanks for reaching out.

From our documentation it seems that http_proxy should be lower case - though the other two are in acceptable in upper case or lower case.

I can see we note as well

Rally will use this proxy server only for downloading benchmark-related data. It will not use this proxy for the actual benchmark.

the elasticsearch client, which is what rally uses to interact with elasticsearch doesnt support proxies at the moment - there is a work around, though that would need to be added to rally, which i'm not sure if that is something that is the right thing to add.

Ref: elastic/elasticsearch-py#2217

[franciscopalos]

Hi,

We have tested with http_proxy and https_proxy in lower case and the result is the same, As you comment before its appear that it should be a new feature from elasticsearch client used by Rally.

Thanks & kind Regards