Test pipeline does not work with already existing cluster

[stevengoossensB]

While most elastic-package commands work while running them with specific environments, to avoid having to spin up a stack every time again, the test pipeline fails because of a docker command which is being ran. Is there a way to overcome this?

[jsoriano]

Hey @stevengoossensB, I am trying to improve the situation for these use cases on #2298.

[stevengoossensB]

I found a workaround by accident.

It seems the only really blocker is that the elastic-package test pipeline commands use docker at some point to get logs (not really sure why). I've circumvented this by having a lightweight docker-compose file, that spins up a stack called elastic-package-stack with lightweight containers, which tricks the command in thinking a local stack exists. When setting the environment variables to a remote stack, it will now run the _simulate on a remote cluster and still return the correct results test results.

[jsoriano]

@stevengoossensB have you tried with the last version of elastic-package? The new environment stack provider could work for your case.

At the moment this new provider modifies the Fleet configuration of the existing deployment, so don't use it if your deployment is used for production purposes. If you want to try it, set the environment variables with the info of your existing stack, at least ELASTIC_PACKAGE_ELASTICSEARCH_HOST and ELASTIC_PACKAGE_KIBANA_HOST, and then run elastic-package stack up --provider environment.

[stevengoossensB]

Hi,

@jsoriano This does much more then what we want. I've further debugged the issue and it seems the test pipeline runs 3 different Docker commands:

• docker ps -a --filter label=com.docker.compose.project=elastic-package-stack
• docker inspect <id's of containers>
• docker compose version --short

This is not the case for the benchmark pipeline, which works like a charm on a remote cluster. I've found where this is done in the code, but I'm not sure whether I see the point of this validation. Would it be an idea to add a flag to skip the stack validation?

Best regards,

Steven