bk: use OIDC to tear-down the cloud resources #2567

v1v · 2025-05-05T09:26:33Z

What

Use OIDC to access the AWS cloud resources and run the tear-down.
This will run in parallel while we migrate.
cloud-reaper does not support OIDC, hence creating a service account that will be deleted afterwards

Why

Avoid using service accounts and use the Keyless/OIDC approach.

Test

See https://buildkite.com/elastic/elastic-package-cloud-cleanup/builds/459

Further details

Uses https://github.com/elastic/oblt-aws-auth-buildkite-plugin

Requires #2568, so i can test this PR in isolation before merging it.

This will run in parallel while we do the migration Avoid using service accounts and use the Keyless/OIDC approach Uses https://github.com/elastic/oblt-aws-auth-buildkite-plugin

.buildkite/scripts/cloud-cleanup-oidc.sh

this should help with the lack of support for OIDC in cloud-reaper

v1v · 2025-05-05T13:24:29Z

.buildkite/hooks/pre-command

 if [[ "$BUILDKITE_PIPELINE_SLUG" == "elastic-package-cloud-cleanup" && "$BUILDKITE_STEP_KEY" == "cloud-cleanup" ]]; then
+    AWS_SERVICE_ACCOUNT_SECRET_PATH=kv/ci-shared/platform-ingest/aws_ingest_ci


let's define the variable where it's used

v1v · 2025-05-05T13:25:05Z

.buildkite/scripts/cloud-cleanup-oidc.sh

@@ -0,0 +1,229 @@
+#!/usr/bin/env bash


I copied it from .buildkite/scripts/cloud-cleanup.sh and added the support for OIDC.

v1v · 2025-05-05T13:25:34Z

.buildkite/configs/cleanup.aws.yml

@@ -5,8 +5,8 @@ accounts:
  - name: "${ACCOUNT_PROJECT}"
    driver: "aws"
    options:
-      key: '${ACCOUNT_KEY}'
-      secret: '${ACCOUNT_SECRET}'
+      key: '${AWS_ACCESS_KEY_ID}'


To help with using the existing env variables created by https://github.com/elastic/oblt-aws-auth-buildkite-plugin

v1v · 2025-05-05T13:27:31Z

.buildkite/scripts/cloud-cleanup-oidc.sh

+# As long as cloud reaper does not support OIDC authentication.
+create_aws_ephemeral_user() {
+    # Generate a unique name for the ephemeral IAM user.
+    EPHEMERAL_USER="ephemeral-admin-$(date +%s)"
+    echo "Creating IAM user: ${EPHEMERAL_USER}"
+    aws iam create-user --user-name "${EPHEMERAL_USER}" \
+        --tags Key=ephemeral,Value=true Key=division,Value=engineering Key=org,Value=obs Key=environment,Value=ci Key=repo,Value=elastic-package Key=created_at,Value="$(date -Is)"
+
+    echo "Attaching AdministratorAccess policy to ${EPHEMERAL_USER}..."
+    aws iam attach-user-policy --user-name "${EPHEMERAL_USER}" --policy-arn arn:aws:iam::aws:policy/AdministratorAccess
+
+    echo "Creating access keys for ${EPHEMERAL_USER}..."
+    creds_json=$(aws iam create-access-key --user-name "${EPHEMERAL_USER}")
+    AWS_ACCESS_KEY_ID_EPHEMERAL=$(echo "$creds_json" | jq -r '.AccessKey.AccessKeyId')
+    AWS_SECRET_ACCESS_KEY_EPHEMERAL=$(echo "$creds_json" | jq -r '.AccessKey.SecretAccessKey')
+    export EPHEMERAL_USER AWS_ACCESS_KEY_ID_EPHEMERAL AWS_SECRET_ACCESS_KEY_EPHEMERAL
+}
+
+# Define cleanup function to delete the ephemeral IAM user regardless of script outcome.
+cleanup_ephemeral_user() {
+    echo "Cleaning up ephemeral IAM user: ${EPHEMERAL_USER}"
+    aws iam detach-user-policy --user-name "${EPHEMERAL_USER}" --policy-arn arn:aws:iam::aws:policy/AdministratorAccess
+    key_id=$(echo "$creds_json" | jq -r '.AccessKey.AccessKeyId')
+    aws iam delete-access-key --user-name "${EPHEMERAL_USER}" --access-key-id "${key_id}"
+    aws iam delete-user --user-name "${EPHEMERAL_USER}"
+    echo "Ephemeral IAM user ${EPHEMERAL_USER} deleted."
+}
+trap cleanup_ephemeral_user EXIT
+
+cloud_reaper_aws() {
+    echo "--- Configuring ephemeral user"
+    create_aws_ephemeral_user


This is what I added to help with the OIDC when running cloud-reaper

v1v · 2025-05-05T13:27:50Z

.buildkite/scripts/cloud-cleanup-oidc.sh

+echo "--- Installing awscli"
+with_aws_cli
+


Run earlier, to help with using the aws cli to create the ephemeral service account

mrodm · 2025-05-05T15:41:18Z

.buildkite/scripts/cloud-cleanup-oidc.sh

+    AWS_ACCESS_KEY_ID_EPHEMERAL=$(echo "$creds_json" | jq -r '.AccessKey.AccessKeyId')
+    AWS_SECRET_ACCESS_KEY_EPHEMERAL=$(echo "$creds_json" | jq -r '.AccessKey.SecretAccessKey')
+    export EPHEMERAL_USER AWS_ACCESS_KEY_ID_EPHEMERAL AWS_SECRET_ACCESS_KEY_EPHEMERAL


Should AWS_ACCESS_KEY_ID_EPHEMERAL and AWS_SECRET_ACCESS_KEY_EPHEMERAL be renamed to follow the patterns so their contents are redacted ?

https://buildkite.com/docs/pipelines/configure/managing-log-output#redacted-environment-variables

Maybe something like AWS_ACCESS_KEY_ID_EPHEMERAL_SECRET and AWS_SECRET_ACCESS_KEY_EPHEMERAL_SECRET ? Or is it not needed in this scenario ?

It does not matter at all, those env variables are not masked, unless they are set in the pre-command.

When creating env variable on the fly, there is no way to redact values

Ok, got it!

.buildkite/scripts/cloud-cleanup-oidc.sh

mrodm

Thanks!

v1v · 2025-05-05T16:02:03Z

I'm running this build to validate if the recent changes with dry-run: false work as expected

elasticmachine · 2025-05-05T16:33:39Z

💚 Build Succeeded

Buildkite Build
Commit: aaad895

History

💚 Build #5231 succeeded b8999af
💚 Build #5229 succeeded 51771ba
💚 Build #5225 succeeded 9c0faa9

cc @v1v

bk: use OIDC to tear-down the cloud resources

9c0faa9

This will run in parallel while we do the migration Avoid using service accounts and use the Keyless/OIDC approach Uses https://github.com/elastic/oblt-aws-auth-buildkite-plugin

v1v mentioned this pull request May 5, 2025

catalog-info: grant build access to the obs-ci team #2568

Merged

v1v self-assigned this May 5, 2025

v1v added 2 commits May 5, 2025 13:15

fix

e65d705

debug

51771ba

v1v commented May 5, 2025

View reviewed changes

.buildkite/scripts/cloud-cleanup-oidc.sh Outdated Show resolved Hide resolved

v1v added 3 commits May 5, 2025 15:03

create ephemeral user

8042e94

this should help with the lack of support for OIDC in cloud-reaper

add more tags

0fe3b95

install earlier

b8999af

v1v requested review from mrodm and a team May 5, 2025 13:23

v1v marked this pull request as ready for review May 5, 2025 13:23

v1v requested a review from a team as a code owner May 5, 2025 13:23

v1v commented May 5, 2025

View reviewed changes

kuisathaverat approved these changes May 5, 2025

View reviewed changes

mrodm reviewed May 5, 2025

View reviewed changes

v1v commented May 5, 2025

View reviewed changes

.buildkite/scripts/cloud-cleanup-oidc.sh Show resolved Hide resolved

mrodm approved these changes May 5, 2025

View reviewed changes

Update .buildkite/scripts/cloud-cleanup-oidc.sh

aaad895

v1v enabled auto-merge (squash) May 5, 2025 16:09

v1v merged commit bddbc78 into main May 5, 2025
3 checks passed

v1v deleted the feature/cloud-reaper-oidc branch May 5, 2025 16:33

v1v mentioned this pull request May 6, 2025

bk: remove cloud cleanup without AWS OIDC #2573

Merged

mrodm mentioned this pull request May 7, 2025

[CI] Allow to run each cloud cleanup step independently in DRY_RUN false mode #2581

Merged

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

bk: use OIDC to tear-down the cloud resources #2567

bk: use OIDC to tear-down the cloud resources #2567

Uh oh!

v1v commented May 5, 2025 •

edited

Loading

Uh oh!

Uh oh!

v1v May 5, 2025

Uh oh!

v1v May 5, 2025

Uh oh!

v1v May 5, 2025

Uh oh!

v1v May 5, 2025

Uh oh!

v1v May 5, 2025

Uh oh!

mrodm May 5, 2025

Uh oh!

v1v May 5, 2025 •

edited

Loading

Uh oh!

mrodm May 5, 2025

Uh oh!

Uh oh!

mrodm left a comment

Uh oh!

v1v commented May 5, 2025

Uh oh!

Uh oh!

elasticmachine commented May 5, 2025

Uh oh!

Uh oh!

		if [[ "$BUILDKITE_PIPELINE_SLUG" == "elastic-package-cloud-cleanup" && "$BUILDKITE_STEP_KEY" == "cloud-cleanup" ]]; then
		AWS_SERVICE_ACCOUNT_SECRET_PATH=kv/ci-shared/platform-ingest/aws_ingest_ci

bk: use OIDC to tear-down the cloud resources #2567

bk: use OIDC to tear-down the cloud resources #2567

Uh oh!

Conversation

v1v commented May 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What

Why

Test

Further details

Uh oh!

Uh oh!

v1v May 5, 2025

Choose a reason for hiding this comment

Uh oh!

v1v May 5, 2025

Choose a reason for hiding this comment

Uh oh!

v1v May 5, 2025

Choose a reason for hiding this comment

Uh oh!

v1v May 5, 2025

Choose a reason for hiding this comment

Uh oh!

v1v May 5, 2025

Choose a reason for hiding this comment

Uh oh!

mrodm May 5, 2025

Choose a reason for hiding this comment

Uh oh!

v1v May 5, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

mrodm May 5, 2025

Choose a reason for hiding this comment

Uh oh!

Uh oh!

mrodm left a comment

Choose a reason for hiding this comment

Uh oh!

v1v commented May 5, 2025

Uh oh!

Uh oh!

elasticmachine commented May 5, 2025

💚 Build Succeeded

History

Uh oh!

Uh oh!

v1v commented May 5, 2025 •

edited

Loading

v1v May 5, 2025 •

edited

Loading