Skip to content

Upgrade k8s to v1.32.0 and kind to v0.27.0 #2502

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Merged
merged 5 commits into from
Apr 3, 2025
Merged

Upgrade k8s to v1.32.0 and kind to v0.27.0 #2502

merged 5 commits into from
Apr 3, 2025

Conversation

mauri870
Copy link
Member

@mauri870 mauri870 requested a review from a team as a code owner March 31, 2025 16:58
@mauri870 mauri870 self-assigned this Mar 31, 2025
@mauri870 mauri870 marked this pull request as draft March 31, 2025 17:12
@mauri870
Copy link
Member Author

The builds are failing with:

kind v0.27.0 go1.23.6 linux/amd64
/opt/buildkite-agent/builds/bk-agent-prod-gcp-1743440530189926369/elastic/elastic-package/bin/kind
/opt/buildkite-agent/builds/bk-agent-prod-gcp-1743440530189926369/elastic/elastic-package/bin/kubectl: line 1: syntax error near unexpected token `<'
/opt/buildkite-agent/builds/bk-agent-prod-gcp-1743440530189926369/elastic/elastic-package/bin/kubectl: line 1: `<?xml version='1.0' encoding='UTF-8'?><Error><Code>NoSuchKey</Code><Message>The specified key does not exist.</Message><Details>No such object: kubernetes-release/release/v1.32.0/bin/linux/amd64/kubectl</Details></Error>'

Looks like we are relying on a public GCS bucket to download releases, that is an implementation detail of the k8s binary releases and with v1.32 this no longer works. See kubernetes/kubernetes#127796.

We have to change this to use dl.k8s.io.

@mauri870 mauri870 mentioned this pull request Mar 31, 2025
Merged
5 tasks
@mauri870 mauri870 marked this pull request as ready for review March 31, 2025 17:31
@mauri870 mauri870 mentioned this pull request Mar 31, 2025
Merged
8 tasks
@mauri870 mauri870 marked this pull request as draft April 1, 2025 11:23
@mauri870
Copy link
Member Author

mauri870 commented Apr 1, 2025

/test

@mauri870
Copy link
Member Author

mauri870 commented Apr 2, 2025 •
edited
Loading

@mrodm I'm having a bit of trouble with the CI, seems like kube state metrics is not starting properly:

2025/04/01 20:23:41 DEBUG Error received when checking status of resource kube-state-metrics. Error: 'client rate limiter Wait returned an error: context deadline exceeded', Resource details: 'Resource: "/v1, Resource=Services", GroupVersionKind: "/v1, Kind=service"

Any tips on how to go about debugging this?

mrodm added 2 commits April 2, 2025 17:17
@mrodm
Set the k8s node version using env var
0048b99 
Setting the "--image" parameter is the same done in the integrations
repository.
@mrodm
Update nginx image used in the statefulset
054acec
@mrodm
Copy link
Contributor

mrodm commented Apr 2, 2025

@mrodm I'm having a bit of trouble with the CI, seems like kube state metrics is not starting properly:

2025/04/01 20:23:41 DEBUG Error received when checking status of resource kube-state-metrics. Error: 'client rate limiter Wait returned an error: context deadline exceeded', Resource details: 'Resource: "/v1, Resource=Services", GroupVersionKind: "/v1, Kind=service"

Any tips on how to go about debugging this?

This is unexpected, since the tests run in the PR (elastic/integrations#13365) in the integrations repository finished successfully.

@mauri870 Are the definitions in the kubernetes package (for the subset of the data streams) here different from the one in the integrations repository?

I've tried to run it locally with these commands (installing the specified versions of kind and kubectl of this PR):

kind create cluster --config scripts/kind-config.yaml

cd test/packages/with-kind/kubernetes
elastic-package build -v
elastic-package stack up -v -d

elastic-package test system -v --data-streams state_pod

This will try to create the corresponding resources in the kind cluster. Looking at the pods created there is one in ImagePullBackOff:

 $ kubectl get pods 
NAME                   READY   STATUS             RESTARTS   AGE
hello-29060098-q4kxl   0/1     Completed          0          2m33s
hello-29060099-h9ppk   0/1     Completed          0          93s
hello-29060100-cbnl5   0/1     Completed          0          33s
hello-jdmkb            0/1     Completed          0          3m11s
web-0                  0/1     ImagePullBackOff   0          3m11s

Running kubectl describe pod in that pod shows these errors:

  Warning  Failed     21s (x5 over 3m19s)  kubelet            Error: ErrImagePull
  Normal   BackOff    7s (x12 over 3m18s)  kubelet            Back-off pulling image "k8s.gcr.io/nginx-slim:0.8"
  Warning  Failed     7s (x12 over 3m18s)  kubelet            Error: ImagePullBackOff

I've found that this image was updated in this PR elastic/integrations#6310 to be docker.io/nginx:1.24.0-alpine-slim

Trying updating the docker image here 054acec

mrodm
mrodm reviewed Apr 2, 2025
View reviewed changes
scripts/test-check-packages.sh
@@ -90,7 +90,7 @@ fi

if [ "${PACKAGE_TEST_TYPE:-other}" == "with-kind" ]; then
# Boot up the kind cluster
kind create cluster --config "$PWD/scripts/kind-config.yaml"
kind create cluster --config "$PWD/scripts/kind-config.yaml" --image "kindest/node:${K8S_VERSION}"
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Update the command to be the same as in the integrations repository. This will ensure that the k8s node versions are the same as the one set in the environment variable.

mrodm
mrodm reviewed Apr 2, 2025
View reviewed changes
test/packages/with-kind/kubernetes/_dev/deploy/k8s/ss.yaml
@@ -16,7 +16,7 @@ spec:
terminationGracePeriodSeconds: 10
containers:
- name: nginx
image: k8s.gcr.io/nginx-slim:0.8
image: docker.io/nginx:1.24.0-alpine-slim
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

@mauri870 Do you know if there could be other changes (from the package in the integrations repository) that should be applied in this package ?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I'm not aware of any additional changes.

mrodm
mrodm reviewed Apr 2, 2025
View reviewed changes
test/packages/with-kind/kubernetes/data_stream/state_cronjob/_dev/deploy/k8s/deployment.yaml
@@ -21,7 +21,7 @@ spec:
spec:
automountServiceAccountToken: true
containers:
- image: registry.k8s.io/kube-state-metrics/kube-state-metrics:v2.7.0
- image: registry.k8s.io/kube-state-metrics/kube-state-metrics:v2.15.0
Copy link
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't find this definition in the integrations repo for the kubernetes package.

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I think it is applied with Kustomize here https://github.com/elastic/integrations/blob/619d4001668a27da69dd661e2f1300faef3bcccf/packages/kubernetes/_dev/deploy/k8s/kustomization.yaml#L10

@elasticmachine
Copy link
Collaborator

💚 Build Succeeded

History

cc @mauri870

@mauri870 mauri870 marked this pull request as ready for review April 2, 2025 17:50
@mauri870
Copy link
Member Author

mauri870 commented Apr 2, 2025

@mrodm Thanks for helping out with the CI issues!

@mauri870 mauri870 requested a review from mrodm April 2, 2025 19:33
mrodm
mrodm approved these changes Apr 3, 2025
View reviewed changes
Copy link
Contributor

@mrodm mrodm left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

👍

@mauri870 mauri870 merged commit ff281d2 into elastic:main Apr 3, 2025
3 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@mrodm mrodm mrodm approved these changes

Assignees

@mauri870 mauri870

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.
3 participants
@mauri870 @mrodm @elasticmachine