Namespace Parallelism Limit

[SerRichard]

In the release version v3.6.7, the parallelism option for specific namespaces was added using the label workflows.argoproj.io/parallelism-limit, I'm having trouble getting the feature to work. I've combed the issues and noticed I needed to add a cluster role for get, watch, and list verbs for namespace objects, which I have done, but the feature still does not seem to work.

The simplest example I have currently is using the quick-start-guide and editing the argo-workflows-controller service account to have the correct rbac verbs for namespaces.

Does anyone have any input on what may be missing, or how I could sufficiently modify the quick-start yaml to be sure this feature is enabled?

@isubasinghe Hope you don't mind the tag, as I see it's something that you implemented, so hoping you can help 🙏

P.S I felt this might be a user issue, so thought I'd open a discussion instead of an issue!

Thanks in advance!

[SerRichard]

So, I noticed this morning that the controller did indeed apply the limit from the namespace label some time over the weekend. I've been unable to pinpoint when exactly this happened, but it does not seem to be updating every 5 minutes configured in the controller watcher.

[isubasinghe]

Hmm thanks @SerRichard ill have a look into this tomorrow, apologies for the delay.

[isubasinghe]

Interesting @SerRichard I cannot seem to replicate this at all.
What could be happening is the loss of information if the api is overloaded perhaps?

I am assuming you did get the new fix with the correct RBAC permissions since you state that in the comment you made.

Are you seeing any logs regarding the namespace parallelism changing ?
I guess it could be the throttler?

argo-workflows/workflow/controller/ns_watcher.go

Line 78 in 44f9ea7

updateNS(logger, ns, wfc.throttler.UpdateNamespaceParallelism, wfc.throttler.ResetNamespaceParallelism)

Or perhaps it is hitting this? https://github.com/argoproj/argo-workflows/blob/main/workflow/controller/ns_watcher.go#L75-L77

I am unable to reproduce your issue unfortunately. Do you have the capacity to patch and test something for me? We could annotate the ns_watcher.go file with a bunch of logs. I don't really see anything incorrect in that code unfortunately, it is a stock standard informer.

[isubasinghe]

You can narrow down the logs by adding filtering for scope=ns_watcher

[SerRichard]

@isubasinghe No worries on delay, we're all busy 😄 Just to be sure of the permission fix. I've deployed argo-workflows via the community helm chart, so I created a separate cluster role and binding to add this to the controller service account. This should be correct as far as I understand? I'm more suspect that it's something here, as I don't see any logs for the namespace attempting to change.

Failing the above being the issue, I'm happy to annotate the watcher and deploy in my environments to further debug.

[isubasinghe]

thanks hahaha :) I meant this fix: #14459, I don't think it's been released so you the helm chart install wouldn't have it in that case.

But just to be clear, you are seeing no logs when you filter by scope=ns_watcher ?

[SerRichard]

Hey, apologies for the delayed response! I tried updating the community helm chart with the change to the config map and that didn't manage to fix it. Do you know of any other edits I would need to make to the helm chart for this functionality?

For annotating the watcher, are there any docs covering building the the controller locally? The dev manual doesn't seem to cover that. Also, I'm not sure I understand where I would add scope=ns_watcher, would that be to the container command?

[isubasinghe]

All good!

Are you a nix user by chance? There is a flake you can use to get it running locally pretty easily.
There is also the devcontainer.

But I meant can you search your controller logs for "scope=ns_watcher"?