Poor performance when enqueuing many throttled jobs because of unindexed queries

[Intrepidd]

When the concurrency extension looks up for existing queued jobs within the throttle period, it performs the following query :

good_job/lib/good_job/active_job_extensions/concurrency.rb

Lines 93 to 95 in b557525

	enqueued_within_period = GoodJob::Job.where(concurrency_key: key)
	.where(GoodJob::Job.arel_table[:created_at].gt(throttle_period.ago))
	.count

In my case, this query will run in 200-300ms for each enqueue attempt, so it adds up pretty quickly when trying to enqueue hundreds of jobs.

The index on concurrency_key has a where finished_at is NULL, so it can't be used here.

Would it make sense to have an index solely on concurrency_key ?

Thanks

[bensheldon]

Good catch on that. I think you're correct that the current index is insufficient for the throttle query 😓

I think the correct index here would be [concurrency_key, :created_at]. That would also cover the execution throttling (as it can use just the first column of the compound index):

https://github.com/bensheldon/good_job/blob/b557525a9003a1fc1a434742dae58053b947e708/lib/good_job/active_job_extensions/concurrency.rb#L162C1-L164C165

[Intrepidd]

Thanks ! I will make a PR shortly if that's ok

[Intrepidd]

Just opened #1605 to try and fix this

[Intrepidd]

Closing as it's been merged, thanks !

[Intrepidd]

Just wanted to report that it worked very well, thanks again !