HTTP calls experience delay due to rate limiting

[jakamobiii]

Software Versions

OS: Linux
JVM: Oracle Corporation 1.8.0_161 AMD64
Version: 4.8.0

Expected behavior

When a Java-tron API call reaches the rate limit, it should immediately return to inform the client that it has been rate-limited, rather than waiting for an extended period.

Actual behavior

When the Java-tron API call reaches the rate limit, it will enter a queue waiting state and will not return until the items are processed. This will cause the request delay time to be uncontrollable, and the delay time may be very long in high-concurrency scenarios.

Frequency

100% appearance

Steps to reproduce the behavior

1. Limit the rate of a certain interface. You can set a smaller threshold to facilitate problem reproduction, as shown below.

2. After the first request, the system will enter the rate limit state. When the second request is made, the result will not be returned for a rather long time, as shown below.

[waynercheung]

@jakamobiii Hi, you are right, rate limiting is implemented using Guava's RateLimiter.acquire(), the acquire() method requests a single permit. If the current rate allows, it returns immediately. When the rate limit has been reached, it blocks the calling thread until a permit becomes available.

So, if no permits are available, the calling thread is suspended until a permit can be acquired.

Maybe we can use the tryAcquire(long timeout, TimeUnit unit) method instead.

[Sunny6889]

@waynercheung is tryAcquire(long timeout, TimeUnit unit) or some similar functions supported by Guava's library? we better use an official supported one.

[waynercheung]

@waynercheung is tryAcquire(long timeout, TimeUnit unit) or some similar functions supported by Guava's library? we better use an official supported one.

Yes, it is. For more details about tryAcquire(long timeout, TimeUnit unit) you can get from https://github.com/google/guava/blob/master/guava/src/com/google/common/util/concurrent/RateLimiter.java#L351

[Sunny6889]

@waynercheung is tryAcquire(long timeout, TimeUnit unit) or some similar functions supported by Guava's library? we better use an official supported one.

Yes, it is. For more details about tryAcquire(long timeout, TimeUnit unit) you can get from https://github.com/google/guava/blob/master/guava/src/com/google/common/util/concurrent/RateLimiter.java#L351

Great, then how to add this timeout parameters? Is it possible in API level

[waynercheung]

@jakamobiii Hi, what is the negative impact of the current version?

[jakamobiii]

@jakamobiii Hi, what is the negative impact of the current version?

The API call does not return results for a long time, which does not provide a good user experience.

[xxo1shine]

If you prioritize system stability and performance controllability, I suggest you use a non-blocking solution when the API access reaches the rate limit:

Recommendation reasons are as follows:

1. The system is more stable. No backlog of requests, no thread blocking, and avoid service avalanche.
2. The caller experience is clear. The client immediately knows that the rate is limited, and can use Retry-After to prompt the retry time.
3. Friendly to high concurrency. No queue waiting or resource exhaustion in high concurrency scenarios.
4. Easy to control flow control and log tracking. Each flow limit can be monitored or alarmed, which is convenient for tuning strategies.

[Sunny6889]

How about add this non-block option in configuration file? and make the default value = existing false.

[xxo1shine]

How about add this non-block option in configuration file? and make the default value = existing false.

Adding configuration options increases maintenance costs. Telling users earlier can also improve user experience, let users understand the current status, and better control the rate. Personally, I think it is enough to use non-blocking mode directly.