ratelimitprocessor: Implement dynamic rate limit

[marclop]

This PR introduces dynamic rate limiting to the ratelimitprocessor when using the gubernator type. This feature allows the processor to automatically adjust the rate limit based on traffic patterns, providing a more responsive and adaptive way to manage high-volume data streams.

The goal is to create a rate limit that isn't fixed but adapts to the actual traffic flowing through the processor. It should rise to accommodate sustained increases in traffic and decrease when traffic subsides, all while preventing sudden, massive spikes from destabilizing the system.

Key Features:

• Dynamic Rate Adjustment: The rate limit is no longer static but adapts to the actual traffic flowing through the processor.

• Simple Window-based Algorithm: The dynamic limit is calculated using the previous window's traffic rate multiplied by a configurable factor.

• Configuration: A new dynamic_limits section in the configuration allows enabling and tuning the behavior.

• Prorated Current Window: The calculation for the current window's traffic rate is prorated based on the elapsed time to prevent the limit from dropping artificially at the start of a new window.

• Static Override: A static_only option has been added to the overrides configuration to disable dynamic rate limiting for specific use cases.

How it Works:

The system's memory of recent traffic is based on two "sliding windows":

• Current Window: The most recent, active time block. Its size is defined by the window_duration configuration parameter (e.g., 2m). The system is actively counting the number of hits that occur within this window.
• Previous Window: The time block of the same size that came immediately before the current one.

When the clock moves to the next window, the old "Current Window" becomes the "Previous Window", and a new "Current Window" begins.

From these windows, the key metric is derived:

• previous_rate: Average rate of traffic within the previous window (normalized per second).

The final dynamic limit is calculated with a simple formula:

dynamic_limit = max(static_rate, previous_rate * window_multiplier)

The max(static_rate, ...) ensures a minimum guaranteed throughput, while the previous_rate * window_multiplier allows the limit to scale based on actual observed traffic patterns.

Tuning the Dynamic Limiter:

The behavior of the dynamic limiter can be tuned by adjusting the following parameters:

• window_duration:

  • Small (e.g., 1m): More reactive to traffic changes.
  • Large (e.g., 5m): Smoother, more stable limit based on longer observation periods.

• window_multiplier:

  • High (e.g., 2.0): Allows the limit to grow more aggressively.
  • Low (e.g., 1.2): More conservative, slower growth.

This mechanism allows the rate limit to adapt to sustained increases in traffic while the window_multiplier provides protection against sudden spikes that could destabilize the system.

Screenshot

Tested using this config:

ratelimit:
  rate: 20_468_200   # bytes allowed per 10s window
  burst: 4_093_640   # 2x rate for 1s window for handling traffic spikes
  type: gubernator
  strategy: bytes
  throttle_interval: 10s
  dynamic_limits:
    enabled: true
    window_duration: 2m
    window_multiplier: 1.4