Skip to content

Automatically wait for retries in gh eval #75

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Merged
merged 2 commits into from
Jul 21, 2025
Merged

Conversation

sgoedecke
Copy link
Collaborator

@sgoedecke sgoedecke commented Jul 21, 2025

This PR updates gh eval to listen for the retry-after and x-retry-timeremaining headers and automatically wait.

If we can't read either of those headers, we just wait for a minute.

@sgoedecke ➜ /workspaces/gh-models (sgoedecke/eval-retries) $ ./gh-models eval examples/sample_prompt.yml 
Running evaluation: Sample Evaluation
Description: A sample evaluation for testing the eval command
Model: deepseek-v3-0324
Test cases: 2

Running test case 1/2...
  ✓ PASSED
    ✓ string evaluator (score: 1.00)
      Expected to contain: 'hello'
    ✓ similarity check (score: 0.25)
      LLM evaluation matched choice: '2'

Running test case 2/2...
    Rate limited, waiting 49s before retry (attempt 1/4)... 

@sgoedecke sgoedecke marked this pull request as ready for review July 21, 2025 22:20
@Copilot Copilot AI review requested due to automatic review settings July 21, 2025 22:21
@sgoedecke sgoedecke requested a review from a team as a code owner July 21, 2025 22:21
Copy link
Contributor

@Copilot Copilot AI left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull Request Overview

This PR adds automatic retry functionality for rate-limited requests in the gh eval command. The implementation listens for rate limit headers and waits for the specified duration before retrying.

  • Introduces a new RateLimitError type with retry timing information
  • Adds automatic retry logic in the evaluation command with configurable wait times
  • Refactors model calling logic to support retry functionality

Reviewed Changes

Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.

File Description
internal/azuremodels/azure_client.go Adds rate limit error handling and RateLimitError type definition
internal/azuremodels/rate_limit_test.go Comprehensive test coverage for rate limit error handling logic
cmd/eval/eval.go Implements retry logic with automatic waiting and refactors model calling

@sgoedecke sgoedecke force-pushed the sgoedecke/eval-retries branch from ff4d73e to e618ed9 Compare July 21, 2025 22:27
@sgoedecke sgoedecke requested a review from jalafel July 21, 2025 22:37
@sgoedecke sgoedecke force-pushed the sgoedecke/eval-retries branch from b7ae777 to ffabf58 Compare July 21, 2025 22:40
@sgoedecke sgoedecke merged commit b21bd7a into main Jul 21, 2025
5 checks passed
@sgoedecke sgoedecke deleted the sgoedecke/eval-retries branch July 21, 2025 22:50
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

2 participants