-
Notifications
You must be signed in to change notification settings - Fork 14
Add a --json mode to evals #55
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Conversation
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Pull Request Overview
Adds a new machine-readable output mode (via --json
) to the gh models eval
command and refactors chat‐completion helpers into the prompt package.
- Introduce
--json
flag to emit structured JSON summary instead of human output. - Extract
GetAzureChatMessageRole
andBuildChatCompletionOptions
intopkg/prompt/prompt.go
for reuse. - Update CLI handlers, tests, and documentation to support JSON mode.
Reviewed Changes
Copilot reviewed 5 out of 5 changed files in this pull request and generated no comments.
Show a summary per file
File | Description |
---|---|
pkg/prompt/prompt.go | Added role‐parsing and option‐building helpers; missing import needed. |
cmd/run/run.go | Refactored message‐role switch to use new prompt helper. |
cmd/eval/eval.go | Implemented --json flag, JSON serialization, and refactored model call. |
cmd/eval/eval_test.go | Added tests covering JSON output format and CLI behavior. |
README.md | Documented gh models eval usage, including JSON flag. |
Comments suppressed due to low confidence (1)
pkg/prompt/prompt.go:9
- The new
GetAzureChatMessageRole
function usesfmt.Errorf
butfmt
is not imported; addimport "fmt"
to the import block.
github.com/github/gh-models/internal/azuremodels
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Nice job on the json support 🚢
} | ||
|
||
// BuildChatCompletionOptions creates a ChatCompletionOptions with the file's model and parameters | ||
func (f *File) BuildChatCompletionOptions(messages []azuremodels.ChatMessage) azuremodels.ChatCompletionOptions { |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
Thanks for the new functions!
Adds a --json flag to
gh models eval
so it can output machine-readable results:Also adopts some refactors suggested in #54 (extracting some functions and adding a docs link)