Provide a cleaner progress bar when running evaluators in parallel #38
Conversation
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
There was a problem hiding this comment.
Pull Request Overview
This PR aims to improve the clarity of progress reporting when evaluators run in parallel by introducing a simple tqdm position registry and integrating it across multiple evaluator modules.
- Added TqdmPositionRegistry to manage progress bar positions.
- Updated trajectory and RAG evaluators to use the registry for clearer progress display.
- Wrapped workflow execution with progress bar updates for overall evaluation tasks.
Reviewed Changes
Copilot reviewed 4 out of 4 changed files in this pull request and generated 1 comment.
| File | Description |
|---|---|
| src/aiq/eval/utils/tqdm_position_registry.py | Introduces a registry to claim/release tqdm positions to avoid clashes. |
| src/aiq/eval/trajectory_evaluator/evaluate.py | Uses the registry with a dedicated progress bar for trajectory evaluation. |
| src/aiq/eval/rag_evaluator/evaluate.py | Integrates a dynamically positioned progress bar with registry support. |
| src/aiq/eval/evaluate.py | Adds progress updates during workflow execution by wrapping evaluation calls. |
Comments suppressed due to low confidence (1)
src/aiq/eval/trajectory_evaluator/evaluate.py:66
- [nitpick] Consider revising the docstring in 'process_item' (lines 67-71) to remove the dangling hyphen after 'tuple of' for improved clarity.
async def process_item(item: EvalInputItem) -> tuple[float, dict]:
yczhang-nv
approved these changes
Mar 31, 2025
Maintain a simple position registry to separate and clearly display each evaluators progress. This is particularly needed if the dataset is large. Signed-off-by: Anuradha Karuppiah <anuradhak@nvidia.com>
Signed-off-by: Anuradha Karuppiah <anuradhak@nvidia.com>
Signed-off-by: Anuradha Karuppiah <anuradhak@nvidia.com>
AnuradhaKaruppiah
force-pushed
the
tqdm-fixes
branch
from
89bc190 to
9e5f2c8
Compare
AnuradhaKaruppiah
added
enhancement
feature request
To use the position registry Signed-off-by: Anuradha Karuppiah <anuradhak@nvidia.com>
|
/merge |
ericevans-nv
pushed a commit
to ericevans-nv/agent-iq
that referenced
this pull request
Apr 14, 2025
…VIDIA#38) This PR aims to improve the clarity of progress reporting when evaluators run in parallel by introducing a simple tqdm position registry and integrating it across multiple evaluator modules. Added TqdmPositionRegistry to manage progress bar positions. Updated trajectory and RAG evaluators to use the registry for clearer progress display. Wrapped workflow execution with progress bar updates for overall evaluation tasks. ``` Evaluating Ragas nv_context_relevance: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 6/6 [00:11<00:00, 1.85s/it] Evaluating Ragas nv_response_groundedness: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 6/6 [00:12<00:00, 2.10s/it] Evaluating Ragas nv_accuracy: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 6/6 [00:13<00:00, 2.19s/it] Evaluating Trajectory: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 6/6 [00:12<00:00, 2.05s/it] ``` Authors: - Anuradha Karuppiah (https://github.com/AnuradhaKaruppiah) Approvers: - Yuchen Zhang (https://github.com/yczhang-nv) URL: NVIDIA#38
yczhang-nv
pushed a commit
to yczhang-nv/NeMo-Agent-Toolkit
that referenced
this pull request
Apr 21, 2025
…VIDIA#38) This PR aims to improve the clarity of progress reporting when evaluators run in parallel by introducing a simple tqdm position registry and integrating it across multiple evaluator modules. Added TqdmPositionRegistry to manage progress bar positions. Updated trajectory and RAG evaluators to use the registry for clearer progress display. Wrapped workflow execution with progress bar updates for overall evaluation tasks. ``` Evaluating Ragas nv_context_relevance: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 6/6 [00:11<00:00, 1.85s/it] Evaluating Ragas nv_response_groundedness: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 6/6 [00:12<00:00, 2.10s/it] Evaluating Ragas nv_accuracy: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 6/6 [00:13<00:00, 2.19s/it] Evaluating Trajectory: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 6/6 [00:12<00:00, 2.05s/it] ``` Authors: - Anuradha Karuppiah (https://github.com/AnuradhaKaruppiah) Approvers: - Yuchen Zhang (https://github.com/yczhang-nv) URL: NVIDIA#38 Signed-off-by: Yuchen Zhang <134643420+yczhang-nv@users.noreply.github.com>
yczhang-nv
pushed a commit
to yczhang-nv/NeMo-Agent-Toolkit
that referenced
this pull request
May 8, 2025
…VIDIA#38) This PR aims to improve the clarity of progress reporting when evaluators run in parallel by introducing a simple tqdm position registry and integrating it across multiple evaluator modules. Added TqdmPositionRegistry to manage progress bar positions. Updated trajectory and RAG evaluators to use the registry for clearer progress display. Wrapped workflow execution with progress bar updates for overall evaluation tasks. ``` Evaluating Ragas nv_context_relevance: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 6/6 [00:11<00:00, 1.85s/it] Evaluating Ragas nv_response_groundedness: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 6/6 [00:12<00:00, 2.10s/it] Evaluating Ragas nv_accuracy: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 6/6 [00:13<00:00, 2.19s/it] Evaluating Trajectory: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 6/6 [00:12<00:00, 2.05s/it] ``` Authors: - Anuradha Karuppiah (https://github.com/AnuradhaKaruppiah) Approvers: - Yuchen Zhang (https://github.com/yczhang-nv) URL: NVIDIA#38 Signed-off-by: Yuchen Zhang <134643420+yczhang-nv@users.noreply.github.com>
AnuradhaKaruppiah
added a commit
to AnuradhaKaruppiah/oss-agentiq
that referenced
this pull request
Aug 4, 2025
…VIDIA#38) This PR aims to improve the clarity of progress reporting when evaluators run in parallel by introducing a simple tqdm position registry and integrating it across multiple evaluator modules. Added TqdmPositionRegistry to manage progress bar positions. Updated trajectory and RAG evaluators to use the registry for clearer progress display. Wrapped workflow execution with progress bar updates for overall evaluation tasks. ``` Evaluating Ragas nv_context_relevance: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 6/6 [00:11<00:00, 1.85s/it] Evaluating Ragas nv_response_groundedness: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 6/6 [00:12<00:00, 2.10s/it] Evaluating Ragas nv_accuracy: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 6/6 [00:13<00:00, 2.19s/it] Evaluating Trajectory: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 6/6 [00:12<00:00, 2.05s/it] ``` Authors: - Anuradha Karuppiah (https://github.com/AnuradhaKaruppiah) Approvers: - Yuchen Zhang (https://github.com/yczhang-nv) URL: NVIDIA#38
scheckerNV
pushed a commit
to scheckerNV/aiq-factory-reset
that referenced
this pull request
Aug 22, 2025
…VIDIA#38) This PR aims to improve the clarity of progress reporting when evaluators run in parallel by introducing a simple tqdm position registry and integrating it across multiple evaluator modules. Added TqdmPositionRegistry to manage progress bar positions. Updated trajectory and RAG evaluators to use the registry for clearer progress display. Wrapped workflow execution with progress bar updates for overall evaluation tasks. ``` Evaluating Ragas nv_context_relevance: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 6/6 [00:11<00:00, 1.85s/it] Evaluating Ragas nv_response_groundedness: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 6/6 [00:12<00:00, 2.10s/it] Evaluating Ragas nv_accuracy: 100%|████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 6/6 [00:13<00:00, 2.19s/it] Evaluating Trajectory: 100%|███████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████████| 6/6 [00:12<00:00, 2.05s/it] ``` Authors: - Anuradha Karuppiah (https://github.com/AnuradhaKaruppiah) Approvers: - Yuchen Zhang (https://github.com/yczhang-nv) URL: NVIDIA#38
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
This PR aims to improve the clarity of progress reporting when evaluators run in parallel by introducing a simple tqdm position registry and integrating it across multiple evaluator modules.
Added TqdmPositionRegistry to manage progress bar positions.
Updated trajectory and RAG evaluators to use the registry for clearer progress display.
Wrapped workflow execution with progress bar updates for overall evaluation tasks.