Improving object detection benchmarking process for unrelated datasets

[patel-zeel]

Search before asking

• I have searched the Supervision issues and found no similar feature requests.

Description

Context

If a model is trained on a dataset, benchmarking with supervision's evaluation API is straightforward.

However, if the dataset was not used to train a model (unrelated dataset), one should follow the "how-to" guide on Benchmarking a model.

What if we can improve this with a simpler API?

Use case

In cases where the dataset was not used to train the model, people would need to: 1) remap the class names to match with the dataset; 2) transform the class IDs to match with the dataset. Here is what a boilerplate code would look like:

class_mapping = {"dog": "animal", "cat": "animal", "eagle": "bird"}
for _, image, target in dataset:
    result = model.infer(image)[0]
    detection = sv.Detections.from_inference(result)
    
    # check if all mapped values are within the dataset classes
    if not all([value in dataset.classes for value in class_mapping.values()]):
        raise ValueError("All mapped values must be in dataset classes")
    
    # remap class names
    detection['class_name'] = list(map(lambda name: class_mapping[name] if name in class_maping else name, detection['class_name']))

    # remove predicted classes not in the dataset
    detection = detection[np.isin(detection['class_name'], dataset.classes)]

    # remap Class IDs based on Class names
    detection.class_id = np.array([dataset.classes.index(name) for name in detection['class_name']])

Proposed API for the above transformation:

class_mapping = {"dog": "animal", "cat": "animal", "eagle": "bird"}
for _, image, target in dataset:
    result = model.infer(image)[0]
    detection = sv.Detections.from_inference(result)
    detection = detection.transform(dataset, class_mapping=class_mapping)

Where the transform method implements the same logic described in the boilerplate. Note that class_mapping is an optional argument for the cases where class names are the same between a model and a dataset.

Additional

No response

Are you willing to submit a PR?

• Yes I'd like to help by submitting a PR!

[Muhammedswalihu]

Hi! I’d like to work on this enhancement.

I have experience building benchmarking tools for ML pipelines and can help extend the supervision toolkit to support robust object detection evaluations across unrelated or non-standardized datasets.

To start, I’ll:

• Review the current metrics implementation
• Draft a design for a benchmarking wrapper that supports label schema alignment (e.g., class mapping, flexible IOU)
• Include a notebook for testing and visualization

Let me know if there are any preferences or edge cases I should be aware of before submitting a PR. Thanks!