Skip to content

kolena.metrics#

InferenceMatches #

Bases: Generic[GT, Inf]

The result of match_inferences, providing lists of matches between ground truth and inference objects, unmatched ground truths, and unmatched inferences. After applying some confidence threshold on returned inference objects, InferenceMatches can be used to calculate metrics such as precision and recall.

Objects are of type BoundingBox or Polygon, depending on the type of inputs provided to match_inferences.

matched: List[Tuple[GT, Inf]] instance-attribute #

Pairs of matched ground truth and inference objects above the IoU threshold, along with the calculated IoU. Considered as true positive detections after applying some confidence threshold.

unmatched_gt: List[GT] instance-attribute #

Unmatched ground truth objects. Considered as false negatives.

unmatched_inf: List[Inf] instance-attribute #

Unmatched inference objects, along with the maximum IoU over all ground truths. Considered as false positives after applying some confidence threshold.

MulticlassInferenceMatches #

Bases: Generic[GT, Inf]

The result of match_inferences_multiclass, providing lists of matches between ground truth and inference objects, unmatched ground truths, and unmatched inferences.

Unmatched ground truths may be matched with an inference of a different class when no inference of its own class is suitable, i.e. a "confused" match. MultiClassInferenceMatches can be used to calculate metrics such as precision and recall per class, after applying some confidence threshold on the returned inference objects.

Objects are of type LabeledBoundingBox or LabeledPolygon, depending on the type of inputs provided to match_inferences_multiclass.

matched: List[Tuple[GT, Inf]] instance-attribute #

Pairs of matched ground truth and inference objects above the IoU threshold, along with the calculated IoU. Considered as true positive detections after applying some confidence threshold.

unmatched_gt: List[Tuple[GT, Optional[Inf]]] instance-attribute #

Pairs of unmatched ground truth objects with its confused inference object (i.e. IoU above threshold with mismatching label) and calculated IoU, if such an inference exists. Considered as false negatives and "confused" detections.

unmatched_inf: List[Inf] instance-attribute #

Unmatched inference objects, along with the maximum IoU over all ground truths. Considered as false positives after applying some confidence threshold.

accuracy(true_positives, false_positives, false_negatives, true_negatives) #

Accuracy represents the proportion of inferences that are correct (including both positives and negatives).

\[ \text{Accuracy} = \frac{\text{# TP} + \text{# TN}} {\text{# TP} + \text{# FP} + \text{# FN} + \text{# TN}} \]

Parameters:

Name Type Description Default
true_positives int

Number of true positive inferences.

required
false_positives int

Number of false positive inferences.

required
false_negatives int

Number of false negatives.

required
true_negatives int

Number of true negatives.

required

f1_score(true_positives, false_positives, false_negatives) #

F1-score is the harmonic mean between precision and recall.

\[ \begin{align} \text{F1} &= \frac{2}{\frac{1}{\text{Precision}} + \frac{1}{\text{Recall}}} \\[1em] &= 2 \times \frac{\text{Precision} \times \text{Recall}}{\text{Precision} + \text{Recall}} \end{align} \]

Parameters:

Name Type Description Default
true_positives int

Number of true positive inferences.

required
false_positives int

Number of false positive inferences.

required
false_negatives int

Number of false negatives.

required

fpr(true_negatives, false_positives) #

False positive rate represents the proportion of negative ground truths that were incorrectly predicted as positive by the model.

\[ \text{FPR} = \frac{\text{# False Positives}}{\text{# False Positives} + \text{# True Negatives}} \]

Parameters:

Name Type Description Default
true_negatives int

Number of true negatives.

required
false_positives int

Number of false positives.

required

precision(true_positives, false_positives) #

Precision represents the proportion of inferences that are correct.

\[ \text{Precision} = \frac{\text{# True Positives}}{\text{# True Positives} + \text{# False Positives}} \]

Parameters:

Name Type Description Default
true_positives int

Number of true positive inferences.

required
false_positives int

Number of false positive inferences.

required

recall(true_positives, false_negatives) #

Recall (TPR or sensitivity) represents the proportion of ground truths that were successfully predicted.

\[ \text{Recall} = \frac{\text{# True Positives}}{\text{# True Positives} + \text{# False Negatives}} \]

Parameters:

Name Type Description Default
true_positives int

Number of true positive inferences.

required
false_negatives int

Number of false negatives.

required

specificity(true_negatives, false_positives) #

Specificity (TNR) represents the proportion of negative ground truths that were correctly predicted.

\[ \text{Specificity} = \frac{\text{# True Negatives}}{\text{# True Negatives} + \text{# False Positives}} \]

Parameters:

Name Type Description Default
true_negatives int

Number of true negatives.

required
false_positives int

Number of false positives.

required

iou(a, b) #

Compute the Intersection Over Union (IoU) of two geometries.

Parameters:

Name Type Description Default
a Union[BoundingBox, Polygon]

The first geometry in computation.

required
b Union[BoundingBox, Polygon]

The second geometry in computation.

required

Returns:

Type Description
float

The value of the IoU between geometries a and b.

match_inferences(ground_truths, inferences, *, ignored_ground_truths=None, mode='pascal', iou_threshold=0.5, required_match_fields=None) #

Matches model inferences with annotated ground truths using the provided configuration.

This matcher does not consider labels, which is appropriate for single class object matching. To match with multiple classes (i.e. heeding label classifications), use the multiclass matcher match_inferences_multiclass.

Available modes:

  • pascal (PASCAL VOC): For every inference by order of highest confidence, the ground truth of highest IoU is its match. Multiple inferences are able to match with the same ignored ground truth. See the PASCAL VOC paper for more information.

Parameters:

Name Type Description Default
ground_truths List[Geometry]

A list of BoundingBox or Polygon ground truths.

required
inferences List[ScoredGeometry]

A list of ScoredBoundingBox or ScoredPolygon inferences.

required
ignored_ground_truths Optional[List[Geometry]]

Optionally specify a list of BoundingBox or Polygon ground truths to ignore. These ignored ground truths and any inferences matched with them are omitted from the returned InferenceMatches.

None
mode Literal['pascal']

The matching methodology to use. See available modes above.

'pascal'
iou_threshold float

The IoU (intersection over union, see iou) threshold for valid matches.

0.5
required_match_fields Optional[List[str]]

Optionally specify a list of fields that must match between the inference and ground truth for them to be considered a match.

None

Returns:

Type Description
InferenceMatches[GT, Inf]

InferenceMatches containing the matches (true positives), unmatched ground truths (false negatives) and unmatched inferences (false positives).

match_inferences_multiclass(ground_truths, inferences, *, ignored_ground_truths=None, mode='pascal', iou_threshold=0.5, required_match_fields=None) #

Matches model inferences with annotated ground truths using the provided configuration.

This matcher considers label values matching per class. After matching inferences and ground truths with equivalent label values, unmatched inferences and unmatched ground truths are matched once more to identify confused matches, where localization succeeded (i.e. IoU above iou_threshold) but classification failed (i.e. mismatching label values).

Available modes:

  • pascal (PASCAL VOC): For every inference by order of highest confidence, the ground truth of highest IoU is its match. Multiple inferences are able to match with the same ignored ground truth. See the PASCAL VOC paper for more information.

Parameters:

Name Type Description Default
ground_truths List[LabeledGeometry]

A list of LabeledBoundingBox or LabeledPolygon ground truths.

required
inferences List[ScoredLabeledGeometry] required
ignored_ground_truths Optional[List[LabeledGeometry]]

Optionally specify a list of LabeledBoundingBox or LabeledPolygon ground truths to ignore. These ignored ground truths and any inferences matched with them are omitted from the returned MulticlassInferenceMatches.

None
mode Literal['pascal']

The matching methodology to use. See available modes above.

'pascal'
iou_threshold float

The IoU threshold cutoff for valid matches.

0.5
required_match_fields Optional[List[str]]

Optionally specify a list of fields that must match between the inference and ground truth for them to be considered a match.

None

Returns:

Type Description
MulticlassInferenceMatches[GT, Inf]

MulticlassInferenceMatches containing the matches (true positives), unmatched ground truths (false negatives), and unmatched inferences (false positives).