kolena.metrics

InferenceMatches

Bases: Generic[GT, Inf]

The result of match_inferences, providing lists of matches between ground truth and inference objects, unmatched ground truths, and unmatched inferences. After applying some confidence threshold on returned inference objects, InferenceMatches can be used to calculate metrics such as precision and recall.

Objects are of type BoundingBox or Polygon, depending on the type of inputs provided to match_inferences.

matched: List[Tuple[GT, Inf]] instance-attribute

Pairs of matched ground truth and inference objects above the IoU threshold, along with the calculated IoU. Considered as true positive detections after applying some confidence threshold.

unmatched_gt: List[GT] instance-attribute

Unmatched ground truth objects. Considered as false negatives.

unmatched_inf: List[Inf] instance-attribute

Unmatched inference objects, along with the maximum IoU over all ground truths. Considered as false positives after applying some confidence threshold.

MulticlassInferenceMatches

Bases: Generic[GT, Inf]

The result of match_inferences_multiclass, providing lists of matches between ground truth and inference objects, unmatched ground truths, and unmatched inferences.

Unmatched ground truths may be matched with an inference of a different class when no inference of its own class is suitable, i.e. a "confused" match. MultiClassInferenceMatches can be used to calculate metrics such as precision and recall per class, after applying some confidence threshold on the returned inference objects.

Objects are of type LabeledBoundingBox or LabeledPolygon, depending on the type of inputs provided to match_inferences_multiclass.

matched: List[Tuple[GT, Inf]] instance-attribute

Pairs of matched ground truth and inference objects above the IoU threshold, along with the calculated IoU. Considered as true positive detections after applying some confidence threshold.

unmatched_gt: List[Tuple[GT, Optional[Inf]]] instance-attribute

Pairs of unmatched ground truth objects with its confused inference object (i.e. IoU above threshold with mismatching label) and calculated IoU, if such an inference exists. Considered as false negatives and "confused" detections.

unmatched_inf: List[Inf] instance-attribute

Unmatched inference objects, along with the maximum IoU over all ground truths. Considered as false positives after applying some confidence threshold.

accuracy(true_positives, false_positives, false_negatives, true_negatives)

Accuracy represents the proportion of inferences that are correct (including both positives and negatives).

\[ \text{Accuracy} = \frac{\text{# TP} + \text{# TN}} {\text{# TP} + \text{# FP} + \text{# FN} + \text{# TN}} \]

• Metrics Glossary: Accuracy ↗

Parameters:

Name	Type	Description	Default
true_positives	int	Number of true positive inferences.	required
false_positives	int	Number of false positive inferences.	required
false_negatives	int	Number of false negatives.	required
true_negatives	int	Number of true negatives.	required

f1_score(true_positives, false_positives, false_negatives)

F₁-score is the harmonic mean between precision and recall.

\[ \begin{align} \text{F1} &= \frac{2}{\frac{1}{\text{Precision}} + \frac{1}{\text{Recall}}} \\[1em] &= 2 \times \frac{\text{Precision} \times \text{Recall}}{\text{Precision} + \text{Recall}} \end{align} \]

• Metrics Glossary: F₁-score ↗

Parameters:

Name	Type	Description	Default
true_positives	int	Number of true positive inferences.	required
false_positives	int	Number of false positive inferences.	required
false_negatives	int	Number of false negatives.	required

fpr(true_negatives, false_positives)

False positive rate represents the proportion of negative ground truths that were incorrectly predicted as positive by the model.

\[ \text{FPR} = \frac{\text{# False Positives}}{\text{# False Positives} + \text{# True Negatives}} \]

• Metrics Glossary: False Positive Rate ↗

Parameters:

Name	Type	Description	Default
true_negatives	int	Number of true negatives.	required
false_positives	int	Number of false positives.	required

precision(true_positives, false_positives)

Precision represents the proportion of inferences that are correct.

\[ \text{Precision} = \frac{\text{# True Positives}}{\text{# True Positives} + \text{# False Positives}} \]

• Metrics Glossary: Precision ↗

Parameters:

Name	Type	Description	Default
true_positives	int	Number of true positive inferences.	required
false_positives	int	Number of false positive inferences.	required

recall(true_positives, false_negatives)

Recall (TPR or sensitivity) represents the proportion of ground truths that were successfully predicted.

\[ \text{Recall} = \frac{\text{# True Positives}}{\text{# True Positives} + \text{# False Negatives}} \]

• Metrics Glossary: Recall ↗

Parameters:

Name	Type	Description	Default
true_positives	int	Number of true positive inferences.	required
false_negatives	int	Number of false negatives.	required

specificity(true_negatives, false_positives)

Specificity (TNR) represents the proportion of negative ground truths that were correctly predicted.

\[ \text{Specificity} = \frac{\text{# True Negatives}}{\text{# True Negatives} + \text{# False Positives}} \]

• Metrics Glossary: Specificity ↗

Parameters:

Name	Type	Description	Default
true_negatives	int	Number of true negatives.	required
false_positives	int	Number of false positives.	required

iou(a, b)

Compute the Intersection Over Union (IoU) of two geometries.

• Metrics Glossary: Intersection over Union (IoU) ↗

Parameters:

Name	Type	Description	Default
a	Union[BoundingBox, Polygon]	The first geometry in computation.	required
b	Union[BoundingBox, Polygon]	The second geometry in computation.	required

Returns:

Type	Description
float	The value of the IoU between geometries a and b.

match_inferences(ground_truths, inferences, *, ignored_ground_truths=None, mode='pascal', iou_threshold=0.5, required_match_fields=None)

Matches model inferences with annotated ground truths using the provided configuration.

This matcher does not consider labels, which is appropriate for single class object matching. To match with multiple classes (i.e. heeding label classifications), use the multiclass matcher match_inferences_multiclass.

Available modes:

• pascal (PASCAL VOC): For every inference by order of highest confidence, the ground truth of highest IoU is its match. Multiple inferences are able to match with the same ignored ground truth. See the PASCAL VOC paper for more information.

• Metrics Glossary: Geometry Matching ↗

Parameters:

Name	Type	Description	Default
ground_truths	List[Geometry]	A list of BoundingBox or Polygon ground truths.	required
inferences	List[ScoredGeometry]	A list of ScoredBoundingBox or ScoredPolygon inferences.	required
ignored_ground_truths	Optional[List[Geometry]]	Optionally specify a list of BoundingBox or Polygon ground truths to ignore. These ignored ground truths and any inferences matched with them are omitted from the returned InferenceMatches.	None
mode	Literal['pascal']	The matching methodology to use. See available modes above.	'pascal'
iou_threshold	float	The IoU (intersection over union, see iou) threshold for valid matches.	0.5
required_match_fields	Optional[List[str]]	Optionally specify a list of fields that must match between the inference and ground truth for them to be considered a match.	None

Returns:

Type	Description
InferenceMatches[GT, Inf]	InferenceMatches containing the matches (true positives), unmatched ground truths (false negatives) and unmatched inferences (false positives).

match_inferences_multiclass(ground_truths, inferences, *, ignored_ground_truths=None, mode='pascal', iou_threshold=0.5, required_match_fields=None)

Matches model inferences with annotated ground truths using the provided configuration.

This matcher considers label values matching per class. After matching inferences and ground truths with equivalent label values, unmatched inferences and unmatched ground truths are matched once more to identify confused matches, where localization succeeded (i.e. IoU above iou_threshold) but classification failed (i.e. mismatching label values).

Available modes:

• pascal (PASCAL VOC): For every inference by order of highest confidence, the ground truth of highest IoU is its match. Multiple inferences are able to match with the same ignored ground truth. See the PASCAL VOC paper for more information.

• Metrics Glossary: Geometry Matching ↗

Parameters:

Name	Type	Description	Default
ground_truths	List[LabeledGeometry]	A list of LabeledBoundingBox or LabeledPolygon ground truths.	required
inferences	List[ScoredLabeledGeometry]	A list of ScoredLabeledBoundingBox or ScoredLabeledPolygon inferences.	required
ignored_ground_truths	Optional[List[LabeledGeometry]]	Optionally specify a list of LabeledBoundingBox or LabeledPolygon ground truths to ignore. These ignored ground truths and any inferences matched with them are omitted from the returned MulticlassInferenceMatches.	None
mode	Literal['pascal']	The matching methodology to use. See available modes above.	'pascal'
iou_threshold	float	The IoU threshold cutoff for valid matches.	0.5
required_match_fields	Optional[List[str]]	Optionally specify a list of fields that must match between the inference and ground truth for them to be considered a match.	None

Returns:

Type	Description
MulticlassInferenceMatches[GT, Inf]	MulticlassInferenceMatches containing the matches (true positives), unmatched ground truths (false negatives), and unmatched inferences (false positives).