Skip to content

Experimental Features#

kolena._experimental.quality_standard #

download_quality_standard_result(dataset, models, metric_groups=None) #

Download quality standard result given a dataset and list of models.

Parameters:

Name Type Description Default
dataset str

The name of the dataset.

required
models List[str]

The names of the models.

required
metric_groups Union[List[str], None]

The names of the metric groups to include in the result. Note all metric groups are included when this value is None.

None

Returns:

Type Description
DataFrame

A Dataframe containing the quality standard result.

kolena._experimental.search #

upload_dataset_embeddings(dataset_name, key, df_embedding) #

Upload a list of search embeddings for a dataset.

Parameters:

Name Type Description Default
dataset_name str

String value indicating the name of the dataset for which the embeddings will be uploaded.

required
key str

String value uniquely corresponding to the model used to extract the embedding vectors. This is typically a locator.

required
df_embedding DataFrame

Dataframe containing id fields for identifying datapoints in the dataset and the associated embeddings as numpy.typing.ArrayLike of numeric values.

required

Raises:

Type Description
NotFoundError

The given dataset does not exist.

InputValidationError

The provided input is not valid.

kolena._experimental.object_detection #

compute_object_detection_results(dataset_name, df, *, ground_truths_field='ground_truths', raw_inferences_field='raw_inferences', gt_ignore_property=None, iou_threshold=0.5, threshold_strategy='F1-Optimal', min_confidence_score=0.01, batch_size=10000) #

Compute metrics of the model for the dataset.

Dataframe df should include a locator column that would match to that of corresponding datapoint and an inference column that should be a list of scored BoundingBoxes.

Parameters:

Name Type Description Default
dataset_name str

Dataset name.

required
df DataFrame

Dataframe for model results.

required
ground_truths_field str

Field name in datapoint with ground truth bounding boxes, defaulting to "ground_truths".

'ground_truths'
raw_inferences_field str

Column in model result DataFrame with raw inference bounding boxes, defaulting to "raw_inferences".

'raw_inferences'
gt_ignore_property Optional[str]

Field on the ground truth bounding boxes used to determine if the bounding box should be ignored. Bounding boxes will be ignored if this field exists and is equal to True.

None
iou_threshold float

The IoU ↗ threshold, defaulting to 0.5.

0.5
threshold_strategy Union[Literal['F1-Optimal'], float, Dict[str, float]]

The confidence threshold strategy. It can either be a fixed confidence threshold such as 0.5 or 0.75, or "F1-Optimal" to find the threshold maximizing F1 score.

'F1-Optimal'
min_confidence_score float

The minimum confidence score to consider for the evaluation. This is usually set to reduce noise by excluding inferences with low confidence score.

0.01
batch_size int

number of results to process per iteration.

10000

Returns:

Type Description
DataFrame

A DataFrame of the computed results

upload_object_detection_results(dataset_name, model_name, df, *, ground_truths_field='ground_truths', raw_inferences_field='raw_inferences', gt_ignore_property=None, iou_threshold=0.5, threshold_strategy='F1-Optimal', min_confidence_score=0.01, batch_size=10000) #

Compute metrics and upload results of the model computed by compute_object_detection_results for the dataset.

Dataframe df should include a locator column that would match to that of corresponding datapoint and an inference column that should be a list of scored BoundingBoxes.

Parameters:

Name Type Description Default
dataset_name str

Dataset name.

required
model_name str

Model name.

required
df DataFrame

Dataframe for model results.

required
ground_truths_field str

Field name in datapoint with ground truth bounding boxes, defaulting to "ground_truths".

'ground_truths'
raw_inferences_field str

Column in model result DataFrame with raw inference bounding boxes, defaulting to "raw_inferences".

'raw_inferences'
gt_ignore_property Optional[str]

Name of a property on the ground truth bounding boxes used to determine if the bounding box should be ignored. Bounding boxes will be ignored if this property exists and is equal to True.

None
iou_threshold float

The IoU ↗ threshold, defaulting to 0.5.

0.5
threshold_strategy Union[Literal['F1-Optimal'], float, Dict[str, float]]

The confidence threshold strategy. It can either be a fixed confidence threshold such as 0.5 or 0.75, or "F1-Optimal" to find the threshold maximizing F1 score.

'F1-Optimal'
min_confidence_score float

The minimum confidence score to consider for the evaluation. This is usually set to reduce noise by excluding inferences with low confidence score.

0.01
batch_size int

number of results to process per iteration.

10000

Returns:

Type Description
None

kolena._experimental.instance_segmentation #

upload_instance_segmentation_results(dataset_name, model_name, df, *, ground_truths_field='ground_truths', raw_inferences_field='raw_inferences', iou_threshold=0.5, threshold_strategy='F1-Optimal', min_confidence_score=0.01, batch_size=10000) #

Compute metrics and upload results of the model for the dataset.

Dataframe df should include a locator column that would match to that of corresponding datapoint and an inference column that should be a list of scored Polygons.

Parameters:

Name Type Description Default
dataset_name str

Dataset name.

required
model_name str

Model name.

required
df DataFrame

Dataframe for model results.

required
ground_truths_field str

Field name in datapoint with ground truth polygons, defaulting to "ground_truths".

'ground_truths'
raw_inferences_field str

Column in model result DataFrame with raw inference polygons, defaulting to "raw_inferences".

'raw_inferences'
iou_threshold float

The IoU ↗ threshold, defaulting to 0.5.

0.5
threshold_strategy Union[Literal['F1-Optimal'], float, Dict[str, float]]

The confidence threshold strategy. It can either be a fixed confidence threshold such as 0.5 or 0.75, or "F1-Optimal" to find the threshold maximizing F1 score..

'F1-Optimal'
min_confidence_score float

The minimum confidence score to consider for the evaluation. This is usually set to reduce noise by excluding inferences with low confidence score.

0.01

Returns:

Type Description
None