Experimental Features

kolena._experimental.quality_standard

download_quality_standard_result(dataset, models, metric_groups=None, intersect_results=True, confidence_level=None, reference_model=None, reference_eval_config='first')

Download quality standard result given a dataset and list of models.

Parameters:

Name	Type	Description	Default
dataset	str	The name of the dataset.	required
models	List[str]	The names of the models.	required
metric_groups	Union[List[str], None]	The names of the metric groups to include in the result.	None
intersect_results	bool	If True, only include datapoint that are common to all models in the metrics calculation. Note all metric groups are included when this value is None.	True
confidence_level	Optional[float]	The confidence score used to calculate the Margin of Error, representing the probability of capturing the true population parameter within the calculated MOE, we recommend setting between 0.9 - 0.99. If this is specified performance delta column will be shown, metric valued will be classified as improved, regressed, similar or unknown according to the MOE.	None
reference_model	Optional[str]	The name of the model to use as a reference for the Margin of Error calculation. This should be one of the models provided in models. If unspecified, the first model of models will be used.	None
reference_eval_config	Optional[Union[Dict[str, Any], Literal['first']]]	The evaluation configuration to use in conjunction with the reference model, if unspecified the first evaluation configuration of the reference model will be used	'first'

Returns:

Type	Description
DataFrame	A Dataframe containing the quality standard result.

copy_quality_standards_from_dataset(dataset, source_dataset, include_metric_groups=True, include_test_cases=True)

Create a quality standard on a dataset by copying from a source dataset. Note that this operation will overwrite the existing quality standards on the dataset if they exist.

Parameters:

Name	Type	Description	Default
dataset	str	The name of the dataset.	required
source_dataset	str	The name of the dataset from which the quality standards should be copied.	required
include_metric_groups	bool	Optional flag to indicate whether to copy the metric groups from the source dataset.	True
include_test_cases	bool	Optional flag to indicate whether to copy the test cases from the source dataset.	True

Returns:

Type	Description
Tuple[List[Dict[str, Any]], List[Dict[str, Any]]]	A tuple of the created metric groups and test cases.

kolena._experimental.search

upload_dataset_embeddings(dataset_name, key, df_embedding)

Upload a list of search embeddings for a dataset.

Parameters:

Name	Type	Description	Default
dataset_name	str	String value indicating the name of the dataset for which the embeddings will be uploaded.	required
key	str	String value uniquely corresponding to the embedding vectors. For example, this can be the name of the embedding model along with the column with which the embedding was extracted, such as resnet50-image_locator.	required
df_embedding	DataFrame	Dataframe containing id fields for identifying datapoints in the dataset and the associated embeddings as numpy.typing.ArrayLike of numeric values.	required

Raises:

Type	Description
NotFoundError	The given dataset does not exist.
InputValidationError	The provided input is not valid.

kolena._experimental.object_detection

compute_object_detection_results(dataset_name, df, *, ground_truths_field='ground_truths', raw_inferences_field='raw_inferences', gt_ignore_property=None, iou_threshold=0.5, threshold_strategy='F1-Optimal', min_confidence_score=0.01, batch_size=10000)

Compute metrics of the model for the dataset.

Dataframe df should include a locator column that would match to that of corresponding datapoint and an inference column that should be a list of scored BoundingBoxes.

Parameters:

Name	Type	Description	Default
dataset_name	str	Dataset name.	required
df	DataFrame	Dataframe for model results.	required
ground_truths_field	str	Field name in datapoint with ground truth bounding boxes, defaulting to "ground_truths".	'ground_truths'
raw_inferences_field	str	Column in model result DataFrame with raw inference bounding boxes, defaulting to "raw_inferences".	'raw_inferences'
gt_ignore_property	Optional[str]	Field on the ground truth bounding boxes used to determine if the bounding box should be ignored. Bounding boxes will be ignored if this field exists and is equal to True.	None
iou_threshold	float	The IoU ↗ threshold, defaulting to 0.5.	0.5
threshold_strategy	Union[Literal['F1-Optimal'], float, Dict[str, float]]	The confidence threshold strategy. It can either be a fixed confidence threshold such as 0.5 or 0.75, or "F1-Optimal" to find the threshold maximizing F1 score.	'F1-Optimal'
min_confidence_score	float	The minimum confidence score to consider for the evaluation. This is usually set to reduce noise by excluding inferences with low confidence score.	0.01
batch_size	int	number of results to process per iteration.	10000

Returns:

Type	Description
DataFrame	A DataFrame of the computed results

upload_object_detection_results(dataset_name, model_name, df, *, ground_truths_field='ground_truths', raw_inferences_field='raw_inferences', gt_ignore_property=None, iou_threshold=0.5, threshold_strategy='F1-Optimal', min_confidence_score=0.01, batch_size=10000, required_match_fields=None)

Compute metrics and upload results of the model computed by compute_object_detection_results for the dataset.

Dataframe df should include a locator column that would match to that of corresponding datapoint and an inference column that should be a list of scored BoundingBoxes.

Parameters:

Name	Type	Description	Default
dataset_name	str	Dataset name.	required
model_name	str	Model name.	required
df	DataFrame	Dataframe for model results.	required
ground_truths_field	str	Field name in datapoint with ground truth bounding boxes, defaulting to "ground_truths".	'ground_truths'
raw_inferences_field	str	Column in model result DataFrame with raw inference bounding boxes, defaulting to "raw_inferences".	'raw_inferences'
gt_ignore_property	Optional[str]	Name of a property on the ground truth bounding boxes used to determine if the bounding box should be ignored. Bounding boxes will be ignored if this property exists and is equal to True.	None
iou_threshold	float	The IoU ↗ threshold, defaulting to 0.5.	0.5
threshold_strategy	Union[Literal['F1-Optimal'], float, Dict[str, float]]	The confidence threshold strategy. It can either be a fixed confidence threshold such as 0.5 or 0.75, or "F1-Optimal" to find the threshold maximizing F1 score.	'F1-Optimal'
min_confidence_score	float	The minimum confidence score to consider for the evaluation. This is usually set to reduce noise by excluding inferences with low confidence score.	0.01
batch_size	int	number of results to process per iteration.	10000
required_match_fields	Optional[List[str]]	Optionally specify a list of fields that must match between the inference and ground truth for them to be considered a match.	None

Returns:

Type	Description
None

kolena._experimental.instance_segmentation

upload_instance_segmentation_results(dataset_name, model_name, df, *, ground_truths_field='ground_truths', raw_inferences_field='raw_inferences', iou_threshold=0.5, threshold_strategy='F1-Optimal', min_confidence_score=0.01, batch_size=10000)

Compute metrics and upload results of the model for the dataset.

Dataframe df should include a locator column that would match to that of corresponding datapoint and an inference column that should be a list of scored Polygons.

Parameters:

Name	Type	Description	Default
dataset_name	str	Dataset name.	required
model_name	str	Model name.	required
df	DataFrame	Dataframe for model results.	required
ground_truths_field	str	Field name in datapoint with ground truth polygons, defaulting to "ground_truths".	'ground_truths'
raw_inferences_field	str	Column in model result DataFrame with raw inference polygons, defaulting to "raw_inferences".	'raw_inferences'
iou_threshold	float	The IoU ↗ threshold, defaulting to 0.5.	0.5
threshold_strategy	Union[Literal['F1-Optimal'], float, Dict[str, float]]	The confidence threshold strategy. It can either be a fixed confidence threshold such as 0.5 or 0.75, or "F1-Optimal" to find the threshold maximizing F1 score..	'F1-Optimal'
min_confidence_score	float	The minimum confidence score to consider for the evaluation. This is usually set to reduce noise by excluding inferences with low confidence score.	0.01

Returns:

Type	Description
None

kolena._experimental.trace

kolena_trace(func=None, *, dataset_name=None, model_name=None, model_name_field=None, sync_interval=THIRTY_SECONDS, id_fields=None, record_timestamp=True)

Use this decorator to trace the function with Kolena, the input and output of this function will be sent as datapoints and results respectively For example:

@kolena_trace(dataset_name="test_trace", id_fields=["request_id"], record_timestamp=False)
def predict(data, request_id):
    pass

OR

@kolena_trace
def predict(data, request_id):
    pass

Parameters:

Name	Type	Description	Default
func	Optional[Callable]	The function to be traced, this is auto populated when used as a decorator	None
dataset_name	Optional[str]	The name of the dataset to be created, if not provided the function name will be used	None
model_name	Optional[str]	The name of the model to be created, if not provided the function name suffixed with _model will be used	None
model_name_field	Optional[str]	The field in the input that should be used as model name, if this would override the model name	None
sync_interval	int	The interval at which the data should be synced to the server, default is 30 seconds	THIRTY_SECONDS
id_fields	Optional[List[str]]	The fields in the input that should be used as id fields, if not provided a default id field will be used	None
record_timestamp	bool	If True, the timestamp of the input, output, and time elapsed will be recorded, default is True	True

kolena._experimental.dataset

download_results_by_tag(dataset, model_tag, commit=None, include_extracted_properties=False)

Download results given dataset name and model tag. Currently restricted to if the model tag is unique to the model.

Concat dataset with results:

df_dp, results = download_results_by_tag("dataset name", "model tag")
for eval_config, df_result in results:
    df_combined = pd.concat([df_dp, df_result], axis=1)

Parameters:

Name	Type	Description	Default
dataset	str	The name of the dataset.	required
model_tag	str	The tag associated with the model.	required
commit	Optional[str]	The commit hash for version control. Get the latest commit when this value is None.	None
include_extracted_properties	bool	If True, include kolena extracted properties from automated extractions in the datapoints and results as separate columns	False

Returns:

Type	Description
Tuple[DataFrame, List[EvalConfigResults]]	Tuple of DataFrame of datapoints and list of EvalConfigResults.