Skip to content

kolena.workflow.TestCase#

TestCase(name, version=None, description=None, test_samples=None, reset=False) #

Bases: Frozen

A test case holds a list of test samples paired with ground truths representing a testing dataset or a slice of a testing dataset.

Rather than importing this class directly, use the TestCase type definition returned from define_workflow.

workflow: Workflow instance-attribute #

The workflow of this test case. Automatically populated when constructing via test case type returned from define_workflow.

name: str instance-attribute #

The unique name of this test case. Cannot be changed after creation.

version: int instance-attribute #

The version of this test case. A test case's version is automatically incremented whenever it is edited via TestCase.edit.

description: str instance-attribute #

Free-form, human-readable description of this test case. Can be edited at any time via TestCase.edit.

Editor(description, reset) #

description(description) #

Update the description of the test case.

add(test_sample, ground_truth) #

Add a test sample to the test case. When the test sample already exists in the test case, its ground truth is overwritten with the ground truth provided here.

Parameters:

Name Type Description Default
test_sample TestSample

The test sample to add.

required
ground_truth GroundTruth

The ground truth for the test sample.

required

remove(test_sample) #

Remove a test sample from the test case. Does nothing if the test sample is not in the test case.

Parameters:

Name Type Description Default
test_sample TestSample

The test sample to remove.

required

create(name, description=None, test_samples=None) classmethod #

Create a new test case with the provided name.

Parameters:

Name Type Description Default
name str

The name of the new test case to create.

required
description Optional[str]

Optional free-form description of the test case to create.

None
test_samples Optional[List[Tuple[TestSample, GroundTruth]]]

Optionally specify a set of test samples and ground truths to populate the test case.

None

Returns:

Type Description
TestCase

The newly created test case.

load(name, version=None) classmethod #

Load an existing test case with the provided name.

Parameters:

Name Type Description Default
name str

The name of the test case to load.

required
version Optional[int]

Optionally specify a particular version of the test case to load. Defaults to the latest version when unset.

None

Returns:

Type Description
TestCase

The loaded test case.

load_test_samples() #

Load all TestSamples and GroundTruths contained in this test case.

Returns:

Type Description
List[Tuple[TestSample, GroundTruth]]

A list of each test sample, paired with its ground truth, in this test case.

iter_test_samples() #

Iterate through all TestSamples and GroundTruths contained in this test case.

Returns:

Type Description
Iterator[Tuple[TestSample, GroundTruth]]

An iterator yielding each test sample, paired with its ground truth, in this test case.

edit(reset=False) #

Edit this test case in a context:

with test_case.edit() as editor:
    # perform as many editing actions as desired
    editor.add(...)
    editor.remove(...)

Changes are committed to the Kolena platform when the context is exited.

Parameters:

Name Type Description Default
reset bool

Clear all existing test samples in the test case.

False

init_many(data, reset=False) classmethod #

Experimental

This function is considered experimental, so beware that it is subject to changes even in patch releases.

Create, load or edit multiple test cases.

test_cases = TestCase.init_many([
    ("test-case 1", [(test_sample_1, ground_truth_1), ...]),
    ("test-case 2", [(test_sample_2, ground_truth_2), ...])
])

test_suite = TestSuite("my test suite", test_cases=test_cases)

Changes are committed to the Kolena platform together. If there is an error, none of the edits would take effect.

Parameters:

Name Type Description Default
data List[Tuple[str, List[Tuple[TestSample, GroundTruth]]]]

A list of tuples where each tuple is a test case name and a set of test samples and ground truths tuples for the test case.

required
reset bool

If a test case of the same name already exists, overwrite with the provided test_samples.

False

Returns:

Type Description
List[TestCase]

The test cases.