Skip to content

kolena.workflow.TestSuite#

TestSuite(name, version=None, description=None, test_cases=None, reset=False, tags=None) #

Bases: Frozen

A test suite groups together one or more test cases. Typically a test suite represents a benchmark test dataset, with test cases representing different meaningful subsets, or slices, or this benchmark.

Rather than importing this class directly, use the TestSuite type definition returned from define_workflow.

workflow: Workflow instance-attribute #

The workflow of this test suite. Automatically populated when constructing via test suite type returned from define_workflow.

name: str instance-attribute #

The unique name of this test suite.

version: int instance-attribute #

The version of this test suite. A test suite's version is automatically incremented whenever it is edited via TestSuite.edit.

description: str instance-attribute #

Free-form, human-readable description of this test suite. Can be edited at any time via TestSuite.edit.

test_cases: List[TestCase] instance-attribute #

The TestCase objects belonging to this test suite.

tags: Set[str] instance-attribute #

The tags associated with this test suite.

Editor(test_cases, description, tags, reset) #

description(description) #

Update the description of the test suite.

add(test_case) #

Add a test case to this test suite. If a different version of the test case already exists in this test suite, it is replaced.

Parameters:

Name Type Description Default
test_case TestCase

The test case to add to the test suite.

required

remove(test_case) #

Remove a test case from this test suite. Does nothing if the test case is not in the test suite.

Parameters:

Name Type Description Default
test_case TestCase

The test case to remove.

required

create(name, description=None, test_cases=None, tags=None) classmethod #

Create a new test suite with the provided name.

Parameters:

Name Type Description Default
name str

The name of the new test suite to create.

required
description Optional[str]

Optional free-form description of the test suite to create.

None
test_cases Optional[List[TestCase]]

Optionally specify a list of test cases to populate the test suite.

None
tags Optional[Set[str]]

Optionally specify a set of tags to attach to the test suite.

None

Returns:

Type Description
TestSuite

The newly created test suite.

load(name, version=None) classmethod #

Load an existing test suite with the provided name.

Parameters:

Name Type Description Default
name str

The name of the test suite to load.

required
version Optional[int]

Optionally specify a particular version of the test suite to load. Defaults to the latest version when unset.

None

Returns:

Type Description
TestSuite

The loaded test suite.

load_all(*, tags=None) classmethod #

Load the latest version of all test suites with this workflow.

Parameters:

Name Type Description Default
tags Optional[Set[str]]

Optionally specify a set of tags to apply as a filter. The loaded test suites will include only test suites with tags matching each of these specified tags, i.e. test_suite.tags.intersection(tags) == tags.

None

Returns:

Type Description
List[TestSuite]

The latest version of all test suites, with matching tags when specified.

edit(reset=False) #

Edit this test suite in a context:

with test_suite.edit() as editor:
    # perform as many editing actions as desired
    editor.add(...)
    editor.remove(...)

Changes are committed to the Kolena platform when the context is exited.

Parameters:

Name Type Description Default
reset bool

Clear all existing test cases in the test suite.

False

load_test_samples() #

Load test samples for all test cases within this test suite.

Returns:

Type Description
List[Tuple[TestCase, List[TestSample]]]

A list of TestCases, each paired with the list of TestSamples it contains.