kolena.workflow.TestSuite

TestSuite(name, version=None, description=None, test_cases=None, reset=False, tags=None)

Bases: Frozen

A test suite groups together one or more test cases. Typically a test suite represents a benchmark test dataset, with test cases representing different meaningful subsets, or slices, or this benchmark.

Rather than importing this class directly, use the TestSuite type definition returned from define_workflow.

workflow: Workflow instance-attribute

The workflow of this test suite. Automatically populated when constructing via test suite type returned from define_workflow.

name: str instance-attribute

The unique name of this test suite.

version: int instance-attribute

The version of this test suite. A test suite's version is automatically incremented whenever it is edited via TestSuite.edit.

description: str instance-attribute

Free-form, human-readable description of this test suite. Can be edited at any time via TestSuite.edit.

test_cases: List[TestCase] instance-attribute

The TestCase objects belonging to this test suite.

tags: Set[str] instance-attribute

The tags associated with this test suite.

Editor(test_cases, description, tags, reset)

description(description)

Update the description of the test suite.

add(test_case)

Add a test case to this test suite. If a different version of the test case already exists in this test suite, it is replaced.

Parameters:

Name	Type	Description	Default
test_case	TestCase	The test case to add to the test suite.	required

remove(test_case)

Remove a test case from this test suite. Does nothing if the test case is not in the test suite.

Parameters:

Name	Type	Description	Default
test_case	TestCase	The test case to remove.	required

create(name, description=None, test_cases=None, tags=None) classmethod

Create a new test suite with the provided name.

Parameters:

Name	Type	Description	Default
name	str	The name of the new test suite to create.	required
description	Optional[str]	Optional free-form description of the test suite to create.	None
test_cases	Optional[List[TestCase]]	Optionally specify a list of test cases to populate the test suite.	None
tags	Optional[Set[str]]	Optionally specify a set of tags to attach to the test suite.	None

Returns:

Type	Description
TestSuite	The newly created test suite.

load(name, version=None) classmethod

Load an existing test suite with the provided name.

Parameters:

Name	Type	Description	Default
name	str	The name of the test suite to load.	required
version	Optional[int]	Optionally specify a particular version of the test suite to load. Defaults to the latest version when unset.	None

Returns:

Type	Description
TestSuite	The loaded test suite.

load_all(*, tags=None) classmethod

Load the latest version of all test suites with this workflow.

Parameters:

Name	Type	Description	Default
tags	Optional[Set[str]]	Optionally specify a set of tags to apply as a filter. The loaded test suites will include only test suites with tags matching each of these specified tags, i.e. test_suite.tags.intersection(tags) == tags.	None

Returns:

Type	Description
List[TestSuite]	The latest version of all test suites, with matching tags when specified.

edit(reset=False)

Edit this test suite in a context:

with test_suite.edit() as editor:
    # perform as many editing actions as desired
    editor.add(...)
    editor.remove(...)

Changes are committed to the Kolena platform when the context is exited.

Parameters:

Name	Type	Description	Default
reset	bool	Clear all existing test cases in the test suite.	False

load_test_samples()

Load test samples for all test cases within this test suite.

Returns:

Type	Description
List[Tuple[TestCase, List[TestSample]]]	A list of TestCases, each paired with the list of TestSamples it contains.