Skip to content

kolena.workflow.test#

TestRun(model, test_suite, evaluator=None, configurations=None, reset=False) #

Bases: Frozen

A Model tested on a TestSuite using a specific Evaluator implementation.

Parameters:

Name Type Description Default
model Model

The model being tested.

required
test_suite TestSuite

The test suite on which to test the model.

required
evaluator Optional[Union[Evaluator, BasicEvaluatorFunction]]

An optional evaluator implementation. Requires a previously configured server-side evaluator to default to if omitted. (Please see BasicEvaluatorFunction for type definition.)

None
configurations Optional[List[EvaluatorConfiguration]]

a list of configurations to use when running the evaluator.

None
reset bool

overwrites existing inferences if set.

False

run() #

Run the testing process, first extracting inferences for all test samples in the test suite then performing evaluation.

load_test_samples() #

Load the test samples in the test suite that do not yet have inferences uploaded.

Returns:

Type Description
List[TestSample]

a list of all test samples in the test suite still requiring inferences.

iter_test_samples() #

Iterate through the test samples in the test suite that do not yet have inferences uploaded.

Returns:

Type Description
Iterator[TestSample]

an iterator over each test sample still requiring inferences.

upload_inferences(inferences) #

Upload inferences from a model.

Parameters:

Name Type Description Default
inferences List[Tuple[TestSample, Inference]]

the inferences, paired with their corresponding test samples, to upload.

required

evaluate() #

Perform evaluation by computing metrics for individual test samples, in aggregate across test cases, and across the complete test suite at each EvaluatorConfiguration.

test(model, test_suite, evaluator=None, configurations=None, reset=False) #

Test a Model on a TestSuite using a specific Evaluator implementation.

from kolena.workflow import test

test(model, test_suite, evaluator, reset=True)

Parameters:

Name Type Description Default
model Model

The model being tested, implementing the infer method.

required
test_suite TestSuite

The test suite on which to test the model.

required
evaluator Optional[Union[Evaluator, BasicEvaluatorFunction]]

An optional evaluator implementation. Requires a previously configured server-side evaluator to default to if omitted. (Please see BasicEvaluatorFunction for type definition.)

None
configurations Optional[List[EvaluatorConfiguration]]

A list of configurations to use when running the evaluator.

None
reset bool

Overwrites existing inferences if set.

False