tools¶
tools.tests¶
This module contains all tests.

class
RegressionTest
(model, data, test_definition_version='last', model_version='last', data_version='last', repo_info=<pailab.ml_repo.repo_objects.RepoInfo object>)¶ Bases:
pailab.tools.tests.Test
Regression test.
Note
In general, tests are automatically constructed and run using
pailab.ml_repo.repo.run_tests()
. As a user, there is nearly no need to construct a test by hand.A regression test compares a specified measure of a reference model described by a label to the respective measure of the model to be tested. It fails, if the measure of the tested model is greater then a given tolerance of the reference measure, i.e. the test fails if
 measuremeasure_ref < tol and an absolute tolerance is defined,
 measuremeasure_ref < tol*measure_ref if a relative tolerance is used.
All the attributes specific for the regression test (i.e. not contained in the base class) are retrieved during the run of the test from the underlying testdefinition.
Parameters:  model (str) – Name of model for which the test is applied.
 data (str) – Name of dataset used in the test.
 test_definition_version (str, optional) – Defaults to latest version. Version of the tests’s underlying
pailab.tools.tests.TestDefinition
that is used as basis for the test.  model_version (str, optional) – Defaults to latest version. Version of the model the test is applied to.
 data_version (str, optional) – Defaults to latest version. Version of the data used in the test.

test_definition
¶ Name of underlying
pailab.tools.tests.TestDefinition
.Type: str

model
¶ Name of model for which the test is applied.
Type: str

data
¶ Name of dataset used in the test.
Type: str

test_definition_version
¶ Version of the tests’s underlying
pailab.tools.tests.TestDefinition
that is used as basis for the test.Type: str

model_version
¶ Version of the model the test is applied to.
Type: str

data_version
¶ Version of the data used in the test.
Type: str

result
¶ Describes the state of the test.
Type: str, ‘not run’, ‘failed’, ‘succeeded’

details
¶ Contains details when test fails, otherwise empty dict.
Type: dict

get_modifier_versions
(ml_repo)¶ Get the modifier versions
Parameters: repo (MLrepository) – repository used to get and store the data Returns: tuple of string, dict – return the object name and the modifiers

class
RegressionTestDefinition
(reference='prod', models=None, data=None, labels=None, measures=None, tol=0.001, repo_info=<pailab.ml_repo.repo_objects.RepoInfo object>, relative=False)¶ Bases:
pailab.tools.tests.TestDefinition
Definition of a regression test.
A regression test compares a specified measure of a reference model described by a label to the respective measure of the model to be tested. It fails, if the measure of the tested model is greater then a given tolerance of the reference measure, i.e. the test fails if
 measuremeasure_ref < tol and an absolute tolerance is defined,
 measuremeasure_ref < tol*measure_ref if a relative tolerance is used.
Note
The tests needs the chosen measure(s) to be computed, therefore you have to take care that the measure has been added to the repo (using
pailab.ml_repo.repo.MLRepo.add_measure()
)Examples
Add a test for the model ‘my_model’ on a data set named ‘test_data’ which checks if the maximum error of the model is not greater than 10% in relation to the error of the reference model defined by the label ‘production_model’
>>> test_def = RegressionTestDefinition(models=['my_model'], reference ='production_model', data = ['test_data'], measures = ['max'], tol = 0.1, relative = True) >>> ml_repo.add(test_def)
Add a test applied to all models in the repo (always the latest versions of the models are used within the tests)
>>> test_def = RegressionTestDefinition(models=None, reference ='production_model', data = ['test_data'], measures = ['max'], tol = 0.1, relative = True) >>> ml_repo.add(test_def)
Parameters:  models (iterable with str items, optional) – Defaults to None. Iterable (e.g. list of str) returning names of the models to be tested.
 data (iterable with str items, optional) – Defaults to None. Iterable (e.g. list of str) returning names of the data used for testing.
 labels (iterable with str items, optional) – Defaults to []. Iterable returning labels defining models to be tested.
 measures ([type], optional) – Defaults to None. List of measures used in the test
 reference (str, optional) – Defaults to ‘prod’. Label defining the reference model to which the measures are compared.
 tol (float, optional) – Defaults to 1e3. Tolerance, if relative is False, the test fails if new_valueref_value < tol, otherwise if new_valueref_value < tol*ref_value.
 relative (bool, optional) – Defaults to False.
 repo_info (RepoInfo, optional) Defaults to RepoInfo() –

models
¶ List of strings defining the models to be tested.
Type: list of str

labels
¶ List of strings defining the labels to be tested.
Type: list of str

data
¶ List of strings defining the names of the data to be tested
Type: list of str

class
Test
(model, data, test_definition_version='last', model_version='last', data_version='last', repo_info=<pailab.ml_repo.repo_objects.RepoInfo object>)¶ Bases:
pailab.ml_repo.repo.Job
Base class for all tests.
Note
In general, tests are automatically constructed and run using
pailab.ml_repo.repo.run_tests()
. As a user, there is nearly no need to construct a test by hand.Parameters:  model (str) – Name of model for which the test is applied.
 data (str) – Name of dataset used in the test.
 test_definition_version (str, optional) – Defaults to latest version. Version of the tests’s underlying
pailab.tools.tests.TestDefinition
that is used as basis for the test.  model_version (str, optional) – Defaults to latest version. Version of the model the test is applied to.
 data_version (str, optional) – Defaults to latest version. Version of the data used in the test.
 model – Name of model for which the test is applied.
 data – Name of dataset used in the test.
 test_definition_version – Defaults to latest version. Version of the tests’s underlying
pailab.tools.tests.TestDefinition
that is used as basis for the test.  model_version – Defaults to latest version. Version of the model the test is applied to.
 data_version – Defaults to latest version. Version of the data used in the test.

test_definition
¶ Name of underlying
pailab.tools.tests.TestDefinition
.Type: str

model
¶ Name of model for which the test is applied.
Type: str

data
¶ Name of dataset used in the test.
Type: str

test_definition_version
¶ Version of the tests’s underlying
pailab.tools.tests.TestDefinition
that is used as basis for the test.Type: str

model_version
¶ Version of the model the test is applied to.
Type: str

data_version
¶ Version of the data used in the test.
Type: str

result
¶ Describes the state of the test.
Type: str, ‘not run’, ‘failed’, ‘succeeded’

details
¶ Contains details when test fails, otherwise empty dict.
Type: dict

class
TestDefinition
(models=None, data=None, labels=[], repo_info=<pailab.ml_repo.repo_objects.RepoInfo object>)¶ Bases:
pailab.ml_repo.repo_objects.RepoObject
,abc.ABC
Abstract base class for all test definitions.
A test definition defines the framework such as models and data the tests are applied to. It also provides a create method which creates the test cases for a special model and data version.
Parameters:  models (iterable with str items, optional) – Defaults to None. Iterable (e.g. list of str) returning names of the models to be tested.
 data (iterable with str items, optional) – Defaults to None. Iterable (e.g. list of str) returning names of the data used for testing.
 labels (iterable with str items, optional) – Defaults to []. Iterable returning labels defining models to be tested.
 repo_info (RepoInfo, optional) Defaults to RepoInfo() –

models
¶ List of strings defining the models to be tested.
Type: list of str

labels
¶ List of strings defining the labels to be tested.
Type: list of str

data
¶ List of strings defining the names of the data to be tested
Type: list of str

create
(ml_repo: pailab.ml_repo.repo.MLRepo)¶ Create a set of tests for models of the repository.
Parameters:  ml_repo (MLRepo) – ml repo
 models (dict, optional) – Defaults to {}. Dictionary of model names to version numbers to apply tests for. If empty, all latest models are used.
 data (dict, optional) – Defaults to {}. Dictionary of data the tests are applied to. If empty, all latest test and train data will be used.
 labels (list, optional) – Defaults to []. List of labels to which the tests are applied.
Returns: [description]
Return type: [type]
tools.tree¶
This module contains all functions and classes for the MLTree. The MLTree buils a treelike structure of the objects in a given repository. This allows the user to access objects in a comfortable way allowing for autocompletion (i.e. in Jupyter notebooks).
To use it one can simply call the pailab.tools.tree.MLTree.add_tree()
method to
add such a tree to the current repository:
>>from pailab.tools.tree import MLTree
>>MLTree.add_tree(ml_repo)
After the tree has been added, one can simply use the tree. Here, using autocompletion makes the basic work wih repo objects quite simply. Each tree node provides useful functions that can be applied:
load
loads the object of the given tree node or the child tree nodes of the current node. a After calling load the respective nodes have a new attributeobj
that contains the respective loaded object. To load all objects belonging to the models subtree like parameters, evaluations or measures one can call:>> ml_repo.tree.models.load()
history
lists the history of all objects of the respective subtree, where history excepts certain parameters such as a range of versions or which repo object information to include. To list th history of all training data just use:>> ml_repo.tree.training_data.history()
modifications
lists all objects of the respective subtree that have been modified and no yet been committed.
There are also node dependent function (depending on what object the node represents).

class
MLTree
(ml_repo)¶ Bases:
object

static
add_tree
(ml_repo)¶ Adds an MLTree to a repository.
Parameters: ml_repo (MLRepo) – the repository the tre is added

modifications
()¶ Return a dictionary of all objects that were modified but no yet commited to the repository.
Returns: dictionary mapping object ids to dictionary of the modified attributes Return type: dict

reload
(**kwargs)¶ Method to reload the tree after objects have been added or deleted from the repository.

static
tools.interpretation¶
This module contains functions for model agnostic interpretation methods.

class
ICE_Results
¶ Bases:
object

compute_cluster_average
(ice_results_2)¶ [summary]
Parameters: ice_results_2 ([type]) – [description] Returns: Matrix containing the average of values from ice_results_2 over the different clusters from this result. Return type: numpy matrix


compute_ice
(ml_repo, x_values, data, model=None, model_label=None, model_version='last', data_version='last', y_coordinate=0, x_coordinate=0, start_index=0, end_index=1, cache=False, clustering_param=None, scale='')¶ Compute individual conditional expectation (ice) for a given dataset and model
Parameters:  ml_repo (MLRepo) – MLRepo used to retrieve model and data and be used in caching.
 x_values (list) – List of x values for the ICE.
 data (str, DataSet, RawData) – Either name of data or directly the data object which is used as basis for ICE (an ICE is computed at each datapoint of the data).
 model (str, optional) – Name of model in the MLRepo for which the ICE will be computed. If None, model_label must be specified, defining the model to be used. Defaults to None.
 model_label (str, optional) – Label defining the model to be used. Defaults to None.
 model_version (str, optional) – Version of model to be used for ICE. Only needed if model is specified. Defaults to RepoStore.LAST_VERSION.
 data_version (str, optional) – Version of data used. Defaults to RepoStore.LAST_VERSION.
 y_coordinate (int or str, optional) – Defines ycoordinate (either by name or coordinate index) for which the ICE is computed. Defaults to 0.
 x_coordinate (int or str, optional) – Defines xcoordinate (either by name or coordinate index) for which the ICE is computed. Defaults to 0.
 start_index (int, optional) – Defines the start index of the data to be used in ICE computation (data[start_index:end_index] will be used). Defaults to 0.
 end_index (int, optional) – Defines the end index of the data to be used in ICE computation (data[start_index:end_index] will be used). Defaults to 1.
 cache (bool, optional) – If True, results will be cached. Defaults to False.
 clustering_param (dict or None, optional) – Dictionary of parameters for method functional_clustering that is called if the parameter is not None and applies functional clustering to the ICE curves.
 scale (str or int, optional) – String defining the scaling for the functions before functional clustering is applied. Scaling is perfomred by dividing the vector of the yvalues of the ICE by the respective vector norm defined by scaling. The scaling must be one of numpy’s valid strings for linalg.norm’s ord parameter. If string is empty, no scaling will be applied. Defaults to ‘’.
Returns: result object containing all relevant data (including functional clustering)
Return type:

generate_prototypes
(ml_repo, data, n_prototypes, n_criticisms, data_version='last', use_x=True, data_start_index=0, data_end_index=1, metric='rbf', witness_penalty=1.0, **kwds)¶ This methods computes for a given test/training dataset prototypes and criticisms and adds them as separate test data sets to the repository.
This methods computes for given test/training dataset prototypes and criticisms, i.e. datapoints from th given set that are typical representatives (prototypes) and datapoints that are not well representatives (criticisms). Here, a simple greedy algorithm using MDM2 is used to compute the prototypes and a witness function together with some simple penalty are used to compute the criticisms (see e.g. C. Molnar, Interpretable Machine Learning).
Parameters:  ml_repo (MLRepo) – The repository used to retrieve data and store prototypes/criticisms.
 data (str) – Name of data used for computation.
 n_prototypes (int) – Number of prototypes.
 n_criticisms (int) – Number of criticisms.
 data_version (str) – Version of data to be used. Defaults to RepoStore.LAST_VERSION.
 use_x (bool) – Flags that determine if prototypes are computed w.r.t. x or y coordinates. Defaults to True.
 data_start_index (int) – Startindex of data used.
 data_end_index (int) – Endindex of data used.
 metric (str or callable, optional) –
The metric to use when calculating kernel between instances in a feature array. If metric is a string, it must be one of the metrics in sklearn.metrics.pairwise.PAIRWISE_KERNEL_FUNCTIONS. If metric is precomputed, X is assumed to be a kernel matrix. Alternatively, if metric is a callable function, it is called on each pair of instances (rows) and the resulting value recorded. The callable should take two arrays from X as input and return a value indicating the distance between them. Currently, sklearn provides the following strings: ‘additive_chi2’, ‘chi2’, ‘linear’, ‘poly’, ‘polynomial’, ‘rbf’,
‘laplacian’, ‘sigmoid’, ‘cosine’  witness_penalty (float) – Penalty parameter to include some penalty to avoid to close criticisms.
 **kwds – optional keyword parameters Any further parameters are passed directly to the kernel function.
Raises: Exception
– If sklearn is not installedReturns: List of indices defining the datapoints which are the resulting prototypes. list of int: List of indices defining the datapoints which are the resulting criticisms.
Return type: list of int