Techniques that cross-validate a model, as the last step of the "analyst".

A validator measures how well the dataset's model does on rows that it did not learn from, without using the test rows. It divides the training rows into folds with one of scikit-learn's cross-validation splitters, fits a new copy of the model to the rest of the training rows for each fold, and scores the copy on the fold with the critic's metrics. The scores of each fold are a table, stored in the dataset's tables under the validator's name, and their means are stored in its metrics as "cv_{metric}" (such as "cv_roc_auc"), so that a scorecard or an experiment's comparison shows them beside the scores on the test rows.

Rows that a sampler made up (the dataset's synthetic rows) are never scored. Instead, each sampler that was applied to the training rows is applied again to the rows that each copy of the model learns from. Transformers are not fitted again: a validator scores the model, given the features that the transformers made after learning from all of the training rows.

Contents

Validator: base class for techniques that cross-validate a model. GroupKFold, GroupShuffleSplit, KFold, LeaveOneGroupOut, RepeatedKFold, RepeatedStratifiedKFold, ShuffleSplit, StratifiedGroupKFold, StratifiedKFold, StratifiedShuffleSplit, TimeSeriesSplit: validators that wrap the cross-validation splitter of scikit-learn with the same name. LeaveOneRowOut: a validator that wraps scikit-learn's LeaveOneOut (the name leave_one_out is an encoder).

GroupKFold(contents='sklearn.model_selection.GroupKFold', name='', parameters=dict()) dataclass

Bases: Validator

Folds that keep all of the rows of each group in the same fold.

Wraps

sklearn.model_selection.GroupKFold from scikit-learn.

GroupShuffleSplit(contents='sklearn.model_selection.GroupShuffleSplit', name='', parameters=dict()) dataclass

Bases: Validator

Repeated random splits of the groups (not the rows) into two sets.

Wraps

sklearn.model_selection.GroupShuffleSplit from scikit-learn.

KFold(contents='sklearn.model_selection.KFold', name='', parameters=dict()) dataclass

Bases: Validator

Divides the rows into folds ("n_splits", 5 by default) at random.

Wraps

sklearn.model_selection.KFold from scikit-learn.

LeaveOneGroupOut(contents='sklearn.model_selection.LeaveOneGroupOut', name='', parameters=dict()) dataclass

Bases: Validator

Scores each group with a copy of the model fitted to the other groups.

Wraps

sklearn.model_selection.LeaveOneGroupOut from scikit-learn.

LeaveOneRowOut(contents='sklearn.model_selection.LeaveOneOut', name='', parameters=dict()) dataclass

Bases: Validator

Predicts each row with a copy of the model fitted to every other row.

A fold of one row cannot be scored alone, so the predictions of every fold are scored together. The table has the prediction of each row.

Wraps

sklearn.model_selection.LeaveOneOut from scikit-learn.

RepeatedKFold(contents='sklearn.model_selection.RepeatedKFold', name='', parameters=dict()) dataclass

Bases: Validator

k_fold repeated with different random folds ("n_repeats" times).

Wraps

sklearn.model_selection.RepeatedKFold from scikit-learn.

RepeatedStratifiedKFold(contents='sklearn.model_selection.RepeatedStratifiedKFold', name='', parameters=dict()) dataclass

Bases: Validator

stratified_k_fold repeated with different random folds.

Wraps

sklearn.model_selection.RepeatedStratifiedKFold from scikit-learn.

ShuffleSplit(contents='sklearn.model_selection.ShuffleSplit', name='', parameters=dict()) dataclass

Bases: Validator

Repeated random splits of the rows into two sets (Monte Carlo).

Wraps

sklearn.model_selection.ShuffleSplit from scikit-learn.

StratifiedGroupKFold(contents='sklearn.model_selection.StratifiedGroupKFold', name='', parameters=dict()) dataclass

Bases: Validator

Folds that keep each group together and the classes in proportion.

Wraps

sklearn.model_selection.StratifiedGroupKFold from scikit-learn.

StratifiedKFold(contents='sklearn.model_selection.StratifiedKFold', name='', parameters=dict()) dataclass

Bases: Validator

Folds that keep the share of each class the same in every fold.

Wraps

sklearn.model_selection.StratifiedKFold from scikit-learn.

StratifiedShuffleSplit(contents='sklearn.model_selection.StratifiedShuffleSplit', name='', parameters=dict()) dataclass

Bases: Validator

Repeated random splits into two sets, with the classes in proportion.

Wraps

sklearn.model_selection.StratifiedShuffleSplit from scikit-learn.

TimeSeriesSplit(contents='sklearn.model_selection.TimeSeriesSplit', name='', parameters=dict()) dataclass

Bases: Validator

Scores later rows with copies of the model fitted to earlier rows.

Use this when the rows are ordered in time (or set "order" to a column, such as a date), so that the model is never scored on rows that came before the rows it learned from.

Wraps

sklearn.model_selection.TimeSeriesSplit from scikit-learn.

Validator(contents=None, name='', parameters=dict()) dataclass

Bases: Operation, ABC

Base class for techniques that cross-validate the dataset's model.

contents is the cross-validation splitter to wrap: a class (or its import path) whose split method yields the positions of the rows to learn from and of the rows to score in each fold, as in scikit-learn. A Validator can be used directly to wrap any such class. Unless "shuffle" is false, a splitter that can shuffle the rows does, with the dataset's seed.

Parameters:

Name	Type	Description	Default
name	str	name used to refer to the technique in a workflow. Defaults to None, in which case it is the snake case name of the class.	''
contents	str | Callable[..., Any] | None	the splitter class or its import path. Defaults to None.	None
parameters	GenericDict	keyword arguments for the splitter (such as "n_splits"), and "metrics", "groups", and "order". Defaults to an empty dict.	dict()

pools = False class-attribute

Public Methods

implement(item, metrics=None, groups=None, order=None, **kwargs)

Cross-validates the model on the training rows of item.

Parameters:

Name	Type	Description	Default
item	Dataset	the dataset with a fitted model.	required
metrics	Sequence[str] | None	names of the metrics to score each fold with. Defaults to None, which uses the metrics of a scorecard for the task.	None
groups	str | None	name of the column that identifies each row's group, for a validator that keeps each group in one fold. Defaults to None, in which case the dataset's first group is used.	None
order	str | None	name of the column to order the rows by (such as a date) before they are divided. Defaults to None, which uses the current order of the rows.	None
**kwargs	Any	parameters for the splitter.	{}

Raises:

Type	Description
ValueError	if there is no model, or if the validator needs groups or classes that the dataset does not have.

Returns:

Type	Description
Dataset	The dataset, with a table of the scores of each fold, and the mean of each score in metrics.

Source code in src/amos/validators.py

def implement(
    self,
    item: base.Dataset,
    metrics: Sequence[str] | None = None,
    groups: str | None = None,
    order: str | None = None,
    **kwargs: Any) -> base.Dataset:
    """Cross-validates the model on the training rows of `item`.

    Args:
        item: the dataset with a fitted model.
        metrics: names of the metrics to score each fold with. Defaults to
            `None`, which uses the metrics of a scorecard for the task.
        groups: name of the column that identifies each row's group, for
            a validator that keeps each group in one fold. Defaults to
            `None`, in which case the dataset's first group is used.
        order: name of the column to order the rows by (such as a date)
            before they are divided. Defaults to `None`, which uses the
            current order of the rows.
        **kwargs: parameters for the splitter.

    Raises:
        ValueError: if there is no model, or if the validator needs
            groups or classes that the dataset does not have.

    Returns:
        The dataset, with a table of the scores of each fold, and the
            mean of each score in `metrics`.

    """
    model = self._check(item)
    clone = importlib.import_module('sklearn.base').clone
    training = item._train_rows()
    rows = training[~training.isin(item.synthetic)]
    if order is not None:
        rows = item.data.loc[rows].sort_values(order, kind = 'stable').index
    x = item.data.loc[rows, _columns(model, item)]
    y = item.y.loc[rows]
    column = groups or (item.groups[0] if item.groups else None)
    if self.uses_groups and column is None:
        message = f'{self.name!r} needs the name of a "groups" column'
        raise ValueError(message)
    labels = item.data.loc[rows, column] if self.uses_groups else None
    splitter = self._make_tool(item, kwargs)
    samplers = [
        tool for tool in item.fitted.values()
        if hasattr(tool, 'fit_resample')]
    folds: list[_Fold] = []
    divided = splitter.split(x, y, labels)
    for number, (learn, score) in enumerate(divided, start = 1):
        x_learn, y_learn = x.iloc[learn], y.iloc[learn]
        for sampler in samplers:
            x_learn, y_learn = clone(sampler).fit_resample(x_learn, y_learn)
        estimator = clone(model).fit(x_learn, y_learn)
        folds.append(_Fold(
            number, len(y_learn), *_predict(estimator, x.iloc[score], item)))
    names = evaluators.Scorecard()._metric_names(item, [], metrics)
    summarize = _pool if self.pools else _average
    table, scores = summarize(item, rows, folds, names)
    item.tables[self.name] = table
    for name, score in scores.items():
        item.metrics[f'cv_{name}'] = score['mean']
    item.record(
        self.name,
        tool = utilities.describe_tool(self.contents),
        parameters = utilities.parameters_of(splitter),
        folds = len(table),
        rows = len(rows),
        scores = scores)
    return item