Skip to content

Assumptions

Every statistical technique assumes something about the data. A linear regression assumes that the label is a linear function of the features and that its errors are normal, independent, and of constant variance; a nearest neighbors model assumes that the features are on similar scales. A technique can still be applied when the data does not meet one of its assumptions, but what it finds may be misleading: standard errors that are too small, or a model that is beaten by a simpler one. Some techniques also need something of the data, without which they cannot be applied at all (such as a model that cannot use missing values).

Each technique lists its assumptions and needs by name, in its assumptions and needs:

import amos

print(amos.models.Linear.assumptions)
# ('linearity', 'normal_errors', 'constant_variance', 'independent_errors', 'no_multicollinearity')
print(amos.models.Linear.needs)
# ('numeric_features', 'no_missing_values', 'more_rows_than_features')

How amos checks them

amos tests each assumption and need that it can, at the time when it can:

  • Pretests (in the analyst stage) test an assumption before a technique is applied, on the real training rows (or, for an inferer, on every real row). check_normality, check_multicollinearity, check_linearity, check_scales, check_dispersion, check_features, check_label, check_sample_size, check_missingness, check_proportional_odds, check_overlap, check_instrument, check_parallel_trends, check_continuity, and check_stationarity each test some of them, and check_assumptions runs every pretest that the techniques of the workflow need. The pretests of an inferer's assumptions read its parameters (such as its "treatment" and "instrument").
  • Audits (in the critic stage) test an assumption that can only be tested from a fitted model. audit_residuals tests whether the errors are normal, audit_errors whether they have constant variance, audit_independence whether they are independent (in the dataset's order, and within its groups), and audit_proportional_hazards whether a Cox model's hazards are proportional. audit_assumptions runs every audit that the dataset's model needs, and auto_evaluator applies it to a regression whose model makes such assumptions.

Whether or not a workflow names a pretest, amos runs the pretests of a technique just before it is applied, so that they test the data that the technique sees (after it was scaled or encoded, for example). The results are in the dataset's "assumptions" table and its history. auto_model uses the same pretests to rule out the models whose assumptions the data does not meet.

Not every assumption can be tested, and the "Tests Used" column below says how each one is:

  • Fully testable assumptions (such as normal errors, overlap, or proportional hazards) are decided by their tests.
  • Partly testable assumptions are tested in the part that the data can show. A test can find evidence against them, but it cannot confirm them. The relevance of an instrument is tested, but not that it affects the outcome only through the treatment; the trends before a treatment are compared, but not what would have happened after it; the number of rows at a cutoff is tested for a jump, but not the outcome; and Little's test can reject that values are missing completely at random, but not prove it.
  • Untestable assumptions (that every confounder is measured, that every common cause in causal discovery is measured, and that whether a value is missing does not depend on the missing value itself) are claims about what the data does not hold. They are listed so that a study can argue for them, but amos cannot check them.

With many rows, a test rejects even trivial departures from an assumption, so an assumption fails only if its test rejects it (at the 5% level) and the departure is large enough to matter: for example, errors that are materially skewed or heavy-tailed, or nonlinear terms that explain more than 2% of what a linear fit leaves unexplained.

What amos does when one fails

The assumption_violations setting of the "general" section decides:

Setting What amos does
warn (the default) Warns (with an amos.AmosWarning) and goes on. The violations are in the report, and the comparison table of an experiment has a "violations" column.
raise Raises an amos.AssumptionError.
ignore Goes on without a warning. The results of the tests are still recorded.
skip Leaves the branch of an experiment that uses the technique out of the comparison (and warns outside of an experiment).

A hard need is never let pass, whatever the setting: in an experiment, the branch is left out, and elsewhere an amos.AssumptionError is raised.

[general]
label = target
order = date
assumption_violations = skip

[analyst]
design = experiment
criterion = rmse
steps = split, scale, check, model
split_techniques = time_split
scale_techniques = standard
check_techniques = check_assumptions
model_techniques = linear, ridge, random_forest

[critic]
techniques = audit_assumptions, scorecard

Writing your own

An assumption is a subclass of amos.Assumption (and a need, of amos.Need), named in snake case as techniques are. Name the pretest (or audit) that tests it and write its pretest (or audit) method, then name it in the assumptions of a technique:

import dataclasses

import amos


@dataclasses.dataclass(frozen = True)
class PositiveLabel(amos.Assumption):
    """The label is greater than 0."""

    description = 'the label is greater than 0'
    pretested_by = 'check_label'

    def pretest(self, item, technique, found, cache):
        passed = bool((item.y > 0).all())
        detail = (
            'has a positive label' if passed else
            'assumes a positive label, but some values are 0 or less')
        return self._verdict(technique, self.pretested_by, passed, detail)


@dataclasses.dataclass
class LogLinear(amos.models.Linear):
    """Fits a linear regression of the log of the label."""

    assumptions = (*amos.models.Linear.assumptions, 'positive_label')


print(amos.conditions.borrow('positive_label').description)
# the label is greater than 0

The tables below list every assumption and need that amos knows, and then the assumptions and needs of each technique, by genre (genres whose techniques have none are left out).

Every assumption

Name The technique assumes that Pretest Audit Tests Used
causal_sufficiency every common cause of the variables is measured none: it cannot be tested from the data
constant_variance the errors have the same variance for every row audit_errors Breusch and Pagan's test, with the squares of the features (as in White's test)
continuity the outcome would change smoothly at the cutoff without the treatment check_continuity a binomial test of the numbers of rows just above and just below the cutoff (a simple form of McCrary's density test, which cannot show that the outcome would change smoothly)
equidispersion the variance of counts equals their mean check_dispersion the Pearson dispersion of a Poisson regression
independent_errors the errors of different rows are independent audit_independence Breusch and Godfrey's test and Durbin and Watson's statistic (in the dataset's order), and the intraclass correlation of the errors (within its groups)
linearity the label (or its log-odds) is a linear function of the features check_linearity Ramsey's RESET test (for a regressed label) or Box and Tidwell's test (for two classes)
missing_at_random whether a value is missing depends only on values that are observed none: it cannot be tested from the data
missing_completely_at_random whether a value is missing has nothing to do with any value check_missingness Little's test, and the standardized differences in the means of the features between the rows with and without each missing value
no_multicollinearity no feature is nearly a linear function of the others check_multicollinearity the variance inflation factor of each numeric feature
normal_errors the errors of a regression are normal check_normality audit_residuals D'Agostino and Pearson's test, with the skewness and excess kurtosis of the errors
normal_features the features are normal check_normality the skewness and excess kurtosis of each feature
overlap every row could have been treated or not check_overlap the share of rows whose propensity score (from a logistic regression of the treatment on the features) is below 0.05 or above 0.95
parallel_trends without the treatment, the treated and other rows would have changed alike check_parallel_trends a t test of whether the treated and other units trended differently before the treatment (which cannot show how they would have changed after it)
proportional_hazards each feature multiplies the hazard by the same amount at every time audit_proportional_hazards Grambsch and Therneau's test of the correlation of each feature's Schoenfeld residuals with the rank of the time
proportional_odds each feature has the same effect on the odds of being above every class check_proportional_odds Brant's test, which compares the coefficients of a logistic regression for each cutoff between ordered classes
similar_scales the features are on similar scales check_scales the ratio of the largest to the smallest standard deviation of the numeric features
stationarity the distribution of each series does not change over time check_stationarity the augmented Dickey-Fuller and KPSS tests of each series (it is not stationary if the first does not reject a unit root and the second rejects stationarity)
unconfoundedness every common cause of the treatment and the outcome is a feature none: it cannot be tested from the data
valid_instrument the instrument changes the treatment and affects the outcome only through it check_instrument the F statistic of the instruments in the first-stage regression of the treatment (whether the instrument affects the outcome only through the treatment cannot be tested)

Every need

Name The technique needs Pretest Hard
binary_label the label has two classes check_label yes
count_label the label is counts check_label yes
max_features there are no more features than it works well with check_sample_size no
max_rows there are no more training rows than it is fast enough for check_sample_size no
min_rows there are at least as many training rows as it learns well from check_sample_size no
more_rows_than_features there are more training rows than features check_sample_size yes
multiclass_label the label has three or more classes check_label no
no_missing_values no feature has a missing value check_features yes
numeric_features every feature is a number or a boolean check_features yes
ordinal_label the label has three or more ordered classes check_label no

Imputers (analyst)

Name Assumptions Needs
iterative_impute missing_at_random none
knn_impute missing_at_random none
mean_impute missing_completely_at_random none
median_impute missing_completely_at_random none
mode_impute missing_completely_at_random none

Samplers (analyst)

Name Assumptions Needs
adasyn similar_scales numeric_features, no_missing_values
borderline_smote similar_scales numeric_features, no_missing_values
near_miss similar_scales numeric_features, no_missing_values
random_over none none
random_under none none
smote similar_scales numeric_features, no_missing_values
smote_enn similar_scales numeric_features, no_missing_values
smote_tomek similar_scales numeric_features, no_missing_values
tomek_links similar_scales numeric_features, no_missing_values

Models (analyst)

Name Assumptions Needs
adaboost none numeric_features
baseline none none
binomial_bayes_mixedglm linearity, no_multicollinearity binary_label, numeric_features, no_missing_values
catboost none none
cox proportional_hazards, no_multicollinearity numeric_features, no_missing_values
decision_tree none numeric_features
elastic_net linearity, similar_scales numeric_features, no_missing_values
explainable_boosting none none
extra_trees none numeric_features
fixest linearity, no_multicollinearity numeric_features, no_missing_values, more_rows_than_features
gee no_multicollinearity numeric_features, no_missing_values, more_rows_than_features
generalized_poisson independent_errors count_label, numeric_features, no_missing_values, more_rows_than_features
glm independent_errors, no_multicollinearity numeric_features, no_missing_values, more_rows_than_features
gradient_boosting none none
knn similar_scales numeric_features, no_missing_values, max_rows, max_features
lasso linearity, similar_scales numeric_features, no_missing_values
lightgbm none none
linear linearity, normal_errors, constant_variance, independent_errors, no_multicollinearity numeric_features, no_missing_values, more_rows_than_features
logit linearity, similar_scales numeric_features, no_missing_values
logit_sm linearity, independent_errors, no_multicollinearity binary_label, numeric_features, no_missing_values, more_rows_than_features
mixedlm linearity, normal_errors, constant_variance, no_multicollinearity numeric_features, no_missing_values, more_rows_than_features
mnlogit linearity, independent_errors, no_multicollinearity multiclass_label, numeric_features, no_missing_values, more_rows_than_features
naive_bayes normal_features, no_multicollinearity numeric_features, no_missing_values
negative_binomial independent_errors count_label, numeric_features, no_missing_values, more_rows_than_features
neural_network similar_scales numeric_features, no_missing_values, min_rows
ols linearity, constant_variance, independent_errors, no_multicollinearity numeric_features, no_missing_values, more_rows_than_features
ordinal_regression proportional_odds, independent_errors ordinal_label, numeric_features, no_missing_values, more_rows_than_features
poisson equidispersion, independent_errors count_label, numeric_features, no_missing_values, more_rows_than_features
probit linearity, independent_errors, no_multicollinearity binary_label, numeric_features, no_missing_values, more_rows_than_features
quantile_regression linearity, independent_errors numeric_features, no_missing_values, more_rows_than_features
random_forest none numeric_features
ridge linearity, similar_scales numeric_features, no_missing_values
robust_regression linearity, independent_errors numeric_features, no_missing_values, more_rows_than_features
svm similar_scales numeric_features, no_missing_values, max_rows
tabpfn none none
wls linearity, independent_errors, no_multicollinearity numeric_features, no_missing_values, more_rows_than_features
xgboost none numeric_features
zero_inflated_poisson independent_errors count_label, numeric_features, no_missing_values, more_rows_than_features

Inferers (critic)

Name Assumptions Needs
difference_in_differences parallel_trends, overlap none
distance_matching unconfoundedness, overlap none
doubly_robust unconfoundedness, overlap none
dr_learner unconfoundedness, overlap none
glm_adjustment unconfoundedness, overlap none
instrumental_variable valid_instrument none
interactive_iv valid_instrument none
interactive_regression unconfoundedness, overlap none
lpcmci stationarity none
partially_linear unconfoundedness none
partially_linear_iv valid_instrument none
partially_linear_panel unconfoundedness none
partially_logistic unconfoundedness none
pcmci stationarity, causal_sufficiency none
pcmci_plus stationarity, causal_sufficiency none
potential_outcomes unconfoundedness, overlap none
propensity_matching unconfoundedness, overlap none
propensity_stratification unconfoundedness, overlap none
propensity_weighting unconfoundedness, overlap none
quantile_effects unconfoundedness, overlap none
regression_adjustment unconfoundedness, overlap none
regression_discontinuity continuity none
s_learner unconfoundedness, overlap none
sample_selection unconfoundedness none
t_learner unconfoundedness, overlap none
time_series_effect stationarity, causal_sufficiency none
tmle unconfoundedness, overlap none
x_learner unconfoundedness, overlap none