Assumptions¶
Every statistical technique assumes something about the data. A linear regression assumes that the label is a linear function of the features and that its errors are normal, independent, and of constant variance; a nearest neighbors model assumes that the features are on similar scales. A technique can still be applied when the data does not meet one of its assumptions, but what it finds may be misleading: standard errors that are too small, or a model that is beaten by a simpler one. Some techniques also need something of the data, without which they cannot be applied at all (such as a model that cannot use missing values).
Each technique lists its assumptions and needs by name, in its
assumptions and needs:
import amos
print(amos.models.Linear.assumptions)
# ('linearity', 'normal_errors', 'constant_variance', 'independent_errors', 'no_multicollinearity')
print(amos.models.Linear.needs)
# ('numeric_features', 'no_missing_values', 'more_rows_than_features')
How amos checks them¶
amos tests each assumption and need that it can, at the time when it can:
- Pretests (in the analyst stage) test an assumption before a technique
is applied, on the real training rows (or, for an inferer, on every real
row).
check_normality,check_multicollinearity,check_linearity,check_scales,check_dispersion,check_features,check_label,check_sample_size,check_missingness,check_proportional_odds,check_overlap,check_instrument,check_parallel_trends,check_continuity, andcheck_stationarityeach test some of them, andcheck_assumptionsruns every pretest that the techniques of the workflow need. The pretests of an inferer's assumptions read its parameters (such as its "treatment" and "instrument"). - Audits (in the critic stage) test an assumption that can only be
tested from a fitted model.
audit_residualstests whether the errors are normal,audit_errorswhether they have constant variance,audit_independencewhether they are independent (in the dataset'sorder, and within its groups), andaudit_proportional_hazardswhether a Cox model's hazards are proportional.audit_assumptionsruns every audit that the dataset's model needs, andauto_evaluatorapplies it to a regression whose model makes such assumptions.
Whether or not a workflow names a pretest, amos runs the pretests of a
technique just before it is applied, so that they test the data that the
technique sees (after it was scaled or encoded, for example). The results
are in the dataset's "assumptions" table and its history. auto_model
uses the same pretests to rule out the models whose assumptions the data
does not meet.
Not every assumption can be tested, and the "Tests Used" column below says how each one is:
- Fully testable assumptions (such as normal errors, overlap, or proportional hazards) are decided by their tests.
- Partly testable assumptions are tested in the part that the data can show. A test can find evidence against them, but it cannot confirm them. The relevance of an instrument is tested, but not that it affects the outcome only through the treatment; the trends before a treatment are compared, but not what would have happened after it; the number of rows at a cutoff is tested for a jump, but not the outcome; and Little's test can reject that values are missing completely at random, but not prove it.
- Untestable assumptions (that every confounder is measured, that every common cause in causal discovery is measured, and that whether a value is missing does not depend on the missing value itself) are claims about what the data does not hold. They are listed so that a study can argue for them, but amos cannot check them.
With many rows, a test rejects even trivial departures from an assumption, so an assumption fails only if its test rejects it (at the 5% level) and the departure is large enough to matter: for example, errors that are materially skewed or heavy-tailed, or nonlinear terms that explain more than 2% of what a linear fit leaves unexplained.
What amos does when one fails¶
The assumption_violations setting of the "general" section decides:
| Setting | What amos does |
|---|---|
warn (the default) |
Warns (with an amos.AmosWarning) and goes on. The violations are in the report, and the comparison table of an experiment has a "violations" column. |
raise |
Raises an amos.AssumptionError. |
ignore |
Goes on without a warning. The results of the tests are still recorded. |
skip |
Leaves the branch of an experiment that uses the technique out of the comparison (and warns outside of an experiment). |
A hard need is never let pass, whatever the setting: in an experiment,
the branch is left out, and elsewhere an amos.AssumptionError is raised.
[general]
label = target
order = date
assumption_violations = skip
[analyst]
design = experiment
criterion = rmse
steps = split, scale, check, model
split_techniques = time_split
scale_techniques = standard
check_techniques = check_assumptions
model_techniques = linear, ridge, random_forest
[critic]
techniques = audit_assumptions, scorecard
Writing your own¶
An assumption is a subclass of amos.Assumption (and a need, of
amos.Need), named in snake case as techniques are. Name the pretest (or
audit) that tests it and write its pretest (or audit) method, then name
it in the assumptions of a technique:
import dataclasses
import amos
@dataclasses.dataclass(frozen = True)
class PositiveLabel(amos.Assumption):
"""The label is greater than 0."""
description = 'the label is greater than 0'
pretested_by = 'check_label'
def pretest(self, item, technique, found, cache):
passed = bool((item.y > 0).all())
detail = (
'has a positive label' if passed else
'assumes a positive label, but some values are 0 or less')
return self._verdict(technique, self.pretested_by, passed, detail)
@dataclasses.dataclass
class LogLinear(amos.models.Linear):
"""Fits a linear regression of the log of the label."""
assumptions = (*amos.models.Linear.assumptions, 'positive_label')
print(amos.conditions.borrow('positive_label').description)
# the label is greater than 0
The tables below list every assumption and need that amos knows, and then the assumptions and needs of each technique, by genre (genres whose techniques have none are left out).
Every assumption¶
| Name | The technique assumes that | Pretest | Audit | Tests Used |
|---|---|---|---|---|
causal_sufficiency |
every common cause of the variables is measured | none: it cannot be tested from the data | ||
constant_variance |
the errors have the same variance for every row | audit_errors |
Breusch and Pagan's test, with the squares of the features (as in White's test) | |
continuity |
the outcome would change smoothly at the cutoff without the treatment | check_continuity |
a binomial test of the numbers of rows just above and just below the cutoff (a simple form of McCrary's density test, which cannot show that the outcome would change smoothly) | |
equidispersion |
the variance of counts equals their mean | check_dispersion |
the Pearson dispersion of a Poisson regression | |
independent_errors |
the errors of different rows are independent | audit_independence |
Breusch and Godfrey's test and Durbin and Watson's statistic (in the dataset's order), and the intraclass correlation of the errors (within its groups) | |
linearity |
the label (or its log-odds) is a linear function of the features | check_linearity |
Ramsey's RESET test (for a regressed label) or Box and Tidwell's test (for two classes) | |
missing_at_random |
whether a value is missing depends only on values that are observed | none: it cannot be tested from the data | ||
missing_completely_at_random |
whether a value is missing has nothing to do with any value | check_missingness |
Little's test, and the standardized differences in the means of the features between the rows with and without each missing value | |
no_multicollinearity |
no feature is nearly a linear function of the others | check_multicollinearity |
the variance inflation factor of each numeric feature | |
normal_errors |
the errors of a regression are normal | check_normality |
audit_residuals |
D'Agostino and Pearson's test, with the skewness and excess kurtosis of the errors |
normal_features |
the features are normal | check_normality |
the skewness and excess kurtosis of each feature | |
overlap |
every row could have been treated or not | check_overlap |
the share of rows whose propensity score (from a logistic regression of the treatment on the features) is below 0.05 or above 0.95 | |
parallel_trends |
without the treatment, the treated and other rows would have changed alike | check_parallel_trends |
a t test of whether the treated and other units trended differently before the treatment (which cannot show how they would have changed after it) | |
proportional_hazards |
each feature multiplies the hazard by the same amount at every time | audit_proportional_hazards |
Grambsch and Therneau's test of the correlation of each feature's Schoenfeld residuals with the rank of the time | |
proportional_odds |
each feature has the same effect on the odds of being above every class | check_proportional_odds |
Brant's test, which compares the coefficients of a logistic regression for each cutoff between ordered classes | |
similar_scales |
the features are on similar scales | check_scales |
the ratio of the largest to the smallest standard deviation of the numeric features | |
stationarity |
the distribution of each series does not change over time | check_stationarity |
the augmented Dickey-Fuller and KPSS tests of each series (it is not stationary if the first does not reject a unit root and the second rejects stationarity) | |
unconfoundedness |
every common cause of the treatment and the outcome is a feature | none: it cannot be tested from the data | ||
valid_instrument |
the instrument changes the treatment and affects the outcome only through it | check_instrument |
the F statistic of the instruments in the first-stage regression of the treatment (whether the instrument affects the outcome only through the treatment cannot be tested) |
Every need¶
| Name | The technique needs | Pretest | Hard |
|---|---|---|---|
binary_label |
the label has two classes | check_label |
yes |
count_label |
the label is counts | check_label |
yes |
max_features |
there are no more features than it works well with | check_sample_size |
no |
max_rows |
there are no more training rows than it is fast enough for | check_sample_size |
no |
min_rows |
there are at least as many training rows as it learns well from | check_sample_size |
no |
more_rows_than_features |
there are more training rows than features | check_sample_size |
yes |
multiclass_label |
the label has three or more classes | check_label |
no |
no_missing_values |
no feature has a missing value | check_features |
yes |
numeric_features |
every feature is a number or a boolean | check_features |
yes |
ordinal_label |
the label has three or more ordered classes | check_label |
no |
Imputers (analyst)¶
| Name | Assumptions | Needs |
|---|---|---|
iterative_impute |
missing_at_random |
none |
knn_impute |
missing_at_random |
none |
mean_impute |
missing_completely_at_random |
none |
median_impute |
missing_completely_at_random |
none |
mode_impute |
missing_completely_at_random |
none |