iqr#
- class pdcleaner.detection.gaussian.iqr(obj, detector=None, threshold=1.5, inclusive='both', sided='both', normaltest='ignore', pvalue=0.001, transform=None)[source]#
Bases:
_GaussianSeriesDetectorDetect outliers as potential errors in a Series using the IQR method.
Intended to be used by the detect method with the keyword ‘iqr’
>>> series.cleaner.detect.iqr(...) >>> series.cleaner.detect('iqr',...)
This detection method flag values as errors wherever the corresponding Series element is outside the range defined by:
[Q25 - threshold x IQR; Q75 + threshold x IQR]
where
Q25: the 25th percentile
Q75: the 75th percentile
IQR = Q75 - Q25: the interquartile range (the 75th percentile minus the 25th percentile)
threshold: cut-off value (default: 1.5)
The upper and lower are the values of the whiskers extremities in a boxplot.
Note
NA values are not treated as errors.
Warning
A normality test is performed [see https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.normaltest.html ]
If the distribution does not follow a gaussian/normal distribution:
This is ignored if normaltest=’ignore’ (default)
A warning is raised if normaltest = ‘warning’
An exception is raised if normaltest = ‘error’
If the series length is no more than 8, it is considered as not normal
Tip
The series can be “normalized” before applying the detector, using a power-series transformation:
Box-cox with a shift to deal with positive values [see https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.boxcox.html]
Yeo-Johnson [see https://docs.scipy.org/doc/scipy/reference/generated/scipy.stats.yeojohnson.html]
Using the scipy.stats implementations, the optimal parameter lambda is calculated and used for the transformations and its inverse functions.
When a transformation is applied, some parameters are expressed for the transformed series and informations are made available via report().
- Parameters:
threshold (float, default 1.5) –
inclusive ({“both”, “neither”, “left”, “right”}, default "both") – Include boundaries. Whether to set each bound as closed or open.
sided ({“both”, “left”, “right”}, default "both") – Specifies which limits should be applied. If “left”, only apply lower limit If “right”, only apply upper limit If “both”, apply both upper and lower limits
normaltest ({'ignore', 'warn', 'error'} default: 'ignore') – wether to ignore, raise a warning or raise en exception if the normality test fails
pvalue (float, default 1e-3) – pvalue associated with the normality test
- Raises:
TypeError – when threshold is not a number
TypeError – when pvalue is not a number
ValueError – when threshold is negative
ValueError – when pvalue is negative
ValueError – if sided or inclusive has an unvalid value
ValueError – if normaltest is not ‘ignore’, ‘warn’ or ‘error’
UserWarning – if the series is not normal and normaltest = ‘warn’
Exception – if the series is not normal and normaltest = ‘error’
Examples
>>> series = pd.Series([1,2,100,3]) >>> detector = series.cleaner.detect.iqr() >>> print(detector.is_error()) 0 False 1 False 2 True 3 False dtype: bool
Using a transformation
>>> s = pd.Series([0, 0, 0, 0, -100, 1, -1, 1, -6, 6]) >>> iqr_detector = s.cleaner.detect.iqr(transform='boxcox') >>> iqr_detector.report() Detection report ============================================================================== Method: iqr Nb samples: 10 Date: March 23,2022 Nb errors: 3 Time: 10:25:35 Nb rows with NaN: 0 ------------------------------------------------------------------------------ lower -3.045719539555506 upper 2.9570647561524908 inclusive both sided both ------------------------------------------------------------------------------ iqr parameters after boxcox transformation q25 7102.650530146201 q75 7325.937837774465 iqr 223.287307628264 threshold 1.5 transform boxcox lmbda 2.0840437865755472 ------------------------------------------------------------------------------ Series distribution is not normal/gaussian (A boxcox transformation has been applied) ==============================================================================
If the series is tested as normal, the transformation is not useful hence not applied
>>> s = pd.Series([0, 0, 0, 0, -1, 1, -1, 1, -6, 6]) >>> iqr_detector = s.cleaner.detect.iqr(transform='boxcox') >>> iqr_detector.report() Detection report ============================================================================== Method: iqr Nb samples: 10 Date: March 23,2022 Nb errors: 2 Time: 10:12:38 Nb rows with NaN: 0 ------------------------------------------------------------------------------ lower -3.0 upper 3.0 inclusive both sided both ------------------------------------------------------------------------------ iqr parameters q25 -0.75 q75 0.75 iqr 1.5 threshold 1.5 ------------------------------------------------------------------------------ Series distribution has been tested as normal with p=0.001(The transformation has not been applied) ==============================================================================
Attributes Summary
Keyword to indicate if boundaries are included {“both”, “neither”, “left”, “right”}
Indices of the rows detected as errors
Value of the interquantile range used to detect outliers
is the series normal/gaussian according to the test ?
The lambda that maximizes the log-likelihood function of the transformation
Lower bound
Number of rows detected as errors
Normality test result behavior
The object (Series or DataFrame) containing the data to which the detection is applied
pvalue for normality test
Value of the 25th percentile used to detect outliers
Value of the 75th percentile used to detect outliers
Keyword to indicate if detection is one side or both {"both", "right", "left"}
Threshold value used to detect outliers
Distribution transformation
Upper bound
Methods Summary
detected()Series or DataFrame containing only the detected errors
Returns True if any error has been detected, False otherwise
is_error()Return a boolean same-sized object indicating if the values are flagged as errors
Return a boolean same-sized object indicating if the values are NOT flagged as errors
plot([color, errors_color, compact, limits, ...])plot a visualization representing an overview of the treated data and colored according to the validity of the values:
report()prints a detection report
valid()Series or DataFrame containing only the valid values
Attributes Documentation
- inclusive#
Keyword to indicate if boundaries are included {“both”, “neither”, “left”, “right”}
- index#
Indices of the rows detected as errors
- iqr#
Value of the interquantile range used to detect outliers
- isnormal#
is the series normal/gaussian according to the test ?
- lmbda#
The lambda that maximizes the log-likelihood function of the transformation
- lower#
Lower bound
- n_errors#
Number of rows detected as errors
- name = 'iqr'#
- normaltest#
Normality test result behavior
- obj#
The object (Series or DataFrame) containing the data to which the detection is applied
- pvalue#
pvalue for normality test
- q25#
Value of the 25th percentile used to detect outliers
- q75#
Value of the 75th percentile used to detect outliers
- sided#
Keyword to indicate if detection is one side or both {“both”, “right”, “left”}
- threshold#
Threshold value used to detect outliers
- transform#
Distribution transformation
- upper#
Upper bound
Methods Documentation
- detected()#
Series or DataFrame containing only the detected errors
- has_errors() bool#
Returns True if any error has been detected, False otherwise
- is_error() Series#
Return a boolean same-sized object indicating if the values are flagged as errors
- not_error() Series#
Return a boolean same-sized object indicating if the values are NOT flagged as errors
- plot(color='green', errors_color='red', compact=False, limits=True, figsize=None)#
plot a visualization representing an overview of the treated data and colored according to the validity of the values:
a scatter plot representing the values in the treated series.
a histogram representing the distribution of values.
a kernel density estimate plot visualizing the distribution of values.
a boxplot showing the distribution of values.
- Parameters:
color (palette name (Default: "green")) – Color associated to legitimate values. Should be something that can be interpreted by seaborn’s color_palette()
errors_color (palette name (Default: "red")) – Color associated to erroneous values. Should be something that can be interpreted by seaborn’s color_palette()
compact (Bool (Default: False)) – If True, compact the plots around valid values and show the number of erroneous values on the scatter plot
limits (Bool (Default: True)) – If True, draw horizontal lines showing the lower and upper values delimiting the allowed values
figsize ((float, float) (Default: None)) – width and height of the figure.
- Returns:
axs – an array of length 4 containing the matplotlib axes representing the plots
- Return type:
array of matplotlib.axes._subplots.AxesSubplot
Examples
>>> series = pd.Series([-5, 1, 2 , 3, 8, 12]) >>> detector = series.cleaner.detect.bounded(lower=0, upper=10) >>> detector.plot()
- report()#
prints a detection report
- valid()#
Series or DataFrame containing only the valid values