B \`@sdZddlZddlZddlZddlmZddlmZddlm Z ddl m Z m Z ddl mZddlmZdd lmZdd lmZdd lmZdd l mZdd lmZejjZejjZddgZddZddZddZGddde e Z Gddde e Z!dS)z)Transformers for missing value imputationN)sparse)stats) BaseEstimatorTransformerMixin) check_array) _get_median)check_is_fitted) FLOAT_DTYPES)_object_dtype_isnan) is_scalar_nan)sixMissingIndicator SimpleImputercCs2|jjdkr.t|tjs.td|jt|dS)N)fiuzn'X' and 'missing_values' types are expected to be both numerical. Got X.dtype={} and type(missing_values)={}.)dtypekind isinstancenumbersReal ValueErrorformattype)Xmissing_valuesr-lib/python3.7/site-packages/sklearn/impute.py_check_inputs_dtype!s  rcCsTt|rD|jjdkrt|S|jjdkr:tj|jtdSt|Sn t ||SdS)z-Compute the boolean mask X == missing_values.r)rr)rN) r rrnpZisnanZzerosshapeboolr Zequal)rZ value_to_maskrrr _get_mask*s    r#c Cs|jdkrNttdtt|}WdQRX|dd}|dd}nd}d}|dkrl|dkrltjS||krx|S||kr|S||kr||kr|S|SdS)zCompute the most frequent value in a 1d array extended with [extra_value] * n_repeat, where extra_value is assumed to be not part of the array.rignoreNr) sizewarningscatch_warnings simplefilterRuntimeWarningrmoder nan)ZarrayZ extra_valueZn_repeatr*Zmost_frequent_valueZmost_frequent_countrrr_most_frequent;s$    r,c@sPeZdZdZejddddfddZdd Zdd d Zd d Z ddZ ddZ dS)ra_ Imputation transformer for completing missing values. Read more in the :ref:`User Guide `. Parameters ---------- missing_values : number, string, np.nan (default) or None The placeholder for the missing values. All occurrences of `missing_values` will be imputed. strategy : string, optional (default="mean") The imputation strategy. - If "mean", then replace missing values using the mean along each column. Can only be used with numeric data. - If "median", then replace missing values using the median along each column. Can only be used with numeric data. - If "most_frequent", then replace missing using the most frequent value along each column. Can be used with strings or numeric data. - If "constant", then replace missing values with fill_value. Can be used with strings or numeric data. .. versionadded:: 0.20 strategy="constant" for fixed value imputation. fill_value : string or numerical value, optional (default=None) When strategy == "constant", fill_value is used to replace all occurrences of missing_values. If left to the default, fill_value will be 0 when imputing numerical data and "missing_value" for strings or object data types. verbose : integer, optional (default=0) Controls the verbosity of the imputer. copy : boolean, optional (default=True) If True, a copy of X will be created. If False, imputation will be done in-place whenever possible. Note that, in the following cases, a new copy will always be made, even if `copy=False`: - If X is not an array of floating values; - If X is encoded as a CSR matrix. Attributes ---------- statistics_ : array of shape (n_features,) The imputation fill value for each feature. Examples -------- >>> import numpy as np >>> from sklearn.impute import SimpleImputer >>> imp_mean = SimpleImputer(missing_values=np.nan, strategy='mean') >>> imp_mean.fit([[7, 2, 3], [4, np.nan, 6], [10, 5, 9]]) ... # doctest: +NORMALIZE_WHITESPACE SimpleImputer(copy=True, fill_value=None, missing_values=nan, strategy='mean', verbose=0) >>> X = [[np.nan, 2, 3], [4, np.nan, 6], [10, np.nan, 9]] >>> print(imp_mean.transform(X)) ... # doctest: +NORMALIZE_WHITESPACE [[ 7. 2. 3. ] [ 4. 3.5 6. ] [10. 3.5 9. ]] Notes ----- Columns which only contained missing values at `fit` are discarded upon `transform` if strategy is not "constant". meanNrTcCs"||_||_||_||_||_dS)N)rstrategy fill_valueverbosecopy)selfrr.r/r0r1rrr__init__s zSimpleImputer.__init__c Csddddg}|j|kr(td||j|jdkr8d}nt}t|jsLd}nd}yt|d |||jd }WnJtk r}z,d t|krtd |j|j j n|Wdd}~XYnXt ||j|j j d krtd|j |S)Nr-median most_frequentconstantz4Can only use these strategies: {0} got strategy={1})r5r6Tz allow-nancsc) accept_sparserforce_all_finiter1zcould not convertzFCannot use {0} strategy with non-numeric data. Received datatype :{1}.)rrrOzSimpleImputer does not support data with dtype {0}. Please provide either a numeric array (with a floating point or integer dtype) or categorical data represented either as an array with integer dtype or an array of string values with an object dtype.) r.rrr r rrr1strrrr)r2rZallowed_strategiesrr9Zverrr_validate_inputs0         zSimpleImputer._validate_inputcCs||}|jdkr,|jjdkr&d}q2d}n|j}|jdkrb|jjdkrbt|tjsbtd |t |r|j dkrtdq| ||j|j ||_n|||j|j ||_|S)aIFit the imputer on X. Parameters ---------- X : {array-like, sparse matrix}, shape (n_samples, n_features) Input data, where ``n_samples`` is the number of samples and ``n_features`` is the number of features. Returns ------- self : SimpleImputer N)rrrrZ missing_valuer6zT'fill_value'={0} is invalid. Expected a numerical value when imputing numerical datazdImputation not possible when missing_values == 0 and input is sparse. Provide a dense array instead.)r<r/rrr.rrrrrrissparser _sparse_fit statistics_ _dense_fit)r2ryr/rrrfits.          zSimpleImputer.fitcCs2t|j|}|jdt|j}t|jd}|dkrF||nxt|jdD]}|j|j||j|d} ||j||j|d} | | } t| d} | | } | } ||| } |dkr| j | }|dkrtj n | |||<qV|dkrt | | ||<qV|dkrVt | d| ||<qVW|S)z#Fit the transformer on sparse data.rrr6r-r4r5)r#datar!r diffindptremptyZfillrangesumr%r+rr,)r2rr.rr/Z mask_dataZn_implicit_zeros statisticsrcolumnZ mask_columnZ mask_zerosZn_explicit_zerosZn_zerossrrrr>s0          zSimpleImputer._sparse_fitcCs^t||}tj||d}|dkrRtjj|dd}tj|}tj|tj|<|S|dkrtjj|dd} tj| } tj| tj | <| S|dkr8| }| }|j j dkrtj |jdtd} nt |jd} xZtt|d d |d d D]8\} \} }t|tj}| |} t| tjd| | <qW| S|d krZtj|jd ||j dSd S) z"Fit the transformer on dense data.)maskr-r)axisr4r5r:)rNr6r)r#maZ masked_arrayr r-Zgetdatar+Zgetmaskr4Z getmaskarray transposerrrFr!object enumeratezip logical_notastyper"r,Zfull)r2rr.rr/rLZmasked_XZ mean_maskedr-Z median_maskedr4r5rrowZrow_maskrrrr@-s2     , zSimpleImputer._dense_fitc Cst|d||}|j}|jd|jdkrLtd|jd|jjdf|jdkr\|}nht|tj}t |}||}t |}| rt |jd|}|j rtd||dd|f}t|r<|jdkrtdnVt|j|j}ttj t|jdtjd t|j|} || j|jd d |j|<nFt||j}tj|dd } t|| } t|ddd } | || <|S)zImpute all missing values in X. Parameters ---------- X : {array-like, sparse matrix}, shape (n_samples, n_features) The input data to complete. r?rrz)X has %d features per sample, expected %dr6z-Deleting features without observed values: %sNzdImputation not possible when missing_values == 0 and input is sparse. Provide a dense array instead.)rF)r1)rM)r r<r?r!rr.r#r r+rS flatnonzeroanyaranger0r&warnrr=rrCrepeatlenrEintrDrTrrHwhererO) r2rrIZvalid_statisticsZ invalid_maskZ valid_maskZvalid_statistics_indexesZmissingrLZindexesZ n_missingvaluesZ coordinatesrrr transformas@            zSimpleImputer.transform)N) __name__ __module__ __qualname____doc__r r+r3r<rBr>r@r`rrrrr]sE' 5&4c@sPeZdZdZejdddfddZddZd d Zdd d Z ddZ dddZ d S)ra Binary indicators for missing values. Note that this component typically should not not be used in a vanilla :class:`Pipeline` consisting of transformers and a classifier, but rather could be added using a :class:`FeatureUnion` or :class:`ColumnTransformer`. Read more in the :ref:`User Guide `. Parameters ---------- missing_values : number, string, np.nan (default) or None The placeholder for the missing values. All occurrences of `missing_values` will be indicated (True in the output array), the other values will be marked as False. features : str, optional Whether the imputer mask should represent all or a subset of features. - If "missing-only" (default), the imputer mask will only represent features containing missing values during fit time. - If "all", the imputer mask will represent all features. sparse : boolean or "auto", optional Whether the imputer mask format should be sparse or dense. - If "auto" (default), the imputer mask will be of same type as input. - If True, the imputer mask will be a sparse matrix. - If False, the imputer mask will be a numpy array. error_on_new : boolean, optional If True (default), transform will raise an error when there are features with missing values in transform that have no missing values in fit. This is applicable only when ``features="missing-only"``. Attributes ---------- features_ : ndarray, shape (n_missing_features,) or (n_features,) The features indices which will be returned when calling ``transform``. They are computed during ``fit``. For ``features='all'``, it is to ``range(n_features)``. Examples -------- >>> import numpy as np >>> from sklearn.impute import MissingIndicator >>> X1 = np.array([[np.nan, 1, 3], ... [4, 0, np.nan], ... [8, 1, 0]]) >>> X2 = np.array([[5, 1, np.nan], ... [np.nan, 2, 3], ... [2, 4, 0]]) >>> indicator = MissingIndicator() >>> indicator.fit(X1) MissingIndicator(error_on_new=True, features='missing-only', missing_values=nan, sparse='auto') >>> X2_tr = indicator.transform(X2) >>> X2_tr array([[False, True], [ True, False], [False, False]]) z missing-onlyautoTcCs||_||_||_||_dS)N)rfeaturesr error_on_new)r2rrfrrgrrrr3szMissingIndicator.__init__cCst|r|jdkrt|j|j}|jdkr2tjntj}|||j |j f|j t d}| }| |jdkrtt|j n t|j}|jdkr|}q|jdkr|}nDt|r|}t||j}t|jdd}|jdkrt|}||fS)aCompute the imputer mask and the indices of the features containing missing values. Parameters ---------- X : {ndarray or sparse matrix}, shape (n_samples, n_features) The input data with missing values. Note that ``X`` has been checked in ``fit`` and ``transform`` before to call this function. Returns ------- imputer_mask : {ndarray or sparse matrix}, shape (n_samples, n_features) or (n_samples, n_features_with_missing) The imputer mask of the original data. features_with_missing : ndarray, shape (n_features_with_missing) The features containing missing values. rcsr)r!rr7F)rMT)rr=rr#rCrZ csr_matrixZ csc_matrixindicesr1rEr!r"Zeliminate_zerosr rWrDuniqueZtoarrayZtocscrH)r2rrLZsparse_constructor imputer_maskZmissing_values_maskZfeatures_with_missingrrr_get_missing_features_infos,          z+MissingIndicator._get_missing_features_infocCsPt|jsd}nd}t|dd|d}t||j|jjdkrLtd|j|S)NTz allow-nan)r7rh)r8rr9)rrrr:zMissingIndicator does not support data with dtype {0}. Please provide either a numeric array (with a floating point or integer dtype) or categorical data represented either as an array with integer dtype or an array of string values with an object dtype.)r rrrrrrr)r2rr9rrrr<s    z MissingIndicator._validate_inputNcCs||}|jd|_|jdkr0td|jt|jtj rH|jdksdt|jt sdtd|j|jdkr|| |dn t |j|_|S)a`Fit the transformer on X. Parameters ---------- X : {array-like, sparse matrix}, shape (n_samples, n_features) Input data, where ``n_samples`` is the number of samples and ``n_features`` is the number of features. Returns ------- self : object Returns self. r)z missing-onlyallzD'features' has to be either 'missing-only' or 'all'. Got {} instead.rez9'sparse' has to be a boolean or 'auto'. Got {!r} instead.z missing-only)r<r! _n_featuresrfrrrrr Z string_typesr"rlr rY features_)r2rrArrrrB-s     zMissingIndicator.fitcCst|d||}|jd|jkr,td||\}}|jdkrt||j }|j rp|j dkrptd ||j j dkr|j j |jkr|dd|j f}|S)aGenerate missing values indicator for X. Parameters ---------- X : {array-like, sparse matrix}, shape (n_samples, n_features) The input data to complete. Returns ------- Xt : {ndarray or sparse matrix}, shape (n_samples, n_features) The missing indicator for input data. The data type of ``Xt`` will be boolean. rorz9X has a different number of features than during fitting.z missing-onlyrzSThe features {} have missing values in transform but have no missing values in fit.N) r r<r!rnrrlrfr Z setdiff1drorgr%r)r2rrkrfZfeatures_diff_fit_transrrrr`Ms    zMissingIndicator.transformcCs||||S)aGenerate missing values indicator for X. Parameters ---------- X : {array-like, sparse matrix}, shape (n_samples, n_features) The input data to complete. Returns ------- Xt : {ndarray or sparse matrix}, shape (n_samples, n_features) The missing indicator for input data. The data type of ``Xt`` will be boolean. )rBr`)r2rrArrr fit_transformrszMissingIndicator.fit_transform)N)N) rarbrcrdr r+r3rlr<rBr`rprrrrrs@7 %)"rdr&rZnumpyr Znumpy.marNZscipyrrbaserrZutilsrZutils.sparsefuncsrZutils.validationr r Z utils.fixesr r Z externalsr ZmovesrRmap__all__rr#r,rrrrrrs2           "A