U fH@sddlZddlmZddlZddlZddlZiaddZ ddZ ddZ d d Z d d Z iad dZddZddZddZddZddZddZd"ddZd#ddZd d!ZdS)$N)tqdmc Cs4td||||||f} d} dtkrXtddttd| DrXttd|krXd} t||\}}|jd|jdks~t|} t |j} t |j}t |j}t |jdd }t d d }t d d }t t t|d D]"}| r||||kr||||<q||dkr>||||dd||<q|| d d <||d d <t||d d f |}|d d |d ||f| d d |d ||f<|d d |d ||f|||d ||f<| | || |||dd||<q|td <|td <|td<| td<|||S)a The model is retrained for each test sample with the important features set to a constant. If you want to know how important a set of features is you can ask how the model would be different if those features had never existed. To determine this we can mask those features across the entire training and test datasets, then retrain the model. If we apply compare the output of this retrained model to the original model we can see the effect produced by knowning the features we masked. Since for individualized explanation methods each test sample has a different set of most important features we need to retrain the model for every test sample to get the change in model performance when a specified fraction of the most important features are withheld. IThe retrain based measures can incorrectly evaluate models in some cases!Fargscss|]\}}||kVqdSN.0abrr>/tmp/pip-target-lpfmz8o1/lib/python/shap/benchmark/measures.py sz!remove_retrain.. attr_testTư>nmaskNyp_masked_testz"Retraining for the 'remove' metricr)warningswarn _remove_cacheallzipnpto_arrayshapeAssertionErrorzeros const_randgetrrangelenpredictargsortmeanfit)rX_trainy_trainX_testy_testr model_generatormetric trained_model random_stater cache_match model_masked X_train_tmp X_test_tmprtie_breaking_noiseZ last_nmasklast_yp_masked_testiorderingrrr remove_retrain s@ .        84  r4c Cst||\}}|jd|jdks&t|} t|jd| d} |d} tt|D]T} || dkrXt || ddf | }| |d|| | | |d|| f<qX| | }|||S)zQ Each test sample is masked by setting the important features to a constant. rrrN) rrrcopyrr"rrrr!r )rr$r%r&r'r r(r)r*r+r/r0 mean_valsr2r3rrrr remove_maskHs  * r7c CsZt||\}}|jd|jdks&tt|j} | t| jdd7} |} t|j} t |jd| d} | d}t t |D]}||dkrt ||ddf | }|||d}|d||}tj| |ddfdd|f}| |ddfdd|f}|||||||f||}|| ||f<q|| } ||| S)az The model is revaluated for each test sample with the important features set to an imputed value. Note that the imputation is done using a multivariate normality assumption on the dataset. This depends on being able to estimate the full data covariance matrix (and inverse) accuractly. So X_train.shape[0] should be significantly bigger than X_train.shape[1]. rrrNrrrrZcovTeyer5rrr"rrr!Zlinalginvr )rr$r%r&r'r r(r)r*r+Cr/rr0r6r2r3 observe_inds impute_indsCoo_invCioimputerrr remove_impute^s&    $$ rBc Cs t||\}}|jd|jdks&td} |j\} } t|d| g| | | } t| d}tjj t | | | d}t | D]x}||dkr|t ||ddf |}||ddfdd|d||f| || |d| |d||f<q|| | }t|| | fd}|||S)zq The model is revaluated for each test sample with the important features set to resample background values. rdrZ n_samplesr+rNrrrrZtileZreshapersklearnutilsZresampleZarangerr!r r")rr$r%r&r'r r(r)r*r+nsamplesNMr/r0indsr2r3rrrr remove_resamples    N rLc CsFtdt||\}}|jd|jdks0t|} |d} t|jdd} tt |D]T} || dkr`t || ddf | }| |d|| | | |d|| f<q`|}tt |D]T} || dkrt || ddf | }| |d|| || |d|| f<q|}| | || |}| ||S)a An approximation of holdout that only retraines the model once. This is alse called ROAR (RemOve And Retrain) in work by Google. It is much more computationally efficient that the holdout method because it masks the most important features in every sample and then retrains the model once, instead of retraining the model for every test sample like the holdout metric. rrrrNrrrrrr5r"rrrrr!r#r )Z nmask_trainZ nmask_testr$r%r&r' attr_trainr r(r)r. X_train_meanr0r2r3r/r-yp_test_maskedrrr batch_remove_retrains&   * *  rQc Cs:td||||||f} d} dtkrXtddttd| DrXttd|krXd} t||\}}|jd|jdks~t|} t |j} t |j}t |j}t |jdd }t d d }t d d }t t t|d D](}| r||||kr||||<q|||jdkrD||||dd||<q|| d d <||d d <t||d d f |}|d d |||d f| d d |||d f<|d d |||d f|||||d f<| | || |||dd||<q|td <|td <|td<| td<|||S)a' The model is retrained for each test sample with the non-important features set to a constant. If you want to know how important a set of features is you can ask how the model would be different if only those features had existed. To determine this we can mask the other features across the entire training and test datasets, then retrain the model. If we apply compare the output of this retrained model to the original model we can see the effect produced by only knowning the important features. Since for individualized explanation methods each test sample has a different set of most important features we need to retrain the model for every test sample to get the change in model performance when a specified fraction of the most important features are retained. rFrcss|]\}}||kVqdSrrrrrr r szkeep_retrain..r TrrnkeepNrz Retraining for the 'keep' metricr)rr _keep_cacherrrrrrrrrrrrr r!r"r#)rRr$r%r&r'r r(r)r*r+rr,r-r.r/rr0Z last_nkeepr1r2r3rrr keep_retrains@ .        84  rTc Cst||\}}|jd|jdks&t|} t|j} t|jd| d} |d} tt |D]Z}|||jdkrdt ||ddf | }| |||d| ||||df<qd| | } ||| S)ze The model is revaluated for each test sample with the non-important features set to their mean. rrrN) rrrr5rrrr"rrr!r )rRr$r%r&r'r r(r)r*r+r/rr0r6r2r3rrr keep_masks  * rUc Cs`t||\}}|jd|jdks&tt|j} | t| jdd7} |} t|j} t |jd| d} | d}t t |D]}|||jdkrt ||ddf | }|d||}|||d}tj| |ddfdd|f}| |ddfdd|f}|||||||f||}|| ||f<q|| } ||| S)a~ The model is revaluated for each test sample with the non-important features set to an imputed value. Note that the imputation is done using a multivariate normality assumption on the dataset. This depends on being able to estimate the full data covariance matrix (and inverse) accuractly. So X_train.shape[0] should be significantly bigger than X_train.shape[1]. rrrNr8)rRr$r%r&r'r r(r)r*r+r<r/rr0r6r2r3r=r>r?r@rArrr keep_imputes&   $$ rVc Cs t||\}}|jd|jdks&td} |j\} } t|d| g| | | } t| d}tjj t | | | d}t | D]x}||| kr|t ||ddf |}||ddfdd|||df| || |d| |||df<q|| | }t|| | fd}|||S)zu The model is revaluated for each test sample with the non-important features set to resample background values. rrCrrDNrE)rRr$r%r&r'r r(r)r*r+rHrIrJr/r0rKr2r3rrrr keep_resample@s    N rWc CsRtdt||\}}|jd|jdks0t|} |d} t|jdd} tt |D]Z} || |jdkr`t || ddf | }| ||| d| | ||| df<q`|}tt |D]Z} || |jdkrt || ddf | }| ||| d|| ||| df<q|}| | || |}| ||S)ax An approximation of keep that only retraines the model once. This is alse called KAR (Keep And Retrain) in work by Google. It is much more computationally efficient that the keep method because it masks the unimportant features in every sample and then retrains the model once, instead of retraining the model for every test sample like the keep metric. rrrrNrM)Z nkeep_trainZ nkeep_testr$r%r&r'rNr r(r)r.rOr0r2r3r/r-rPrrr batch_keep_retrain[s&  **  rXc CsDt||\}}|jd|jdks&t||}||t|dS)zX The how well do the features plus a constant base rate sum up to the model output. r)rrrr strip_listsum) r$r%r&r'r r(r)r*Zyp_testrrr local_accuracys r[cGsdd|DS)NcSs(g|] }tt|dr |jn|qS)z'pandas.core.frame.DataFrame'>)strtypeendswithvalues)rr rrr szto_array..r)rrrr rsr]cCs2tj}tj|tj|}tj||S)z0 Generate a random array with a fixed seed. )rrandomseedZrand)sizercold_seedoutrrr rs     rcCs2tj}tj|tj|tj|dS)z2 Shuffle an array in-place with a fixed seed. N)rrbrcshuffle)Zarrrcrerrr const_shuffles   rhcCst|tr|dS|SdS)zw This assumes that if you have a list of outputs you just want the second one (the second class is the '1' class). rN) isinstancelist)attrsrrr rYs rY)ra)ra)numpyrZtqdm.autonotebookrgcrZ sklearn.utilsrFrr4r7rBrLrQrSrTrUrVrWrXr[rrrhrYrrrr s* ?%%@%%