U Ùf~"ã@sâddlZddlZddlZddlZzddlmZWn e k rPddl mZYnXdZ d"dd„Z d#dd „Z d$d d „Zd%d d „Zd&dd„Zd'dd„Zd(dd„Zd)dd„Zd*dd„Zd+dd„Zd,dd„Zdd„Zdd„Zd-d d!„ZdS).éN)Ú urlretrievez2https://github.com/slundberg/shap/raw/master/data/FéàcCsDtd}t t|d||fƒ¡ tj¡}t t|dƒ¡}||fS)aj This is a set of 50 images representative of ImageNet images. This dataset was collected by randomly finding a working ImageNet link and then pasting the original ImageNet image into Google image search restricted to images licensed for reuse. A similar image (now with rights to reuse) was downloaded as a rough replacment for the original ImageNet image. The point is to have a random sample of ImageNet for use as a background distribution for explaining models trained on ImageNet data. Note that because the images are only rough replacements the labels might no longer be correct. Z imagenet50_z %sx%s.npyz labels.csv)Úgithub_data_urlÚnpÚloadÚcacheZastypeÚfloat32Úloadtxt)ÚdisplayÚ resolutionÚprefixÚXÚy©rú4/tmp/pip-target-lpfmz8o1/lib/python/shap/datasets.pyÚ imagenet50 s "rcCs&tj ¡}tj|j|jd}||jfS)z3 Return the boston housing data in a nice package. ©ÚdataÚcolumns)ÚsklearnÚdatasetsZ load_bostonÚpdÚ DataFramerÚ feature_namesÚtarget©r ÚdÚdfrrrÚbostons rc CsHtttdƒƒ}| ¡}W5QRXtjdtjd}d|dd…<||fS)zø Return the clssic IMDB sentiment analysis training data in a nice package. Full data is at: http://ai.stanford.edu/~amaas/data/sentiment/aclImdb_v1.tar.gz Paper to cite when using the data is: http://www.aclweb.org/anthology/P11-1015 zimdb_train.txti¨a©ÚdtyperNiÔ0)ÚopenrrÚ readlinesrZonesÚbool)r ÚfrrrrrÚimdb%s  r%c Cs¤tjttdƒdd}t t t |jdd…df¡¡¡d}tj |j|dftj d}|j|dd …f}t t |j ¡  d¡dk¡d}|jdd…|f}||fS) zÚ Predict total number of non-violent crimes per 100K popuation. This dataset is from the classic UCI Machine Learning repository: https://archive.ics.uci.edu/ml/datasets/Communities+and+Crime+Unnormalized z CommViolPredUnnormalizedData.txtú?)Ú na_valuesNéþÿÿÿrréiîÿÿÿ) rÚread_csvrrrÚwhereÚinvertÚisnanZilocÚarrayÚfloatÚvaluesÚsum)r Úraw_dataZ valid_indsrr Z valid_colsrrrÚcommunitiesandcrime2s þ( r3cCs&tj ¡}tj|j|jd}||jfS)z- Return the diabetes data in a nice package. r)rrZ load_diabetesrrrrrrrrrÚdiabetesIs r4csFtj ¡‰tjˆjˆjd}|r8|‡fdd„ˆjDƒfS|ˆjfSdS)z1 Return the classic iris data in a nice package. rcsg|]}ˆj|‘qSr)Z target_names©Ú.0Úv©rrrÚ Wsziris..N)rrZ load_irisrrrrr)r rrr8rÚirisQs  r:cs"ddddddddd d d d d ddg}tjttdƒdd„|Dƒdt|ƒd}|jdgdd}ttdd„|ƒƒ}|ddk|d<dddddd d!œ‰|D]J\}}|d"kr’|d#krÌt  ‡fd$d„||Dƒ¡||<q’||j j ||<q’|r|jddd%gdd|dj fS|jdd%gdd|dj fSd&S)'z1 Return the Adult census data in a nice package. )ZAger)Z WorkclassÚcategory)Úfnlwgtr)Ú Educationr;)z Education-Numr)zMarital Statusr;)Z Occupationr;)Ú Relationshipr;)ZRacer;)ÚSexr;)z Capital Gainr)z Capital Lossr)zHours per weekr)ZCountryr;)ÚTargetr;z adult.datacSsg|] }|d‘qS)rr)r6rrrrr9gszadult..r&)Únamesr'r r=é)ZaxiscSs |ddkS)Nr)r@r=r)ÚxrrrÚlózadult..r@z >50Krééér))z Not-in-familyZ UnmarriedzOther-relativez Own-childZHusbandZWifer;r>csg|]}ˆ| ¡‘qSr)Ústripr5©Zrcoderrr9ysr<N) rr*rrÚdictZdropÚlistÚfilterrr.ÚcatÚcodesr0)r Zdtypesr2rZ filt_dtypesÚkr rrJrÚadult\sPû  üú "rQcCsjt ttdƒ¡}t ttdƒ¡d}|rX| ¡}dd„|dDƒ|d<|t |¡fS|t |¡fSdS)zP A nicely packaged version of NHANES I data with surivival times as labels. zNHANESI_subset_X.csvzNHANESI_subset_y.csvrcSsg|]}|dkrdnd‘qS)rBZMaleZFemalerr5rrrr9Šsznhanesi..r?N)rr*rrÚcopyrr.©r r rZ X_displayrrrÚnhanesiƒsrTcCsDt ttdƒ¡}t ttdƒ¡}|r8| ¡}||fS||fSdS)zb A nicely packaged version of CRIC data with progression to ESRD within 4 years as the label. zCRIC_time_4yearESRD_X.csvzCRIC_time_4yearESRD_y.csvN)rr*rrrr rRrSrrrÚcrics rUcsœtj ¡}tj d¡d}d}t |¡‰dˆddd…<t |¡}tdddƒD]l}d|||df<||d|f<d|||df<||d|f<d||d|df<||d|df<qL‡fd d „}tj ||¡}|| d¡}t |j |¡|j d} tj   tj   | ¡¡j } t || j ¡} tj  t t || j ¡j ¡t |¡¡d ksPt‚t | tj   |¡j ¡} | } || ƒtj |¡d }tj |¡t | ¡|fS) zr Correlated Groups 60 A simulated dataset with tight correlations among distinct groups of features. réèé<rBérGg®Gáz®ï?rFcs t |ˆ¡S©N©rÚmatmul©r ©ÚbetarrrD³rEzcorrgroups60..gíµ ÷Æ°>ç{®Gáz„?)rÚrandomÚseedÚzerosÚeyeÚrangeÚrandnÚmeanr[ÚTÚshapeZlinalgZcholeskyÚinvZnormZcorrcoefÚAssertionErrorrr)r Úold_seedÚNÚMÚCÚir$ÚX_startZ X_centeredÚSigmaÚWZX_whiteZX_finalr rrr]rÚ corrgroups60›s.      * 2 rscstj ¡}tj d¡d}d}t |¡‰dˆddd…<‡fdd„}tj ||¡}|| d¡}||ƒtj |¡d }tj |¡t |¡|fS) zT A simulated dataset with tight correlations among distinct groups of features. rrVrWrBrXrGcs t |ˆ¡SrYrZr\r]rrrD×rEz%independentlinear60..r_)rr`rarbrerfrr)r rkrlrmr$rpr rrr]rÚindependentlinear60Ès     rtcCstj ttdƒ¡S)z2 A sparse dataset in scipy csr matrix format. z a1a.svmlight)rrÚload_svmlight_filerrrrrrÚa1aäsrvcCshd}tj t|dƒ¡\}}tj t|dƒ¡\}}t t|dƒ¡}t t|dƒ¡}||||||fS)z0 Ranking datasets from lightgbm repository. zPhttps://raw.githubusercontent.com/Microsoft/LightGBM/master/examples/lambdarank/z rank.trainz rank.testzrank.train.queryzrank.test.query)rrrurrr )Z rank_data_urlZx_trainZy_trainZx_testZy_testZq_trainZq_testrrrÚrankês rwcCsh|dkrtj |¡}tj tj t¡d¡}tj |¡s@t |¡tj ||¡}tj |¡sdt ||ƒ|S)NZ cached_data) ÚosÚpathÚbasenameÚjoinÚdirnameÚ__file__ÚisdirÚmkdirÚisfiler)ÚurlÚ file_nameÚdata_dirÚ file_pathrrrrõs     r)Fr)F)F)F)F)F)F)F)F)F)F)N)ZpandasrÚnumpyrZsklearn.datasetsrrxÚurllib.requestrÚ ImportErrorÚurllibrrrr%r3r4r:rQrTrUrsrtrvrwrrrrrÚs,     ' -