U f/@sLddlZddlZddlZddlmZddlmZGdddeZ ddZ dS) N)tqdm) Explainerc@s*eZdZdZd ddZddZdd ZdS) LinearExplainera Computes SHAP values for a linear model, optionally accounting for inter-feature correlations. This computes the SHAP values for a linear model and can account for the correlations among the input features. Assuming features are independent leads to interventional SHAP values which for a linear model are coef[i] * (x[i] - X.mean(0)[i]) for the ith feature. If instead we account for correlations then we prevent any problems arising from colinearity and share credit among correlated features. Accounting for correlations can be computationally challenging, but LinearExplainer uses sampling to estimate a transform that can then be applied to explain any prediction of the model. Parameters ---------- model : (coef, intercept) or sklearn.linear_model.* User supplied linear model either as either a parameter pair or sklearn object. data : (mean, cov), numpy.array, pandas.DataFrame, iml.DenseData or scipy.csr_matrix The background dataset to use for computing conditional expectations. Note that only the mean and covariance of the dataset are used. This means passing a raw data matrix is just a convienent alternative to passing the mean and covariance directly. nsamples : int Number of samples to use when estimating the transformation matrix used to account for feature correlations. feature_dependence : "independent" (default) or "correlation" There are two ways we might want to compute SHAP values, either the full conditional SHAP values or the independent SHAP values. For independent SHAP values we break any dependence structure between features in the model and so uncover how the model would behave if we intervened and changed some of the inputs. For the full conditional SHAP values we respect the correlations among the input features, so if the model depends on one input but that input is correlated with another input, then both get some credit for the model's behavior. The independent option stays "true to the model" meaning it will only give credit to features that are actually used by the model, while the correlation option stays "true to the data" in the sense that it only considers how the model would behave when respecting the correlations in the input data. For sparse case only independent option is supported. Nc Csf||_|dkrtdd}n|dkr4tdd}||_t|tkrht|dkrh|d|_|d|_ntt |drt |d rt|j j dkr|j j ddkr|j d|_|j d|_q|j |_|j |_nt d tt|tt|d r|j}t|tkr&t|dkr&|d|_|d|_nt|dkr:t d n`tj|rh|d|_|dkrt d n2tt|d|_|dkrtj|dd|_tj|jstt|jdrtj|jst|j|_|j|jj|j|_nt|j|j|j|_t|j|_|dkr6tt|jdkd|_ |j|j |_|jdd|j f|j ddf|_|j|j |_t!|j\|_"}t#t#|j"|j|j"j|_t#|j"|j|_t#||j|_tj$%|j\}}|&dkr|jt'|jj dd|_|(|\}} t#||j|_)| |_*n,|dkrV|dkrbtdn t d|dS)NZinterventionalzgThe option feature_dependence="interventional" is has been renamed to feature_dependence="independent"! independentzKThe default value for feature_dependence has been changed to "independent"!rrcoef_ intercept_z"An unknown model type was passed: 'pandas.core.frame.DataFrame'>z0A background data distribution must be provided!DOnly feature_dependence = 'independent' is supported for sparse data correlationF)Zrowvarzmatrix'>:0yE>gHz>gư>rzGSetting nsamples has no effect when feature_dependence = 'independent'!z-Unknown type of feature_dependence provided: )+nsampleswarningswarnfeature_dependencetypetuplelencoefZ intercepthasattrr shaper ExceptionstrendswithvaluesmeancovspsparseissparsenparrayflattenZasmatrixdotTZexpected_valueMwherediag valid_indsduplicate_componentsavg_projmatmulZlinalgZeigmineye_estimate_transformsmean_transformed x_transform) selfmodeldatarrZsum_proje_mean_transformr2r9=/tmp/pip-target-lpfmz8o1/lib/python/shap/explainers/linear.py__init__+sr               &  $   zLinearExplainer.__init__c Cst|j}t||f}t||f}tj|tjd}tt|dD]*}tj |td}t|df}t|D]} || } |} |} |j dd|d| df}|| ddfj } t | | }|j | | f}|t |j | }t| d| df}| dkrR| t ||||ddddf<| ||dddf<|dddf<d||d<|| | f|j| 7<t |j|| ddt |||| dd}|| |d| df|7<t |j|| dt | | || d}|| |d| f|8<|| | f|j| 7<|| |d| df|7<|| |d| f|8<qvqD||}||}||fS) a< Uses block matrix inversion identities to quickly estimate transforms. After a bit of matrix math we can isolate a transform matrix (# features x # features) that is independent of any sample we are explaining. It is the result of averaging over all feature permutations, but we just use a fixed number of samples to estimate the value. TODO: Do a brute force enumeration when # feature subsets is less than nsamples. This could happen through a recursive method that uses the same block matrix inversion as below. ZdtypezEstimating transformsrrrNr)r>r>)rrr"zerosZarangeintrrangerandomshufflerr&r-outer)r3rr'r8r2Zindsr7Z cov_inv_SiSiZcov_SijiZcov_SZ cov_inv_SSdtZuZ coef_R_SiZcoef_R_Sr9r9r:r0sB     $& 6 .  z$LinearExplainer._estimate_transformscsttdrjnttdr2jtjdksVtjdksVtdjdkrtj rtt dt t ddjfjjjjj}t |j}t |jd jf}||ddjf<|Sjd krtj rJtjjdkr(t t jjd Sfd d tjjd DSnHtjjdkrrt jjSfd d tjjd DSdS)aR Estimate the SHAP values for a set of samples. Parameters ---------- X : numpy.array, pandas.DataFrame or scipy.csr_matrix A matrix of samples (# samples x # features) on which to explain the model's output. Returns ------- For models with a single output this returns a matrix of SHAP values (# samples x # features). Each row sums to the difference between the model output for that sample and the expected value of the model output (which is stored as expected_value attribute of the explainer). zpandas.core.series.Series'>r rrz%Instance must have 1 or 2 dimensions!r r Nrrc s*g|]"}ttjj|qSr9)r"r#multiplyrr.0rFXr3r9r: sz/LinearExplainer.shap_values..cs&g|]}tjj|qSr9)r"r#rrrLrNr9r:rPs)rrrrrrAssertionErrorrrr r!rr"r-r*r,r&r2r1r?r'rr#rKrrA)r3rOphiZfull_phir9rNr: shap_valuess*$  0 "zLinearExplainer.shap_values)rN)__name__ __module__ __qualname____doc__r;r0rSr9r9r9r:rs# T@rcCs0tdtt|}tt|||}tj|jdtjd }d}t|jdD]r}d}t|jdD]Z}||dkrntd|||f|||f|||fdkrn|s|d7}d}|||<qnqXt t t ||jdf}d|d <td|jdD]}d||||f<q|j | dj |fS) Nrrr<r>FrrTr=)r"r)sqrtr-Zonesrr@rAabsr?runiquer&sum)CD componentscountrFZ found_grouprEZprojr9r9r:r+s"> r+) numpyr"ZscipyrrZtqdm.autonotebookrZ explainerrrr+r9r9r9r:s  i