B Zg@s~dZddlmZddlmZmZddlZddlm Z ddl m Z m Z m Z mZmZdd d gZGd dde ZGd d d e ZdS) aZ Multivariate Conditional and Unconditional Kernel Density Estimation with Mixed Data Types. References ---------- [1] Racine, J., Li, Q. Nonparametric econometrics: theory and practice. Princeton University Press. (2007) [2] Racine, Jeff. "Nonparametric Econometrics: A Primer," Foundation and Trends in Econometrics: Vol 3: No 1, pp1-88. (2008) http://dx.doi.org/10.1561/0800000009 [3] Racine, J., Li, Q. "Nonparametric Estimation of Distributions with Categorical and Continuous Data." Working Paper. (2000) [4] Racine, J. Li, Q. "Kernel Estimation of Multivariate Conditional Distributions Annals of Economics and Finance 5, 211-235 (2004) [5] Liu, R., Yang, L. "Kernel estimation of multivariate cumulative distribution function." Journal of Nonparametric Statistics (2008) [6] Li, R., Ju, G. "Nonparametric Estimation of Multivariate CDF with Categorical and Continuous Data." Working Paper [7] Li, Q., Racine, J. "Cross-validated local linear nonparametric regression" Statistica Sinica 14(2004), pp. 485-512 [8] Racine, J.: "Consistent Significance Testing for Nonparametric Regression" Journal of Business & Economics Statistics [9] Racine, J., Hart, J., Li, Q., "Testing the Significance of Categorical Predictor Variables in Nonparametric Regression Models", 2006, Econometric Reviews 25, 523-544 )division)rangenextN)kernels) GenericKDEEstimatorSettingsgpke LeaveOneOut _adjust_shapeKDEMultivariateKDEMultivariateConditionalrc@s\eZdZdZdefddZddZddfd d Zdd d Zdd dZ ddZ ddZ dS)r a Multivariate kernel density estimator. This density estimator can handle univariate as well as multivariate data, including mixed continuous / ordered discrete / unordered discrete data. It also provides cross-validated bandwidth selection methods (least squares, maximum likelihood). Parameters ---------- data: list of ndarrays or 2-D ndarray The training data for the Kernel Density Estimation, used to determine the bandwidth(s). If a 2-D array, should be of shape (num_observations, num_variables). If a list, each list element is a separate observation. var_type: str The type of the variables: - c : continuous - u : unordered (discrete) - o : ordered (discrete) The string should contain a type specifier for each variable, so for example ``var_type='ccuo'``. bw: array_like or str, optional If an array, it is a fixed user-specified bandwidth. If a string, should be one of: - normal_reference: normal reference rule of thumb (default) - cv_ml: cross validation maximum likelihood - cv_ls: cross validation least squares defaults: EstimatorSettings instance, optional The default values for (efficient) bandwidth estimation. Attributes ---------- bw: array_like The bandwidth parameters. See Also -------- KDEMultivariateConditional Examples -------- >>> import statsmodels.api as sm >>> nobs = 300 >>> np.random.seed(1234) # Seed random generator >>> c1 = np.random.normal(size=(nobs,1)) >>> c2 = np.random.normal(2, 1, size=(nobs,1)) Estimate a bivariate distribution and display the bandwidth found: >>> dens_u = sm.nonparametric.KDEMultivariate(data=[c1,c2], ... var_type='cc', bw='normal_reference') >>> dens_u.bw array([ 0.39967419, 0.38423292]) NcCs|||_t|j|_t||j|_||_t|j\|_|_|j|jkrNt d| ||j sl| ||_ n |||_ dS)NzGThe number of observations must be larger than the number of variables.)var_typelenk_varsr data data_typenpshapenobs ValueError _set_defaults efficient _compute_bwbw_compute_efficient)selfrrrdefaultsrGlib/python3.7/site-packages/statsmodels/nonparametric/kernel_density.py__init__hs   zKDEMultivariate.__init__cCsXd}|dt|jd7}|dt|jd7}|d|jd7}|d|jd7}|S)z Provide something sane to print.z KDE instance zNumber of variables: k_vars =  zNumber of samples: nobs = zVariable types: zBW selection method: )strrrr _bw_method)rrprrrr__repr__xs zKDEMultivariate.__repr__cCs|S)Nr)xrrrszKDEMultivariate.cCsZt|j}d}xDt|D]8\}}t|| |j|ddf |jd}|||7}qW| S)aV Returns the leave-one-out likelihood function. The leave-one-out likelihood function for the unconditional KDE. Parameters ---------- bw: array_like The value for the bandwidth parameter(s). func: callable, optional Function to transform the likelihood values (before summing); for the log likelihood, use ``func=np.log``. Default is ``f(x) = x``. Notes ----- The leave-one-out kernel estimator of :math:`f_{-i}` is: .. math:: f_{-i}(X_{i})=\frac{1}{(n-1)h} \sum_{j=1,j\neq i}K_{h}(X_{i},X_{j}) where :math:`K_{h}` represents the generalized product kernel estimator: .. math:: K_{h}(X_{i},X_{j}) = \prod_{s=1}^{q}h_{s}^{-1}k\left(\frac{X_{is}-X_{js}}{h_{s}}\right) rN)r data_predictr)r r enumerater r)rrfuncLOOLiX_not_if_irrrloo_likelihoods  zKDEMultivariate.loo_likelihoodc Csx|dkr|j}n t||j}g}xHtt|dD]2}|t|j|j||ddf|j d|j q4Wt |}|S)aR Evaluate the probability density function. Parameters ---------- data_predict: array_like, optional Points to evaluate at. If unspecified, the training data is used. Returns ------- pdf_est: array_like Probability density function evaluated at `data_predict`. Notes ----- The probability density is given by the generalized product kernel estimator: .. math:: K_{h}(X_{i},X_{j}) = \prod_{s=1}^{q}h_{s}^{-1}k\left(\frac{X_{is}-X_{js}}{h_{s}}\right) Nr)rr(r) rr rrrrappendr rrrsqueeze)rr(pdf_estr-rrrpdfs  zKDEMultivariate.pdfc Cs~|dkr|j}n t||j}g}xNtt|dD]8}|t|j|j||ddf|j dddd|j q4Wt |}|S)a Evaluate the cumulative distribution function. Parameters ---------- data_predict: array_like, optional Points to evaluate at. If unspecified, the training data is used. Returns ------- cdf_est: array_like The estimate of the cdf. Notes ----- See http://en.wikipedia.org/wiki/Cumulative_distribution_function For more details on the estimation see Ref. [5] in module docstring. The multivariate CDF for mixed data (continuous and ordered/unordered discrete) is estimated by: .. math:: F(x^{c},x^{d})=n^{-1}\sum_{i=1}^{n}\left[G(\frac{x^{c}-X_{i}}{h})\sum_{u\leq x^{d}}L(X_{i}^{d},x_{i}^{d}, \lambda)\right] where G() is the product kernel CDF estimator for the continuous and L() for the discrete variables. Used bandwidth is ``self.bw``. Nr gaussian_cdfaitchisonaitken_cdf wangryzin_cdf)rr(rckertypeukertypeokertype) rr rrrrr1r rrrr2)rr(cdf_estr-rrrcdfs  zKDEMultivariate.cdfcCsd}ttjtjtjd}|j}|j }|j}t dd|D}|| }t |j } x~t |D]r} xHt|D]<\} } || || |dd| f|| | f| dd| f<qpW| j dd|} | jdd}||7}qbWttjtjtjd}t|j}d}t |j dd|j df} xt|D]x\} }xLt|D]@\} } || || |dd| f || | f| dd| f<q4W| j dd|} || jdd7}q"W||dd|||dS) a Returns the Integrated Mean Square Error for the unconditional KDE. Parameters ---------- bw: array_like The bandwidth parameter(s). Returns ------- CV: float The cross-validation objective function. Notes ----- See p. 27 in [1]_ for details on how to handle the multivariate estimation with mixed data types see p.6 in [2]_. The formula for the cross-validation objective function is: .. math:: CV=\frac{1}{n^{2}}\sum_{i=1}^{n}\sum_{j=1}^{N} \bar{K}_{h}(X_{i},X_{j})-\frac{2}{n(n-1)}\sum_{i=1}^{n} \sum_{j=1,j\neq i}^{N}K_{h}(X_{i},X_{j}) Where :math:`\bar{K}_{h}` is the multivariate product convolution kernel (consult [2]_ for mixed data types). References ---------- .. [1] Racine, J., Li, Q. Nonparametric econometrics: theory and practice. Princeton University Press. (2007) .. [2] Racine, J., Li, Q. "Nonparametric Estimation of Distributions with Categorical and Continuous Data." Working Paper. (2000) r)coucSsg|] }|dkqS)r=r).0r=rrr 3sz(KDEMultivariate.imse..Nr)axis)dictrZgaussian_convolutionZwang_ryzin_convolutionZaitchison_aitken_convolutionrrrrZarrayZprodemptyrrr)sumZgaussianZ wang_ryzinZaitchison_aitkenr )rrFZkertypesrrrZix_contZ_bw_cont_productZKvalr-iiZvtypeZdensZ k_bar_sumr+r,r.rrrimses@3          zKDEMultivariate.imsecCsd}|jf}||fS)z@Helper method to be able to pass needed vars to _compute_subset.r )r)r class_type class_varsrrr_get_class_vars_typeQsz$KDEMultivariate._get_class_vars_type)N)N) __name__ __module__ __qualname____doc__rr r%r0r4r<rIrLrrrrr ,s; $ $ 0Xc@sZeZdZdZefddZddZddfdd Zdd d Zdd dZ ddZ ddZ d S)r au Conditional multivariate kernel density estimator. Calculates ``P(Y_1,Y_2,...Y_n | X_1,X_2...X_m) = P(X_1, X_2,...X_n, Y_1, Y_2,..., Y_m)/P(X_1, X_2,..., X_m)``. The conditional density is by definition the ratio of the two densities, see [1]_. Parameters ---------- endog: list of ndarrays or 2-D ndarray The training data for the dependent variables, used to determine the bandwidth(s). If a 2-D array, should be of shape (num_observations, num_variables). If a list, each list element is a separate observation. exog: list of ndarrays or 2-D ndarray The training data for the independent variable; same shape as `endog`. dep_type: str The type of the dependent variables: c : Continuous u : Unordered (Discrete) o : Ordered (Discrete) The string should contain a type specifier for each variable, so for example ``dep_type='ccuo'``. indep_type: str The type of the independent variables; specifed like `dep_type`. bw: array_like or str, optional If an array, it is a fixed user-specified bandwidth. If a string, should be one of: - normal_reference: normal reference rule of thumb (default) - cv_ml: cross validation maximum likelihood - cv_ls: cross validation least squares defaults: Instance of class EstimatorSettings The default values for the efficient bandwidth estimation Attributes --------- bw: array_like The bandwidth parameters See Also -------- KDEMultivariate References ---------- .. [1] http://en.wikipedia.org/wiki/Conditional_probability_distribution Examples -------- >>> import statsmodels.api as sm >>> nobs = 300 >>> c1 = np.random.normal(size=(nobs,1)) >>> c2 = np.random.normal(2,1,size=(nobs,1)) >>> dens_c = sm.nonparametric.KDEMultivariateConditional(endog=[c1], ... exog=[c2], dep_type='c', indep_type='c', bw='normal_reference') >>> dens_c.bw # show computed bandwidth array([ 0.41223484, 0.40976931]) cCs||_||_|||_t|j|_t|j|_t||j|_t||j|_t |j\|_ |_t |j|jf|_ t |j d|_|||js|||_n |||_dS)Nr)dep_type indep_typerrk_depk_indepr endogexogrrr column_stackrrrrrrr)rrUrVrQrRrrrrrr s    z#KDEMultivariateConditional.__init__cCsd}|dt|jd7}|dt|jd7}|dt|jd7}|d|jd7}|d|jd7}|d|jd7}|S) z Provide something sane to print.z$KDEMultivariateConditional instance z+Number of independent variables: k_indep = r!z'Number of dependent variables: k_dep = zNumber of observations: nobs = z!Independent variable types: zDependent variable types: zBW selection method: )r"rTrSrrRrQr#)rr$rrrr%sz#KDEMultivariateConditional.__repr__cCs|S)Nr)r&rrrr'sz#KDEMultivariateConditional.c Cst|j}t|j}d}xt|D]|\}}t|}t|| |j|ddf |j|jd} t||j d| |j|ddf |jd} | | } ||| 7}q&W| S)a Returns the leave-one-out conditional likelihood of the data. If `func` is not equal to the default, what's calculated is a function of the leave-one-out conditional likelihood. Parameters ---------- bw: array_like The bandwidth parameter(s). func: callable, optional Function to transform the likelihood values (before summing); for the log likelihood, use ``func=np.log``. Default is ``f(x) = x``. Returns ------- L: float The value of the leave-one-out function for the data. Notes ----- Similar to ``KDE.loo_likelihood`, but substitute ``f(y|x)=f(x,y)/f(x)`` for ``f(x)``. rN)rr(r) r rrV__iter__r)rr rQrRrS) rrr*ZyLOOZxLOOr,r-ZY_jr.f_yxf_xr/rrrr0s  z)KDEMultivariateConditional.loo_likelihoodNcCs|dkr|j}n t||j}|dkr,|j}n t||j}g}t||f}x|tt|dD]f}t |j |j ||ddf|j |j d}t |j |jd|j||ddf|j d}|||q^Wt|S)aN Evaluate the probability density function. Parameters ---------- endog_predict: array_like, optional Evaluation data for the dependent variables. If unspecified, the training data is used. exog_predict: array_like, optional Evaluation data for the independent variables. Returns ------- pdf: array_like The value of the probability density at `endog_predict` and `exog_predict`. Notes ----- The formula for the conditional probability density is: .. math:: f(y|x)=\frac{f(x,y)}{f(x)} with .. math:: f(x)=\prod_{s=1}^{q}h_{s}^{-1}k \left(\frac{x_{is}-x_{js}}{h_{s}}\right) where :math:`k` is the appropriate kernel for each variable. Nr)rr(r)rUr rSrVrTrrWrrr rrrQrRr1r2)r endog_predict exog_predictr3r(r-rYrZrrrr4s"    zKDEMultivariateConditional.pdfc Cs&|dkr|j}n t||j}|dkr,|j}n t||j}t|d}t|}xt|D]}t |j |jd|j||ddf|j d|j }t |}t |j d|j|j||ddf|jddddd}t |j |jd|j||ddf|j dd }||jdd } | |j |||<qZW|S) a Cumulative distribution function for the conditional density. Parameters ---------- endog_predict: array_like, optional The evaluation dependent variables at which the cdf is estimated. If not specified the training dependent variables are used. exog_predict: array_like, optional The evaluation independent variables at which the cdf is estimated. If not specified the training independent variables are used. Returns ------- cdf_est: array_like The estimate of the cdf. Notes ----- For more details on the estimation see [2]_, and p.181 in [1]_. The multivariate conditional CDF for mixed data (continuous and ordered/unordered discrete) is estimated by: .. math:: F(y|x)=\frac{n^{-1}\sum_{i=1}^{n}G(\frac{y-Y_{i}}{h_{0}}) W_{h}(X_{i},x)}{\widehat{\mu}(x)} where G() is the product kernel CDF estimator for the dependent (y) variable(s) and W() is the product kernel CDF estimator for the independent variable(s). References ---------- .. [1] Racine, J., Li, Q. Nonparametric econometrics: theory and practice. Princeton University Press. (2007) .. [2] Liu, R., Yang, L. "Kernel estimation of multivariate cumulative distribution function." Journal of Nonparametric Statistics (2008) Nr)rr(rr5r6r7F)rr(rr8r9r:tosum)rr(rr])rB)rUr rSrVrTrrrErr rrRrr2rQrF) rr[r\ZN_data_predictr;r-Zmu_xZ cdf_endogZcdf_exogSrrrr<s2)      zKDEMultivariateConditional.cdfc Cst|j}d}t|j}t|jddf}xt|D]t\}}|dd|jdf}|ddd|jf} t| |} t|| } t||} t||} t ||jd| |j |ddf|j dd}t ||jd| |j |ddf|j dd}t |d|j| | |j ddddd }||| |d }t || |j|ddf |j |j d |}t ||jd| |j |ddf |j d |}|||d d ||7}q8W||S) a The integrated mean square error for the conditional KDE. Parameters ---------- bw: array_like The bandwidth parameter(s). Returns ------- CV: float The cross-validation objective function. Notes ----- For more details see pp. 156-166 in [1]_. For details on how to handle the mixed variable types see [2]_. The formula for the cross-validation objective function for mixed variable types is: .. math:: CV(h,\lambda)=\frac{1}{n}\sum_{l=1}^{n} \frac{G_{-l}(X_{l})}{\left[\mu_{-l}(X_{l})\right]^{2}}- \frac{2}{n}\sum_{l=1}^{n}\frac{f_{-l}(X_{l},Y_{l})}{\mu_{-l}(X_{l})} where .. math:: G_{-l}(X_{l}) = n^{-2}\sum_{i\neq l}\sum_{j\neq l} K_{X_{i},X_{l}} K_{X_{j},X_{l}}K_{Y_{i},Y_{j}}^{(2)} where :math:`K_{X_{i},X_{l}}` is the multivariate product kernel and :math:`\mu_{-l}(X_{l})` is the leave-one-out estimator of the pdf. :math:`K_{Y_{i},Y_{j}}^{(2)}` is the convolution kernel. The value of the function is minimized by the ``_cv_ls`` method of the `GenericKDE` class to return the bw estimates that minimize the distance between the estimated and "true" probability density. References ---------- .. [1] Racine, J., Li, Q. Nonparametric econometrics: theory and practice. Princeton University Press. (2007) .. [2] Racine, J., Li, Q. "Nonparametric Estimation of Distributions with Categorical and Continuous Data." Working Paper. (2000) rrNF)rr(rr]Zgauss_convolutionZwangryzin_convolutionZaitchisonaitken_convolution)rr(rr8r:r9r]rC)rr(r)r rfloatrrZonesr)rSZkronr rVrRrQrF)rrZzLOOZCVrZexpanderrHZXYZYe_LZYe_RZXe_LZXe_RZK_Xi_XlZK_Xj_XlZK2_Yi_YjGZf_X_YZm_xrrrrI]s>/         zKDEMultivariateConditional.imsecCsd}|j|j|jf}||fS)z@Helper method to be able to pass needed vars to _compute_subset.r )rSrQrR)rrJrKrrrrLsz/KDEMultivariateConditional._get_class_vars_type)NN)NN) rMrNrOrPrr r%r0r4r<rIrLrrrrr Xs@ ( 4 HP)rPZ __future__rZstatsmodels.compat.pythonrrZnumpyrrZ _kernel_baserrr r r __all__r r rrrrs   .