ó ‡ˆ\c@südZddlZddlZddlmZmZmZddlm Z m Z ddl m Z ddl m Z ddl mZdd l mZdd l mZdd l mZd d lmZd dlmZd„Zd„Zde fd„ƒYZdS(s Bayesian Gaussian Mixture Model.i’’’’N(tbetalntdigammatgammalni(t BaseMixturet _check_shape(t_check_precision_matrix(t_check_precision_positivity(t_compute_log_det_cholesky(t_compute_precision_cholesky(t_estimate_gaussian_parameters(t_estimate_log_gaussian_probi(t check_array(tcheck_is_fittedcCs&ttj|ƒƒtjt|ƒƒS(sTCompute the log of the Dirichlet distribution normalization term. Parameters ---------- dirichlet_concentration : array-like, shape (n_samples,) The parameters values of the Dirichlet distribution. Returns ------- log_dirichlet_norm : float The log normalization of the Dirichlet distribution. (Rtnptsum(tdirichlet_concentration((s?lib/python2.7/site-packages/sklearn/mixture/bayesian_mixture.pyt_log_dirichlet_norms cCs\||||dtjdƒtjtd|tj|ƒdd…tjfƒdƒ S(s>Compute the log of the Wishart distribution normalization term. Parameters ---------- degrees_of_freedom : array-like, shape (n_components,) The number of degrees of freedom on the covariance Wishart distributions. log_det_precision_chol : array-like, shape (n_components,) The determinant of the precision matrix for each component. n_features : int The number of features. Return ------ log_wishart_norm : array-like, shape (n_components,) The log normalization of the Wishart distribution. gą?g@Ni(tmathtlogR RRtarangetnewaxis(tdegrees_of_freedomtlog_det_precisions_cholt n_features((s?lib/python2.7/site-packages/sklearn/mixture/bayesian_mixture.pyt_log_wishart_norm&s tBayesianGaussianMixturecBsžeZdZddddddddddddddedd d „Zd „Zd „Zd „Zd„Z d„Z d„Z d„Z d„Z d„Zd„Zd„Zd„Zd„Zd„Zd„Zd„Zd„Zd„Zd„Zd„ZRS( só*Variational Bayesian estimation of a Gaussian mixture. This class allows to infer an approximate posterior distribution over the parameters of a Gaussian mixture distribution. The effective number of components can be inferred from the data. This class implements two types of prior for the weights distribution: a finite mixture model with Dirichlet distribution and an infinite mixture model with the Dirichlet Process. In practice Dirichlet Process inference algorithm is approximated and uses a truncated distribution with a fixed maximum number of components (called the Stick-breaking representation). The number of components actually used almost always depends on the data. .. versionadded:: 0.18 Read more in the :ref:`User Guide `. Parameters ---------- n_components : int, defaults to 1. The number of mixture components. Depending on the data and the value of the `weight_concentration_prior` the model can decide to not use all the components by setting some component `weights_` to values very close to zero. The number of effective components is therefore smaller than n_components. covariance_type : {'full', 'tied', 'diag', 'spherical'}, defaults to 'full' String describing the type of covariance parameters to use. Must be one of:: 'full' (each component has its own general covariance matrix), 'tied' (all components share the same general covariance matrix), 'diag' (each component has its own diagonal covariance matrix), 'spherical' (each component has its own single variance). tol : float, defaults to 1e-3. The convergence threshold. EM iterations will stop when the lower bound average gain on the likelihood (of the training data with respect to the model) is below this threshold. reg_covar : float, defaults to 1e-6. Non-negative regularization added to the diagonal of covariance. Allows to assure that the covariance matrices are all positive. max_iter : int, defaults to 100. The number of EM iterations to perform. n_init : int, defaults to 1. The number of initializations to perform. The result with the highest lower bound value on the likelihood is kept. init_params : {'kmeans', 'random'}, defaults to 'kmeans'. The method used to initialize the weights, the means and the covariances. Must be one of:: 'kmeans' : responsibilities are initialized using kmeans. 'random' : responsibilities are initialized randomly. weight_concentration_prior_type : str, defaults to 'dirichlet_process'. String describing the type of the weight concentration prior. Must be one of:: 'dirichlet_process' (using the Stick-breaking representation), 'dirichlet_distribution' (can favor more uniform weights). weight_concentration_prior : float | None, optional. The dirichlet concentration of each component on the weight distribution (Dirichlet). This is commonly called gamma in the literature. The higher concentration puts more mass in the center and will lead to more components being active, while a lower concentration parameter will lead to more mass at the edge of the mixture weights simplex. The value of the parameter must be greater than 0. If it is None, it's set to ``1. / n_components``. mean_precision_prior : float | None, optional. The precision prior on the mean distribution (Gaussian). Controls the extend to where means can be placed. Smaller values concentrate the means of each clusters around `mean_prior`. The value of the parameter must be greater than 0. If it is None, it's set to 1. mean_prior : array-like, shape (n_features,), optional The prior on the mean distribution (Gaussian). If it is None, it's set to the mean of X. degrees_of_freedom_prior : float | None, optional. The prior of the number of degrees of freedom on the covariance distributions (Wishart). If it is None, it's set to `n_features`. covariance_prior : float or array-like, optional The prior on the covariance distribution (Wishart). If it is None, the emiprical covariance prior is initialized using the covariance of X. The shape depends on `covariance_type`:: (n_features, n_features) if 'full', (n_features, n_features) if 'tied', (n_features) if 'diag', float if 'spherical' random_state : int, RandomState instance or None, optional (default=None) If int, random_state is the seed used by the random number generator; If RandomState instance, random_state is the random number generator; If None, the random number generator is the RandomState instance used by `np.random`. warm_start : bool, default to False. If 'warm_start' is True, the solution of the last fitting is used as initialization for the next call of fit(). This can speed up convergence when fit is called several times on similar problems. See :term:`the Glossary `. verbose : int, default to 0. Enable verbose output. If 1 then it prints the current initialization and each iteration step. If greater than 1 then it prints also the log probability and the time needed for each step. verbose_interval : int, default to 10. Number of iteration done before the next print. Attributes ---------- weights_ : array-like, shape (n_components,) The weights of each mixture components. means_ : array-like, shape (n_components, n_features) The mean of each mixture component. covariances_ : array-like The covariance of each mixture component. The shape depends on `covariance_type`:: (n_components,) if 'spherical', (n_features, n_features) if 'tied', (n_components, n_features) if 'diag', (n_components, n_features, n_features) if 'full' precisions_ : array-like The precision matrices for each component in the mixture. A precision matrix is the inverse of a covariance matrix. A covariance matrix is symmetric positive definite so the mixture of Gaussian can be equivalently parameterized by the precision matrices. Storing the precision matrices instead of the covariance matrices makes it more efficient to compute the log-likelihood of new samples at test time. The shape depends on ``covariance_type``:: (n_components,) if 'spherical', (n_features, n_features) if 'tied', (n_components, n_features) if 'diag', (n_components, n_features, n_features) if 'full' precisions_cholesky_ : array-like The cholesky decomposition of the precision matrices of each mixture component. A precision matrix is the inverse of a covariance matrix. A covariance matrix is symmetric positive definite so the mixture of Gaussian can be equivalently parameterized by the precision matrices. Storing the precision matrices instead of the covariance matrices makes it more efficient to compute the log-likelihood of new samples at test time. The shape depends on ``covariance_type``:: (n_components,) if 'spherical', (n_features, n_features) if 'tied', (n_components, n_features) if 'diag', (n_components, n_features, n_features) if 'full' converged_ : bool True when convergence was reached in fit(), False otherwise. n_iter_ : int Number of step used by the best fit of inference to reach the convergence. lower_bound_ : float Lower bound value on the likelihood (of the training data with respect to the model) of the best fit of inference. weight_concentration_prior_ : tuple or float The dirichlet concentration of each component on the weight distribution (Dirichlet). The type depends on ``weight_concentration_prior_type``:: (float, float) if 'dirichlet_process' (Beta parameters), float if 'dirichlet_distribution' (Dirichlet parameters). The higher concentration puts more mass in the center and will lead to more components being active, while a lower concentration parameter will lead to more mass at the edge of the simplex. weight_concentration_ : array-like, shape (n_components,) The dirichlet concentration of each component on the weight distribution (Dirichlet). mean_precision_prior : float The precision prior on the mean distribution (Gaussian). Controls the extend to where means can be placed. Smaller values concentrate the means of each clusters around `mean_prior`. mean_precision_ : array-like, shape (n_components,) The precision of each components on the mean distribution (Gaussian). mean_prior_ : array-like, shape (n_features,) The prior on the mean distribution (Gaussian). degrees_of_freedom_prior_ : float The prior of the number of degrees of freedom on the covariance distributions (Wishart). degrees_of_freedom_ : array-like, shape (n_components,) The number of degrees of freedom of each components in the model. covariance_prior_ : float or array-like The prior on the covariance distribution (Wishart). The shape depends on `covariance_type`:: (n_features, n_features) if 'full', (n_features, n_features) if 'tied', (n_features) if 'diag', float if 'spherical' See Also -------- GaussianMixture : Finite Gaussian mixture fit with EM. References ---------- .. [1] `Bishop, Christopher M. (2006). "Pattern recognition and machine learning". Vol. 4 No. 4. New York: Springer. `_ .. [2] `Hagai Attias. (2000). "A Variational Bayesian Framework for Graphical Models". In Advances in Neural Information Processing Systems 12. `_ .. [3] `Blei, David M. and Michael I. Jordan. (2006). "Variational inference for Dirichlet process mixtures". Bayesian analysis 1.1 `_ itfullgü©ńŅMbP?gķµ ÷Ę°>idtkmeanstdirichlet_processii cCs’tt|ƒjd|d|d|d|d|d|d|d|d |d |ƒ ||_||_| |_| |_| |_| |_| |_ dS( Nt n_componentsttolt reg_covartmax_itertn_initt init_paramst random_statet warm_starttverbosetverbose_interval( tsuperRt__init__tcovariance_typetweight_concentration_prior_typetweight_concentration_priortmean_precision_priort mean_priortdegrees_of_freedom_priortcovariance_prior(tselfRR)RRR R!R"R*R+R,R-R.R/R#R$R%R&((s?lib/python2.7/site-packages/sklearn/mixture/bayesian_mixture.pyR(5s        cCs|jd kr%td|jƒ‚n|jd krJtd|jƒ‚n|jƒ|j|ƒ|j|ƒ|j|ƒd S( s‘Check that the parameters are well defined. Parameters ---------- X : array-like, shape (n_samples, n_features) t sphericalttiedtdiagRslInvalid value for 'covariance_type': %s 'covariance_type' should be in ['spherical', 'tied', 'diag', 'full']Rtdirichlet_distributions–Invalid value for 'weight_concentration_prior_type': %s 'weight_concentration_prior_type' should be in ['dirichlet_process', 'dirichlet_distribution']N(R1R2R3R(RR4(R)t ValueErrorR*t_check_weights_parameterst_check_means_parameterst_check_precision_parameterst _checkcovariance_prior_parameter(R0tX((s?lib/python2.7/site-packages/sklearn/mixture/bayesian_mixture.pyt_check_parametersKs    cCsW|jdkr"d|j|_n1|jdkr@|j|_ntd|jƒ‚dS(s2Check the parameter of the Dirichlet distribution.gš?gsSThe parameter 'weight_concentration_prior' should be greater than 0., but got %.3f.N(R+tNoneRtweight_concentration_prior_R5(R0((s?lib/python2.7/site-packages/sklearn/mixture/bayesian_mixture.pyR6es cCsĘ|j\}}|jd kr*d|_n1|jdkrH|j|_ntd|jƒ‚|jd kr‚|jddƒ|_n@t|jdt j t j gdt ƒ|_t |j|fdƒd S( s˜Check the parameters of the Gaussian distribution. Parameters ---------- X : array-like, shape (n_samples, n_features) gš?gsMThe parameter 'mean_precision_prior' should be greater than 0., but got %.3f.taxisitdtypet ensure_2dtmeansN(tshapeR,R<tmean_precision_prior_R5R-tmeant mean_prior_R R tfloat64tfloat32tFalseR(R0R:t_R((s?lib/python2.7/site-packages/sklearn/mixture/bayesian_mixture.pyR7qs    cCsm|j\}}|jdkr*||_n?|j|dkrL|j|_ntd|d|jfƒ‚dS(sŸCheck the prior parameters of the precision distribution. Parameters ---------- X : array-like, shape (n_samples, n_features) gš?sQThe parameter 'degrees_of_freedom_prior' should be greater than %d, but got %.3f.iN(RBR.R<tdegrees_of_freedom_prior_R5(R0R:RIR((s?lib/python2.7/site-packages/sklearn/mixture/bayesian_mixture.pyR8‹s cCs·|j\}}|jdkr§itjtj|jƒƒd6tjtj|jƒƒd6tj|ddddƒd6tj|ddddƒjƒd6|j |_ n |j dkrt |jd tj tj gd tƒ|_ t|j ||fd |j ƒt|j |j ƒn|j dkr‚t |jd tj tj gd tƒ|_ t|j |fd |j ƒt|j |j ƒn1|jd kr |j|_ ntd |jƒ‚dS(s„Check the `covariance_prior_`. Parameters ---------- X : array-like, shape (n_samples, n_features) RR2R>itddofiR3R1R?R@s%s covariance_priorgsSThe parameter 'spherical covariance_prior' should be greater than 0., but got %.3f.N(RR2(RBR/R<R t atleast_2dtcovtTtvarRDR)tcovariance_prior_R RFRGRHRRRR5(R0R:RIR((s?lib/python2.7/site-packages/sklearn/mixture/bayesian_mixture.pyR9s8"      cCsXt|||j|jƒ\}}}|j|ƒ|j||ƒ|j|||ƒdS(sĖInitialization of the mixture parameters. Parameters ---------- X : array-like, shape (n_samples, n_features) resp : array-like, shape (n_samples, n_components) N(R RR)t_estimate_weightst_estimate_meanst_estimate_precisions(R0R:tresptnktxktsk((s?lib/python2.7/site-packages/sklearn/mixture/bayesian_mixture.pyt _initializeĘs  cCsr|jdkr^d||jtjtj|ddd…ƒddd…dfƒf|_n|j||_dS(s•Estimate the parameters of the Dirichlet distribution. Parameters ---------- nk : array-like, shape (n_components,) Rgš?Ni’’’’iž’’’i(R*R=R thstacktcumsumtweight_concentration_(R0RU((s?lib/python2.7/site-packages/sklearn/mixture/bayesian_mixture.pyRQÖs BcCs\|j||_|j|j|dd…tjf||jdd…tjf|_dS(sĻEstimate the parameters of the Gaussian distribution. Parameters ---------- nk : array-like, shape (n_components,) xk : array-like, shape (n_components, n_features) N(RCtmean_precision_RER Rtmeans_(R0RURV((s?lib/python2.7/site-packages/sklearn/mixture/bayesian_mixture.pyRRčs  cCs[i|jd6|jd6|jd6|jd6|j|||ƒt|j|jƒ|_dS(sģEstimate the precisions parameters of the precision distribution. Parameters ---------- nk : array-like, shape (n_components,) xk : array-like, shape (n_components, n_features) sk : array-like The shape depends of `covariance_type`: 'full' : (n_components, n_features, n_features) 'tied' : (n_features, n_features) 'diag' : (n_components, n_features) 'spherical' : (n_components,) RR2R3R1N(t_estimate_wishart_fullt_estimate_wishart_tiedt_estimate_wishart_diagt_estimate_wishart_sphericalR)Rt covariances_tprecisions_cholesky_(R0RURVRW((s?lib/python2.7/site-packages/sklearn/mixture/bayesian_mixture.pyRSös    cCsį|j\}}|j||_tj|j||fƒ|_xrt|jƒD]a}|||j}|j |||||||j |j |tj ||ƒ|j|Estimate the diag Wishart distribution parameters. Parameters ---------- X : array-like, shape (n_samples, n_features) nk : array-like, shape (n_components,) xk : array-like, shape (n_components, n_features) sk : array-like, shape (n_components, n_features) N( RBRJRdRERPR RRCR\tsquareRb(R0RURVRWRIRRi((s?lib/python2.7/site-packages/sklearn/mixture/bayesian_mixture.pyR`Os   0cCs}|j\}}|j||_||j}|j|||j|jtjtj |ƒdƒ|_ |j |j:_ dS(s8Estimate the spherical Wishart distribution parameters. Parameters ---------- X : array-like, shape (n_samples, n_features) nk : array-like, shape (n_components,) xk : array-like, shape (n_components, n_features) sk : array-like, shape (n_components,) iN( RBRJRdRERPRCR\R RDRkRb(R0RURVRWRIRRi((s?lib/python2.7/site-packages/sklearn/mixture/bayesian_mixture.pyRals   "c Cs&t|dddddddgƒdS(NR[R\R]RdRbt precisions_Rc(R (R0((s?lib/python2.7/site-packages/sklearn/mixture/bayesian_mixture.pyt_check_is_fitted‰s cCsp|j\}}t|tj|ƒ|j|jƒ\}}}|j|ƒ|j||ƒ|j|||ƒdS(s&M step. Parameters ---------- X : array-like, shape (n_samples, n_features) log_resp : array-like, shape (n_samples, n_components) Logarithm of the posterior probabilities (or responsibilities) of the point of each sample in X. N( RBR R texpRR)RQRRRS(R0R:tlog_respt n_samplesRIRURVRW((s?lib/python2.7/site-packages/sklearn/mixture/bayesian_mixture.pyt_m_steps * cCs¦|jdkrt|jd|jdƒ}t|jdƒ}t|jdƒ}||tjdtj||ƒd fƒSt|jƒttj|jƒƒSdS(NRiii’’’’(R*RR[R RYRZR(R0t digamma_sumt digamma_at digamma_b((s?lib/python2.7/site-packages/sklearn/mixture/bayesian_mixture.pyt_estimate_log_weights¢s % cCsÆ|j\}}t||j|j|jƒd|tj|jƒ}|tjdƒtjt d|jtj d|ƒdd…tj fƒdƒ}|d|||j S(Ngą?g@i( RBR R]RcR)R RRdRRRRR\(R0R:RIRt log_gausst log_lambda((s?lib/python2.7/site-packages/sklearn/mixture/bayesian_mixture.pyt_estimate_log_probÆs 1 cCs |jj\}t|j|j|ƒd|tj|jƒ}|jdkrv|jtj t |j||ƒƒ}ntj t |j||ƒƒ}|j dkrĶtj t |jd|jdƒƒ }nt|jƒ}tj tj|ƒ|ƒ ||d|tj tj|jƒƒS(sEstimate the lower bound of the model. The lower bound on the likelihood (of the training data with respect to the model) is used to detect the convergence and has to decrease at each iteration. Parameters ---------- X : array-like, shape (n_samples, n_features) log_resp : array, shape (n_samples, n_components) Logarithm of the posterior probabilities (or responsibilities) of the point of each sample in X. log_prob_norm : float Logarithm of the probability of each sample in X. Returns ------- lower_bound : float gą?R2Rii(RERBRRcR)R RRdRRFRRR*RR[RRnR\(R0Rot log_prob_normRRt log_wisharttlog_norm_weight((s?lib/python2.7/site-packages/sklearn/mixture/bayesian_mixture.pyt_compute_lower_bound¾s "cCs(|j|j|j|j|j|jfS(N(R[R\R]RdRbRc(R0((s?lib/python2.7/site-packages/sklearn/mixture/bayesian_mixture.pyt_get_parametersļs  cCs]|\|_|_|_|_|_|_|jdkr“|jd|jd}|jd|}|jd|tjdtj |d ƒfƒ|_ |j tj |j ƒ:_ n|jtj |jƒ|_ |j dkrtj g|jD]}tj||jƒ^qļƒ|_n@|j dkrItj|j|jjƒ|_n|jd|_dS(NRiii’’’’RR2i(R[R\R]RdRbRcR*R RYtcumprodtweights_RR)tarrayRjRNRl(R0tparamstweight_dirichlet_sumttmpt prec_chol((s?lib/python2.7/site-packages/sklearn/mixture/bayesian_mixture.pyt_set_parametersõs$* &1 N(t__name__t __module__t__doc__R<RHR(R;R6R7R8R9RXRQRRRSR^R_R`RaRmRqRuRxR|R}R…(((s?lib/python2.7/site-packages/sklearn/mixture/bayesian_mixture.pyRAs:ņ        )     "       1 (RˆRtnumpyR t scipy.specialRRRtbaseRRtgaussian_mixtureRRRRR R tutilsR tutils.validationR RRR(((s?lib/python2.7/site-packages/sklearn/mixture/bayesian_mixture.pyts