ó ‡ˆ\c@s(dZddlZddljZddlmZddlZddl m Z m Z ddl m Z mZmZmZddlmZddlmZdd lmZmZmZdd lmZdd lmZd d lmZmZm Z ej!ej"ƒj#Z$d„Z%de e fd„ƒYZ&dS(s6 ============================================================= Online Latent Dirichlet Allocation with variational inference ============================================================= This implementation is modified from Matthew D. Hoffman's onlineldavb code Link: https://github.com/blei-lab/onlineldavb iÿÿÿÿN(tgammalni(t BaseEstimatortTransformerMixin(tcheck_random_statet check_arrayt gen_batchestgen_even_slices(t logsumexp(tcheck_non_negative(tParalleltdelayedteffective_n_jobs(txrange(tNotFittedErrori(t mean_changet_dirichlet_expectation_1dt_dirichlet_expectation_2dcCsotj|ƒ}|j\}} |jd} |rO|jdd|| fƒ} ntj|| fƒ} tjt| ƒƒ} |r‘tj|jƒnd} |r»|j }|j }|j }nx§t |ƒD]™}|r |||||d!}|||||d!}n3tj||dd…fƒd}|||f}| |dd…f}| |dd…fjƒ}|dd…|f}xyt d|ƒD]h}|}tj||ƒt}|tj|||jƒ}t|||ƒt||ƒ|kr”Pq”q”W|| |dd…f<|rÈtj||ƒt}| dd…|fctj|||ƒ7`. Parameters ---------- n_components : int, optional (default=10) Number of topics. doc_topic_prior : float, optional (default=None) Prior of document topic distribution `theta`. If the value is None, defaults to `1 / n_components`. In [1]_, this is called `alpha`. topic_word_prior : float, optional (default=None) Prior of topic word distribution `beta`. If the value is None, defaults to `1 / n_components`. In [1]_, this is called `eta`. learning_method : 'batch' | 'online', default='batch' Method used to update `_component`. Only used in `fit` method. In general, if the data size is large, the online update will be much faster than the batch update. Valid options:: 'batch': Batch variational Bayes method. Use all training data in each EM update. Old `components_` will be overwritten in each iteration. 'online': Online variational Bayes method. In each EM update, use mini-batch of training data to update the ``components_`` variable incrementally. The learning rate is controlled by the ``learning_decay`` and the ``learning_offset`` parameters. .. versionchanged:: 0.20 The default learning method is now ``"batch"``. learning_decay : float, optional (default=0.7) It is a parameter that control learning rate in the online learning method. The value should be set between (0.5, 1.0] to guarantee asymptotic convergence. When the value is 0.0 and batch_size is ``n_samples``, the update method is same as batch learning. In the literature, this is called kappa. learning_offset : float, optional (default=10.) A (positive) parameter that downweights early iterations in online learning. It should be greater than 1.0. In the literature, this is called tau_0. max_iter : integer, optional (default=10) The maximum number of iterations. batch_size : int, optional (default=128) Number of documents to use in each EM iteration. Only used in online learning. evaluate_every : int, optional (default=0) How often to evaluate perplexity. Only used in `fit` method. set it to 0 or negative number to not evalute perplexity in training at all. Evaluating perplexity can help you check convergence in training process, but it will also increase total training time. Evaluating perplexity in every iteration might increase training time up to two-fold. total_samples : int, optional (default=1e6) Total number of documents. Only used in the `partial_fit` method. perp_tol : float, optional (default=1e-1) Perplexity tolerance in batch learning. Only used when ``evaluate_every`` is greater than 0. mean_change_tol : float, optional (default=1e-3) Stopping tolerance for updating document topic distribution in E-step. max_doc_update_iter : int (default=100) Max number of iterations for updating document topic distribution in the E-step. n_jobs : int or None, optional (default=None) The number of jobs to use in the E-step. ``None`` means 1 unless in a :obj:`joblib.parallel_backend` context. ``-1`` means using all processors. See :term:`Glossary ` for more details. verbose : int, optional (default=0) Verbosity level. random_state : int, RandomState instance or None, optional (default=None) If int, random_state is the seed used by the random number generator; If RandomState instance, random_state is the random number generator; If None, the random number generator is the RandomState instance used by `np.random`. n_topics : int, optional (default=None) This parameter has been renamed to n_components and will be removed in version 0.21. .. deprecated:: 0.19 Attributes ---------- components_ : array, [n_components, n_features] Variational parameters for topic word distribution. Since the complete conditional for topic word distribution is a Dirichlet, ``components_[i, j]`` can be viewed as pseudocount that represents the number of times word `j` was assigned to topic `i`. It can also be viewed as distribution over the words for each topic after normalization: ``model.components_ / model.components_.sum(axis=1)[:, np.newaxis]``. n_batch_iter_ : int Number of iterations of the EM step. n_iter_ : int Number of passes over the dataset. Examples -------- >>> from sklearn.decomposition import LatentDirichletAllocation >>> from sklearn.datasets import make_multilabel_classification >>> # This produces a feature matrix of token counts, similar to what >>> # CountVectorizer would produce on text. >>> X, _ = make_multilabel_classification(random_state=0) >>> lda = LatentDirichletAllocation(n_components=5, ... random_state=0) >>> lda.fit(X) # doctest: +ELLIPSIS LatentDirichletAllocation(...) >>> # get topics for some given samples: >>> lda.transform(X[-2:]) array([[0.00360392, 0.25499205, 0.0036211 , 0.64236448, 0.09541846], [0.15297572, 0.00362644, 0.44412786, 0.39568399, 0.003586 ]]) References ---------- [1] "Online Learning for Latent Dirichlet Allocation", Matthew D. Hoffman, David M. Blei, Francis Bach, 2010 [2] "Stochastic Variational Inference", Matthew D. Hoffman, David M. Blei, Chong Wang, John Paisley, 2013 [3] Matthew D. Hoffman's onlineldavb code. Link: https://github.com/blei-lab/onlineldavb i tbatchgffffffæ?g$@i€iÿÿÿÿg€„.Agš™™™™™¹?gü©ñÒMbP?idicCs||_||_||_||_||_||_||_||_| |_| |_ | |_ | |_ | |_ ||_ ||_||_||_dS(N(t n_componentsR%ttopic_word_priortlearning_methodtlearning_decaytlearning_offsettmax_itert batch_sizetevaluate_everyt total_samplestperp_tolR'tmax_doc_update_itertn_jobstverboseR)R-(tselfR@R%RARBRCRDRERFRGRHRIR'RJRKRLR)R-((s?lib/python2.7/site-packages/sklearn/decomposition/online_lda.pyt__init__s"                cCsÒ|jd k r.|j|_tjdtƒn |j|_|jdkr_td|jƒ‚n|jdkr„td|jƒ‚n|j dkr©td|j ƒ‚n|j d krÎtd|j ƒ‚nd S( sCheck model parameters.sUn_topics has been renamed to n_components in version 0.19 and will be removed in 0.21is$Invalid 'n_components' parameter: %rs%Invalid 'total_samples' parameter: %rs'Invalid 'learning_offset' parameter: %rR?tonlines'Invalid 'learning_method' parameter: %rN(R?RO( R-Rt _n_componentstwarningstwarntDeprecationWarningR@t ValueErrorRHRDRB(RM((s?lib/python2.7/site-packages/sklearn/decomposition/online_lda.pyt _check_params4s"    cCsÓt|jƒ|_d|_d|_|jdkrFd|j|_n |j|_|j dkrtd|j|_ n |j |_ d}d|}|jj |||j|fƒ|_ t jt|j ƒƒ|_dS(sInitialize latent variables.iigð?gY@N(RR)t random_state_t n_batch_iter_tn_iter_R%RRPtdoc_topic_prior_RAttopic_word_prior_Rt components_RRRtexp_dirichlet_component_(RMR,t init_gammatinit_var((s?lib/python2.7/site-packages/sklearn/decomposition/online_lda.pyt_init_latent_varsNs      c s|rˆjnd‰tˆjƒ}|dkrXtd|dtdˆjdƒƒ}n|‡‡‡‡fd†tˆjd|ƒDƒƒ}t |Œ\}}t j |ƒ} ˆrôt j ˆj jƒ} x|D]} | | 7} qÐW| ˆj9} nd} | | fS(síE-step in EM update. Parameters ---------- X : array-like or sparse matrix, shape=(n_samples, n_features) Document word matrix. cal_sstats : boolean Parameter that indicate whether to calculate sufficient statistics or not. Set ``cal_sstats`` to True when we need to run M-step. random_init : boolean Parameter that indicate whether to initialize document topic distribution randomly in the E-step. Set it to True in training steps. parallel : joblib.Parallel (optional) Pre-initialized instance of joblib.Parallel. Returns ------- (doc_topic_distr, suff_stats) : `doc_topic_distr` is unnormalized topic distribution for each document. In the literature, this is called `gamma`. `suff_stats` is expected sufficient statistics for the M-step. When `cal_sstats == False`, it will be None. RKRLiic 3sO|]E}ttƒˆ|dd…fˆjˆjˆjˆjˆˆƒVqdS(N(R R=R\RYRJR'(t.0t idx_slice(R#R(R)RM(s?lib/python2.7/site-packages/sklearn/decomposition/online_lda.pys sN(RVRR RKR tmaxRLRRtzipRtvstackRR[R\( RMR#R(t random_inittparallelRKtresultst doc_topicst sstats_listR.R0tsstats((R#R(R)RMs?lib/python2.7/site-packages/sklearn/decomposition/online_lda.pyt_e_stepis"  c CsÖ|j|dtdtd|ƒ\}}|r@|j||_nhtj|j|j|j ƒ}t |ƒ|j d}|jd|9_|j||j||7_tj t |jƒƒ|_ |jd7_dS(sùEM update for 1 iteration. update `_component` by batch VB or online VB. Parameters ---------- X : array-like or sparse matrix, shape=(n_samples, n_features) Document word matrix. total_samples : integer Total number of documents. It is only used when batch_update is `False`. batch_update : boolean Parameter that controls updating method. `True` for batch learning, `False` for online learning. parallel : joblib.Parallel Pre-initialized instance of joblib.Parallel Returns ------- doc_topic_distr : array, shape=(n_samples, n_components) Unnormalized document topic distribution. R(ReRfiiN(RktTrueRZR[RtpowerRDRWRCtfloatRRRR\( RMR#RHt batch_updateRfR:R0tweightt doc_ratio((s?lib/python2.7/site-packages/sklearn/decomposition/online_lda.pyt_em_step¨s cCs#t|ddƒ}t||ƒ|S(s¦check X format check X format and make sure no negative value in X. Parameters ---------- X : array-like or sparse matrix t accept_sparsetcsr(RR(RMR#twhom((s?lib/python2.7/site-packages/sklearn/decomposition/online_lda.pyt_check_non_neg_arrayÚs  c Cs|jƒ|j|dƒ}|j\}}|j}t|dƒsS|j|ƒn||jjdkrŒtd||jjdfƒ‚nt|j ƒ}t d|dt d|j dƒƒS}xIt ||ƒD]8}|j||dd…fd |jd td |ƒqÓWWdQX|S( sýOnline VB with Mini-Batch update. Parameters ---------- X : array-like or sparse matrix, shape=(n_samples, n_features) Document word matrix. y : Ignored Returns ------- self s%LatentDirichletAllocation.partial_fitR[isUThe provided data has %d dimensions while the model was trained with feature size %d.RKRLiNRHRoRf(RURvRRFthasattrR_R[RTR RKR RbRLRRrRHtFalse( RMR#tyR+R,RFRKRfRa((s?lib/python2.7/site-packages/sklearn/decomposition/online_lda.pyt partial_fitès(     cCsN|jƒ|j|dƒ}|j\}}|j}|j}|j}|j}|j|ƒd} t |j ƒ} t d| dt d|j dƒƒp} xft|ƒD]X} |dkrxht||ƒD]5} |j|| dd…fd|d td | ƒqÈWn|j|d|d td | ƒ|dkrÐ| d|dkrÐ|j|d td td | ƒ\}}|j||d tƒ}|j r¤d| d||fGHn| rÇt| |ƒ|jkrÇPn|} n|j rïd| d|fGHn|jd7_q¦WWdQX|j|d td td | ƒ\}}|j||d tƒ|_|S(s~Learn model for the data X with variational Bayes method. When `learning_method` is 'online', use mini-batch update. Otherwise, use batch update. Parameters ---------- X : array-like or sparse matrix, shape=(n_samples, n_features) Document word matrix. y : Ignored Returns ------- self sLatentDirichletAllocation.fitRKRLiiRONRHRoRfR(Ret sub_samplings/iteration: %d of max_iter: %d, perplexity: %.4fsiteration: %d of max_iter: %d(RURvRRERGRBRFR_RR RKR RbRLR RRrRxRlRkt_perplexity_precomp_distrtabsRIRXtbound_(RMR#RyR+R,RERGRBRFt last_boundRKRftiRatdoc_topics_distrR:tbound((s?lib/python2.7/site-packages/sklearn/decomposition/online_lda.pytfitsP       "       cCst|dƒstdƒ‚n|j|dƒ}|j\}}||jjdkrxtd||jjdfƒ‚n|j|dtdtƒ\}}|S(sNTransform data X according to fitted model. Parameters ---------- X : array-like or sparse matrix, shape=(n_samples, n_features) Document word matrix. Returns ------- doc_topic_distr : shape=(n_samples, n_components) Document topic distribution for X. R[s<no 'components_' attribute in model. Please fit model first.s#LatentDirichletAllocation.transformisUThe provided data has %d dimensions while the model was trained with feature size %d.R(Re(RwR RvRR[RTRkRx(RMR#R+R,R.R:((s?lib/python2.7/site-packages/sklearn/decomposition/online_lda.pyt_unnormalized_transformXs cCs<|j|ƒ}||jddƒdd…tjf:}|S(s©Transform data X according to the fitted model. .. versionchanged:: 0.18 *doc_topic_distr* is now normalized Parameters ---------- X : array-like or sparse matrix, shape=(n_samples, n_features) Document word matrix. Returns ------- doc_topic_distr : shape=(n_samples, n_components) Document topic distribution for X. taxisiN(R„tsumRtnewaxis(RMR#R.((s?lib/python2.7/site-packages/sklearn/decomposition/online_lda.pyt transformws)cCsÐd„}tj|ƒ}|j\}}|jjd}d} t|ƒ} t|jƒ} |j} |j} |rŽ|j}|j}|j }nxÝt d|ƒD]Ì}|rß|||||d!}|||||d!}n3t j ||dd…fƒd}|||f}| |dd…t j f| dd…|f}t|ddƒ}| t j||ƒ7} qžW| || || |jƒ7} |r°t|jƒ|}| |9} n| || |j| |ƒ7} | S(sEstimate the variational bound. Estimate the variational bound over "all documents" using only the documents passed in as X. Since log-likelihood of each word cannot be computed directly, we use this bound to estimate it. Parameters ---------- X : array-like or sparse matrix, shape=(n_samples, n_features) Document word matrix. doc_topic_distr : array, shape=(n_samples, n_components) Document topic distribution. In the literature, this is called gamma. sub_sampling : boolean, optional, (default=False) Compensate for subsampling of documents. It is used in calculate bound in online learning. Returns ------- score : float cSsqtj|||ƒ}|tjt|ƒt|ƒƒ7}|tjt||ƒttj|dƒƒƒ7}|S(Ni(RR†R(tpriortdistrtdirichlet_distrtsizetscore((s?lib/python2.7/site-packages/sklearn/decomposition/online_lda.pyt_loglikelihood¥s#3iiNR…(RRRR[RRYRZRRRR RRR‡RRRPRnRH(RMR#R.R{RŽR*R+R@R,Rtdirichlet_doc_topictdirichlet_component_R%RAR1R2R3R4R5R6ttempR<Rq((s?lib/python2.7/site-packages/sklearn/decomposition/online_lda.pyt _approx_bound‹s>       #   cCs=|j|dƒ}|j|ƒ}|j||dtƒ}|S(s?Calculate approximate log-likelihood as score. Parameters ---------- X : array-like or sparse matrix, shape=(n_samples, n_features) Document word matrix. y : Ignored Returns ------- score : float Use approximate bound as score. sLatentDirichletAllocation.scoreR{(RvR„R’Rx(RMR#RyR.R((s?lib/python2.7/site-packages/sklearn/decomposition/online_lda.pyR×sc Cs t|dƒstdƒ‚n|j|dƒ}|dkrN|j|ƒ}nO|j\}}||jdkrtdƒ‚n||jkrtdƒ‚n|jd}|j|||ƒ}|rå|j ƒt |j ƒ|}n |j ƒ}||} t j d| ƒS( sRCalculate approximate perplexity for data X with ability to accept precomputed doc_topic_distr Perplexity is defined as exp(-1. * log-likelihood per word) Parameters ---------- X : array-like or sparse matrix, [n_samples, n_features] Document word matrix. doc_topic_distr : None or array, shape=(n_samples, n_components) Document topic distribution. If it is None, it will be generated by applying transform on X. Returns ------- score : float Perplexity score. R[s<no 'components_' attribute in model. Please fit model first.s$LatentDirichletAllocation.perplexityis8Number of samples in X and doc_topic_distr do not match.s Number of topics does not match.gð¿N(RwR RvRR„RRTRPR’R†RnRHRR( RMR#R.R{R+R@tcurrent_samplesR‚tword_cntt perword_bound((s?lib/python2.7/site-packages/sklearn/decomposition/online_lda.pyR|ìs$       t deprecatedcCs2|dkrtjdtƒn|j|d|ƒS(s,Calculate approximate perplexity for data X. Perplexity is defined as exp(-1. * log-likelihood per word) .. versionchanged:: 0.19 *doc_topic_distr* argument has been deprecated and is ignored because user no longer has access to unnormalized distribution Parameters ---------- X : array-like or sparse matrix, [n_samples, n_features] Document word matrix. doc_topic_distr : None or array, shape=(n_samples, n_components) Document topic distribution. This argument is deprecated and is currently being ignored. .. deprecated:: 0.19 sub_sampling : bool Do sub-sampling or not. Returns ------- score : float Perplexity score. R–s|Argument 'doc_topic_distr' is deprecated and is being ignored as of 0.19. Support for this argument will be removed in 0.21.R{(RQRRRSR|(RMR#R.R{((s?lib/python2.7/site-packages/sklearn/decomposition/online_lda.pyt perplexitys   N(t__name__t __module__t__doc__RRNRUR_RkRrRvRzRƒR„RˆR’RRxR|R—(((s?lib/python2.7/site-packages/sklearn/decomposition/online_lda.pyR>‰s*‘      ? 2  ) G   L  1('RštnumpyRt scipy.sparsetsparseRt scipy.specialRRQtbaseRRtutilsRRRRt utils.fixesRtutils.validationRt utils._joblibR R R texternals.six.movesR t exceptionsR t _online_ldaRRRtfinfoRntepsR R=R>(((s?lib/python2.7/site-packages/sklearn/decomposition/online_lda.pyt s  " g