B \Q@shdZddlmZmZddlmZddlZddlZddlZddl Z ddl m Z ddlmZmZddlmZmZddlmZmZmZdd lmZdd lmZmZdd lmZd d lmZe e j!j"Z#ej$e j%fZ&ddZ'ddZ(ddZ)d7ddZ*ddZ+ddZ,ddZ-ddZ.d8d d!Z/d"d#Z0d9d'd(Z1d:d)d*Z2d+d,Z3d;d.d/Z4dcCst|d|j\}}|dkr.||kr*d}nd}|dkrt||}t|}||||} ||||} t| | t| | | | fSt|||d\} } } t | jt | j} } t| dt| dddf| dddf<t| dt| dddf| dddf<xt d|D] }| dd|f| |ddf}}t |dt |d}}tt |dtt |d}}t |t |}}t |t |}}||||}}||kr||}||}|}n||}||}|}t| ||}||| dd|f<||| |ddf<q,Wd| | |k<d| | |k<|dkrbn|dkr|}|| | dk<|| | dk<n|d krt|}|}t||t| | dkd | | dk<t||t| | dkd | | dk<ntd |d f| | fS) aAlgorithms for NMF initialization. Computes an initial guess for the non-negative rank k matrix approximation for X: X = WH Parameters ---------- X : array-like, shape (n_samples, n_features) The data matrix to be decomposed. n_components : integer The number of components desired in the approximation. init : None | 'random' | 'nndsvd' | 'nndsvda' | 'nndsvdar' Method used to initialize the procedure. Default: 'nndsvd' if n_components < n_features, otherwise 'random'. Valid options: - 'random': non-negative random matrices, scaled with: sqrt(X.mean() / n_components) - 'nndsvd': Nonnegative Double Singular Value Decomposition (NNDSVD) initialization (better for sparseness) - 'nndsvda': NNDSVD with zeros filled with the average of X (better when sparsity is not desired) - 'nndsvdar': NNDSVD with zeros filled with small random values (generally faster, less accurate alternative to NNDSVDa for when sparsity is not desired) - 'custom': use custom matrices W and H eps : float Truncate all values less then this in output to zero. random_state : int, RandomState instance or None, optional, default: None If int, random_state is the seed used by the random number generator; If RandomState instance, random_state is the random number generator; If None, the random number generator is the RandomState instance used by `np.random`. Used when ``random`` == 'nndsvdar' or 'random'. Returns ------- W : array-like, shape (n_samples, n_components) Initial guesses for solving X ~= WH H : array-like, shape (n_components, n_features) Initial guesses for solving X ~= WH References ---------- C. Boutsidis, E. Gallopoulos: SVD based initialization: A head start for nonnegative matrix factorization - Pattern Recognition, 2008 http://tinyurl.com/nndsvd zNMF initializationNrErandom) random_staterrnndsvdanndsvdardz3Invalid init parameter: got %r instead of one of %r)NrSrErUrV)rrrrmeanrZrandnabsr zerosr-ZmaximumZminimumrlenr)r n_componentsrKepsrT n_samples n_featuresavgrngr/r.USVjryZx_pZy_pZx_nZy_nZx_p_nrmZy_p_nrmZx_n_nrmZy_n_nrmZm_pZm_nuvZsigmaZlbdrrr_initialize_nmfsh:    00"&       *, ric Cs|jd}t|j|}t||} |dkrF|jdd|d|7<|dkrV| |8} |rf||} n t|} tj| tj d} t ||| | S)zHelper function for _fit_coordinate_descent Update W to minimize the objective function, iterating once over all coordinates. By symmetry, to update H, one can call _update_coordinate_descent(X.T, Ht, W, ...) rgN)dtype) rrrr'r Zflat permutationZarangeZasarrayZintpr) rr.HtZl1_regZl2_regshufflerTr\HHtXHtrkrrr_update_coordinate_descents    rp-C6?Tc  Cst|jdd} t|dd}t| }xt|D]}d}|t||| ||| |7}| rj|t|j| |||| |7}|dkrv|}|dkrP| rtd|||||kr,| rtd|d Pq,W|| j|fS) a Compute Non-negative Matrix Factorization (NMF) with Coordinate Descent The objective function is minimized with an alternating minimization of W and H. Each minimization is done with a cyclic (up to a permutation of the features) Coordinate Descent. Parameters ---------- X : array-like, shape (n_samples, n_features) Constant matrix. W : array-like, shape (n_samples, n_components) Initial guess for the solution. H : array-like, shape (n_components, n_features) Initial guess for the solution. tol : float, default: 1e-4 Tolerance of the stopping condition. max_iter : integer, default: 200 Maximum number of iterations before timing out. l1_reg_W : double, default: 0. L1 regularization parameter for W. l1_reg_H : double, default: 0. L1 regularization parameter for H. l2_reg_W : double, default: 0. L2 regularization parameter for W. l2_reg_H : double, default: 0. L2 regularization parameter for H. update_H : boolean, default: True Set to True, both W and H will be estimated from initial guesses. Set to False, only W will be estimated. verbose : integer, default: 0 The verbosity level. shuffle : boolean, default: False If true, randomize the order of coordinates in the CD solver. random_state : int, RandomState instance or None, optional, default: None If int, random_state is the seed used by the random number generator; If RandomState instance, random_state is the random number generator; If None, the random number generator is the RandomState instance used by `np.random`. Returns ------- W : array-like, shape (n_samples, n_components) Solution to the non-negative least squares problem. H : array-like, shape (n_components, n_features) Solution to the non-negative least squares problem. n_iter : int The number of iterations done by the algorithm. References ---------- Cichocki, Andrzej, and Phan, Anh-Huy. "Fast local algorithms for large scale nonnegative matrix and tensor factorizations." IEICE transactions on fundamentals of electronics, communications and computer sciences 92.3: 708-721, 2009. C)ordercsr) accept_sparsegrz violation:zConverged at iterationr)r r'rr-rpprint)rr.r/tolmax_iterr=r>r?r@update_HverbosermrTrlran_iterZ violationZviolation_initrrr_fit_coordinate_descents*I     r}c CsR|dkrT| dkrt||j} | r&| } n| } |dkrDt||j}t||} nt|||} t|rx| j}|j}n(| }|}| }|ddkrt ||dk<|ddkrt ||dk<|dkrtj |||dn6|dkr|dC}|dC}||9}n||dC}||9}t| |j} |dkrJ|dkr6tj |dd }|tj ddf} nt|rt |j}xt|jdD]^}t||ddf|}|ddkrt ||dk<||dC}t||j||ddf<qrWn||dC}t||j}|} |dkr| |7} |dkr| ||} t | | dk<| | } | }|dkrF||C}|||| fS) z%update W in Multiplicative Update NMFrNg?rg@r)out)r")r r'copyrrr(r$r%r&r)divider*newaxisemptyrr-)rr.r/rJr=r?gammaH_sumrnrorz numerator denominator WH_safe_XWH_safe_X_datar1r2ZWHHtr4WHidelta_Wrrr_multiplicative_update_w sl                "      rcCs&|dkr.t|j|}tt|j||}nt|||} t|rR| j} |j} n(| } |} | } |ddkrzt | | dk<|ddkrt | | dk<|dkrtj | | | dn6|dkr| dC} | dC} | | 9} n| |dC} | | 9} t|j| }|dkr&tj |dd} d| | dk<| d d tj f}nt|rt |j}xt|jdD]^}t||d d |f}|ddkrt ||dk<||dC}t|j||d d |f<qNWn| |dC} t|j| }|}|dkr||7}|dkr|||}t ||dk<||}|}|dkr"||C}|S) z%update H in Multiplicative Update NMFrg?rg@r)r~r)r"N)r r'rrr(r$r%r&rr)rr*rrrr-)rr.r/rJr>r@rrrrrr1r2ZW_sumZWtWHr4rdelta_Hrrr_multiplicative_update_his`                "      rrDc  Cst} t|}|dkr&dd|} n|dkrr?r@rzr{Z start_timerZ error_at_initZprevious_errorrrnror|rrerrorZ iter_timeZend_timerrr_fit_multiplicative_updatesLF    rrSrBcCst|dtd}t|dt|| ||}t|dkrB|dkrBtd|j\}}|dkrX|}t|trj|dkrvtd|t| tr| dkrtd| t|t j r|dkrtd ||d kr|rt |||fd t |||fd nh|s6t |||fd |d kr&t ||}t ||f|}nt ||f}nt|||| d\}}t| | | \}}}}|dkrt||||| |||||||| d \}}}n<|d krt||||| ||||||| \}}}n td||| kr|dkrtd| t|||fS)aOCompute Non-negative Matrix Factorization (NMF) Find two non-negative matrices (W, H) whose product approximates the non- negative matrix X. This factorization can be used for example for dimensionality reduction, source separation or topic extraction. The objective function is:: 0.5 * ||X - WH||_Fro^2 + alpha * l1_ratio * ||vec(W)||_1 + alpha * l1_ratio * ||vec(H)||_1 + 0.5 * alpha * (1 - l1_ratio) * ||W||_Fro^2 + 0.5 * alpha * (1 - l1_ratio) * ||H||_Fro^2 Where:: ||A||_Fro^2 = \sum_{i,j} A_{ij}^2 (Frobenius norm) ||vec(A)||_1 = \sum_{i,j} abs(A_{ij}) (Elementwise L1 norm) For multiplicative-update ('mu') solver, the Frobenius norm (0.5 * ||X - WH||_Fro^2) can be changed into another beta-divergence loss, by changing the beta_loss parameter. The objective function is minimized with an alternating minimization of W and H. If H is given and update_H=False, it solves for W only. Parameters ---------- X : array-like, shape (n_samples, n_features) Constant matrix. W : array-like, shape (n_samples, n_components) If init='custom', it is used as initial guess for the solution. H : array-like, shape (n_components, n_features) If init='custom', it is used as initial guess for the solution. If update_H=False, it is used as a constant, to solve for W only. n_components : integer Number of components, if n_components is not set all features are kept. init : None | 'random' | 'nndsvd' | 'nndsvda' | 'nndsvdar' | 'custom' Method used to initialize the procedure. Default: 'random'. Valid options: - 'random': non-negative random matrices, scaled with: sqrt(X.mean() / n_components) - 'nndsvd': Nonnegative Double Singular Value Decomposition (NNDSVD) initialization (better for sparseness) - 'nndsvda': NNDSVD with zeros filled with the average of X (better when sparsity is not desired) - 'nndsvdar': NNDSVD with zeros filled with small random values (generally faster, less accurate alternative to NNDSVDa for when sparsity is not desired) - 'custom': use custom matrices W and H update_H : boolean, default: True Set to True, both W and H will be estimated from initial guesses. Set to False, only W will be estimated. solver : 'cd' | 'mu' Numerical solver to use: 'cd' is a Coordinate Descent solver that uses Fast Hierarchical Alternating Least Squares (Fast HALS). 'mu' is a Multiplicative Update solver. .. versionadded:: 0.17 Coordinate Descent solver. .. versionadded:: 0.19 Multiplicative Update solver. beta_loss : float or string, default 'frobenius' String must be in {'frobenius', 'kullback-leibler', 'itakura-saito'}. Beta divergence to be minimized, measuring the distance between X and the dot product WH. Note that values different from 'frobenius' (or 2) and 'kullback-leibler' (or 1) lead to significantly slower fits. Note that for beta_loss <= 0 (or 'itakura-saito'), the input matrix X cannot contain zeros. Used only in 'mu' solver. .. versionadded:: 0.19 tol : float, default: 1e-4 Tolerance of the stopping condition. max_iter : integer, default: 200 Maximum number of iterations before timing out. alpha : double, default: 0. Constant that multiplies the regularization terms. l1_ratio : double, default: 0. The regularization mixing parameter, with 0 <= l1_ratio <= 1. For l1_ratio = 0 the penalty is an elementwise L2 penalty (aka Frobenius Norm). For l1_ratio = 1 it is an elementwise L1 penalty. For 0 < l1_ratio < 1, the penalty is a combination of L1 and L2. regularization : 'both' | 'components' | 'transformation' | None Select whether the regularization affects the components (H), the transformation (W), both or none of them. random_state : int, RandomState instance or None, optional, default: None If int, random_state is the seed used by the random number generator; If RandomState instance, random_state is the random number generator; If None, the random number generator is the RandomState instance used by `np.random`. verbose : integer, default: 0 The verbosity level. shuffle : boolean, default: False If true, randomize the order of coordinates in the CD solver. Returns ------- W : array-like, shape (n_samples, n_components) Solution to the non-negative least squares problem. H : array-like, shape (n_components, n_features) Solution to the non-negative least squares problem. n_iter : int Actual number of iterations. Examples -------- >>> import numpy as np >>> X = np.array([[1,1], [2, 1], [3, 1.2], [4, 1], [5, 0.8], [6, 1]]) >>> from sklearn.decomposition import non_negative_factorization >>> W, H, n_iter = non_negative_factorization(X, n_components=2, ... init='random', random_state=0) References ---------- Cichocki, Andrzej, and P. H. A. N. Anh-Huy. "Fast local algorithms for large scale nonnegative matrix and tensor factorizations." IEICE transactions on fundamentals of electronics, communications and computer sciences 92.3: 708-721, 2009. Fevotte, C., & Idier, J. (2011). Algorithms for nonnegative matrix factorization with the beta-divergence. Neural Computation, 23(9). )rucsc)rvrjz NMF (input X)rz|When beta_loss <= 0 and X contains zeros, the solver may diverge. Please add small values to X, or use a positive beta_loss.NzFNumber of components must be a positive integer; got (n_components=%r)zJMaximum number of iterations must be a positive integer; got (max_iter=%r)z>Tolerance for stopping criteria must be positive; got (tol=%r)Zcustomz NMF (input H)z NMF (input W)rC)rKrTrB)rzr{rmrTzInvalid solver parameter '%s'.zKMaximum number of iteration %d reached. Increase it to improve convergence.)r r9rrLr rrrM INTEGER_TYPESrOrPr!rrrXZfullrZrirAr}rrFrGr)rr.r/r\rKrzrIrJrxryr:r;r<rTr{rmr^r_r`r=r>r?r@r|rrrnon_negative_factorizationAsb           rc @s>eZdZdZdd d Zdd d ZdddZddZddZdS)NMFaNon-Negative Matrix Factorization (NMF) Find two non-negative matrices (W, H) whose product approximates the non- negative matrix X. This factorization can be used for example for dimensionality reduction, source separation or topic extraction. The objective function is:: 0.5 * ||X - WH||_Fro^2 + alpha * l1_ratio * ||vec(W)||_1 + alpha * l1_ratio * ||vec(H)||_1 + 0.5 * alpha * (1 - l1_ratio) * ||W||_Fro^2 + 0.5 * alpha * (1 - l1_ratio) * ||H||_Fro^2 Where:: ||A||_Fro^2 = \sum_{i,j} A_{ij}^2 (Frobenius norm) ||vec(A)||_1 = \sum_{i,j} abs(A_{ij}) (Elementwise L1 norm) For multiplicative-update ('mu') solver, the Frobenius norm (0.5 * ||X - WH||_Fro^2) can be changed into another beta-divergence loss, by changing the beta_loss parameter. The objective function is minimized with an alternating minimization of W and H. Read more in the :ref:`User Guide `. Parameters ---------- n_components : int or None Number of components, if n_components is not set all features are kept. init : 'random' | 'nndsvd' | 'nndsvda' | 'nndsvdar' | 'custom' Method used to initialize the procedure. Default: 'nndsvd' if n_components < n_features, otherwise random. Valid options: - 'random': non-negative random matrices, scaled with: sqrt(X.mean() / n_components) - 'nndsvd': Nonnegative Double Singular Value Decomposition (NNDSVD) initialization (better for sparseness) - 'nndsvda': NNDSVD with zeros filled with the average of X (better when sparsity is not desired) - 'nndsvdar': NNDSVD with zeros filled with small random values (generally faster, less accurate alternative to NNDSVDa for when sparsity is not desired) - 'custom': use custom matrices W and H solver : 'cd' | 'mu' Numerical solver to use: 'cd' is a Coordinate Descent solver. 'mu' is a Multiplicative Update solver. .. versionadded:: 0.17 Coordinate Descent solver. .. versionadded:: 0.19 Multiplicative Update solver. beta_loss : float or string, default 'frobenius' String must be in {'frobenius', 'kullback-leibler', 'itakura-saito'}. Beta divergence to be minimized, measuring the distance between X and the dot product WH. Note that values different from 'frobenius' (or 2) and 'kullback-leibler' (or 1) lead to significantly slower fits. Note that for beta_loss <= 0 (or 'itakura-saito'), the input matrix X cannot contain zeros. Used only in 'mu' solver. .. versionadded:: 0.19 tol : float, default: 1e-4 Tolerance of the stopping condition. max_iter : integer, default: 200 Maximum number of iterations before timing out. random_state : int, RandomState instance or None, optional, default: None If int, random_state is the seed used by the random number generator; If RandomState instance, random_state is the random number generator; If None, the random number generator is the RandomState instance used by `np.random`. alpha : double, default: 0. Constant that multiplies the regularization terms. Set it to zero to have no regularization. .. versionadded:: 0.17 *alpha* used in the Coordinate Descent solver. l1_ratio : double, default: 0. The regularization mixing parameter, with 0 <= l1_ratio <= 1. For l1_ratio = 0 the penalty is an elementwise L2 penalty (aka Frobenius Norm). For l1_ratio = 1 it is an elementwise L1 penalty. For 0 < l1_ratio < 1, the penalty is a combination of L1 and L2. .. versionadded:: 0.17 Regularization parameter *l1_ratio* used in the Coordinate Descent solver. verbose : bool, default=False Whether to be verbose. shuffle : boolean, default: False If true, randomize the order of coordinates in the CD solver. .. versionadded:: 0.17 *shuffle* parameter used in the Coordinate Descent solver. Attributes ---------- components_ : array, [n_components, n_features] Factorization matrix, sometimes called 'dictionary'. reconstruction_err_ : number Frobenius norm of the matrix difference, or beta-divergence, between the training data ``X`` and the reconstructed data ``WH`` from the fitted model. n_iter_ : int Actual number of iterations. Examples -------- >>> import numpy as np >>> X = np.array([[1, 1], [2, 1], [3, 1.2], [4, 1], [5, 0.8], [6, 1]]) >>> from sklearn.decomposition import NMF >>> model = NMF(n_components=2, init='random', random_state=0) >>> W = model.fit_transform(X) >>> H = model.components_ References ---------- Cichocki, Andrzej, and P. H. A. N. Anh-Huy. "Fast local algorithms for large scale nonnegative matrix and tensor factorizations." IEICE transactions on fundamentals of electronics, communications and computer sciences 92.3: 708-721, 2009. Fevotte, C., & Idier, J. (2011). Algorithms for nonnegative matrix factorization with the beta-divergence. Neural Computation, 23(9). NrBrD-C6?rrrFc CsF||_||_||_||_||_||_||_||_| |_| |_ | |_ dS)N) r\rKrIrJrxryrTr:r;r{rm) selfr\rKrIrJrxryrTr:r;r{rmrrr__init__sz NMF.__init__cCst|dtd}t||||j|jd|j|j|j|j|j |j d|j |j |j d\}}}t||||jdd|_|jd|_||_||_|S)aLearn a NMF model for the data X and returns the transformed data. This is more efficient than calling fit followed by transform. Parameters ---------- X : {array-like, sparse matrix}, shape (n_samples, n_features) Data matrix to be decomposed y : Ignored W : array-like, shape (n_samples, n_components) If init='custom', it is used as initial guess for the solution. H : array-like, shape (n_components, n_features) If init='custom', it is used as initial guess for the solution. Returns ------- W : array, shape (n_samples, n_components) Transformed data. )rur)rvrjTr6)rr.r/r\rKrzrIrJrxryr:r;r<rTr{rm)r0r)r r9rr\rKrIrJrxryr:r;rTr{rmr5Zreconstruction_err_r n_components_ components_n_iter_)rrrfr.r/rrrr fit_transforms     zNMF.fit_transformcKs|j|f||S)aLearn a NMF model for the data X. Parameters ---------- X : {array-like, sparse matrix}, shape (n_samples, n_features) Data matrix to be decomposed y : Ignored Returns ------- self )r)rrrfZparamsrrrfitszNMF.fitcCsTt|dt|d|j|j|jd|j|j|j|j|j |j d|j |j |j d\}}}|S)aUTransform the data X according to the fitted NMF model Parameters ---------- X : {array-like, sparse matrix}, shape (n_samples, n_features) Data matrix to be transformed by the model Returns ------- W : array, shape (n_samples, n_components) Transformed data rNFr6)rr.r/r\rKrzrIrJrxryr:r;r<rTr{rm)rrrrrKrIrJrxryr:r;rTr{rm)rrr._rrrr transforms     z NMF.transformcCst|dt||jS)ayTransform data back to its original space. Parameters ---------- W : {array-like, sparse matrix}, shape (n_samples, n_components) Transformed data matrix Returns ------- X : {array-like, sparse matrix}, shape (n_samples, n_features) Data matrix of original shape .. versionadded:: 0.18 r)rrrr)rr.rrrinverse_transforms zNMF.inverse_transform) NNrBrDrrrNrrrF)NNN)N) __name__ __module__ __qualname____doc__rrrrrrrrrrs * r)F)NrRN) rqrrrrrrTrFN)NNNT) rDrrrqrrrrTr)NNNrSTrBrDrqrrrrNNrF)7rZ __future__rrZmathrrFrOrZnumpyrZ scipy.sparseZsparser$baserrZutilsrr Z utils.extmathr r r r Zutils.validationrr exceptionsrZ cdnmf_fastrrZfloat32r]r)ZIntegralZintegerrrrr!r5r(rArLr#rirpr}rrrrrrrrrs\         j    i ^R  Z