\c@ sdZddlmZmZddlZddlmZddlZddlm Z ddl Z ddl Z ddl Z ddl ZddljZddlmZmZddlmZdd lmZdd lmZd d lmZd d lmZddlm Z m!Z!m"Z"ddl#m$Z$ddl#m%Z&ddl'm(Z(dddddddgZ)dZ*dZ+dZ,dZ-de.fdYZ/dee/efd YZ0d!Z1d"ee/fd#YZ2d$Z3d%eefd&YZ4d'e2fd(YZ5dS()uv The :mod:`sklearn.feature_extraction.text` submodule gathers utilities to build feature vectors from text documents. i(tunicode_literalstdivisionN(t defaultdict(t itemgetteri(t BaseEstimatortTransformerMixin(tsix(txrange(t normalizei(t FeatureHasher(tENGLISH_STOP_WORDS(tcheck_is_fittedt check_arrayt FLOAT_DTYPES(t sp_version(t_Mapping(t _IS_32BITuCountVectorizeruENGLISH_STOP_WORDSuTfidfTransformeruTfidfVectorizerustrip_accents_asciiustrip_accents_unicodeu strip_tagscC sUtjd|}||kr"|Sdjg|D]}tj|s/|^q/SdS(uTransform accentuated unicode symbols into their simple counterpart Warning: the python-level loop and join operations make this implementation 20 times slower than the strip_accents_ascii basic normalization. Parameters ---------- s : string The string to strip See also -------- strip_accents_ascii Remove accentuated char for any unicode symbol that has a direct ASCII equivalent. uNFKDuN(t unicodedataRtjoint combining(tst normalizedtc((s>lib/python2.7/site-packages/sklearn/feature_extraction/text.pytstrip_accents_unicode0s cC s+tjd|}|jddjdS(uiTransform accentuated unicode symbols into ascii or nothing Warning: this solution is only suited for languages that have a direct transliteration to ASCII symbols. Parameters ---------- s : string The string to strip See also -------- strip_accents_unicode Remove accentuated char for any unicode symbol. uNFKDuASCIIuignore(RRtencodetdecode(Rt nkfd_form((s>lib/python2.7/site-packages/sklearn/feature_extraction/text.pytstrip_accents_asciiIscC s"tjddtjjd|S(uBasic regexp based HTML / XML tag stripper function For serious HTML/XML preprocessing you should rather use an external library such as lxml or BeautifulSoup. Parameters ---------- s : string The string to strip u <([^>]+)>tflagsu (tretcompiletUNICODEtsub(R((s>lib/python2.7/site-packages/sklearn/feature_extraction/text.pyt strip_tags]s cC sS|dkrtSt|tjr5td|n|dkrEdSt|SdS(Nuenglishunot a built-in stop list: %s(R t isinstanceRt string_typest ValueErrortNonet frozenset(tstop((s>lib/python2.7/site-packages/sklearn/feature_extraction/text.pyt_check_stop_listks  tVectorizerMixincB seZdZejdZdZddZdZ dZ dZ dZ dZ d Zd Zd Zd Zd ZRS(u?Provides common code for text vectorizers (tokenization logic).u\s\s+cC s|jdkr6t|d}|j}WdQXn|jdkrT|j}nt|tr~|j|j|j}n|tj krt dn|S(uDecode the input into a string of unicode symbols The decoding strategy depends on the vectorizer parameters. Parameters ---------- doc : string The string to decode ufilenameurbNufileu?np.nan is an invalid document, expected byte or unicode string.( tinputtopentreadR"tbytesRtencodingt decode_errortnptnanR$(tselftdoctfh((s>lib/python2.7/site-packages/sklearn/feature_extraction/text.pyR{s c C s |dk r4g|D]}||kr|^q}n|j\}}|dkr|}|dkrzt|}|d7}ng}t|}|j}dj} xet|t|d|dD]@} x7t|| dD]!} || || | | !qWqWn|S(uATurn tokens into a sequence of n-grams after stop words filteringiu N(R%t ngram_rangetlisttlentappendRRtmin( R2ttokenst stop_wordstwtmin_ntmax_ntoriginal_tokenstn_original_tokenst tokens_appendt space_jointnti((s>lib/python2.7/site-packages/sklearn/feature_extraction/text.pyt _word_ngramss" (        &c C s|jjd|}t|}|j\}}|dkrUt|}|d7}ng}|j}x\t|t|d|dD]:}x1t||dD]}|||||!qWqW|S(u;Tokenize text_document into a sequence of character n-gramsu i(t _white_spacesR R7R5R6R8RR9( R2t text_documentttext_lenR=R>tngramst ngrams_appendRCRD((s>lib/python2.7/site-packages/sklearn/feature_extraction/text.pyt _char_ngramss     'c C s|jjd|}|j\}}g}|j}x|jD]}d|d}t|}x|t||dD]g}d} ||| | |!x3| ||kr| d7} ||| | |!qW| dkrtPqtqtWq@W|S(uWhitespace sensitive char-n-gram tokenization. Tokenize text_document into a sequence of character n-grams operating only inside word boundaries. n-grams at the edges of words are padded with space.u ii(RFR R5R8tsplitR7R( R2RGR=R>RIRJR<tw_lenRCtoffset((s>lib/python2.7/site-packages/sklearn/feature_extraction/text.pyt_char_wb_ngramss      c s|jdk r|jSd}|js1|n^t|jrL|jnC|jdkrdtn+|jdkr|tntd|j|jrfdSSdS(u<Return a function to preprocess the text before tokenizationcS s|S(N((tx((s>lib/python2.7/site-packages/sklearn/feature_extraction/text.pytsuasciiuunicodeu%Invalid value for "strip_accents": %sc s|jS(N(tlower(RP(t strip_accents(s>lib/python2.7/site-packages/sklearn/feature_extraction/text.pyRQsN(t preprocessorR%RStcallableRRR$t lowercase(R2tnoop((RSs>lib/python2.7/site-packages/sklearn/feature_extraction/text.pytbuild_preprocessors          c s5|jdk r|jStj|jfdS(u@Return a function that splits a string into a sequence of tokensc s j|S(N(tfindall(R3(t token_pattern(s>lib/python2.7/site-packages/sklearn/feature_extraction/text.pyRQ sN(t tokenizerR%RRRZ(R2((RZs>lib/python2.7/site-packages/sklearn/feature_extraction/text.pytbuild_tokenizerscC s t|jS(u,Build or fetch the effective stop words list(R(R;(R2((s>lib/python2.7/site-packages/sklearn/feature_extraction/text.pytget_stop_words scC st|jt|ddkr%dSyt}xY|p=dD]K}t|||}x*|D]"}||krc|j|qcqcWq>Wt|j|_|rtj dt |n| SWn$t k rt|j|_dSXdS(uCheck if stop words are consistent Returns ------- is_consistent : True if stop words are consistent with the preprocessor and tokenizer, False if they are not, None if the check was previously performed, "error" if it could not be performed (e.g. because of the use of a custom preprocessor / tokenizer) u_stop_words_idu}Your stop_words may be inconsistent with your preprocessing. Tokenizing the stop words generated tokens %r not in stop_words.uerrorN(( tidR;tgetattrR%tsetR6taddt_stop_words_idtwarningstwarntsortedt Exception(R2R;t preprocessttokenizet inconsistentR<R:ttoken((s>lib/python2.7/site-packages/sklearn/feature_extraction/text.pyt_check_stop_words_consistencys" !      c stjrjSjjdkrAfdSjdkr`fdSjdkrjjjfdStdjdS( u=Return a callable that handles preprocessing and tokenizationucharc sjj|S(N(RKR(R3(RgR2(s>lib/python2.7/site-packages/sklearn/feature_extraction/text.pyRQ=suchar_wbc sjj|S(N(ROR(R3(RgR2(s>lib/python2.7/site-packages/sklearn/feature_extraction/text.pyRQ@suwordc s%jj|S(N(RER(R3(RgR2R;Rh(s>lib/python2.7/site-packages/sklearn/feature_extraction/text.pyRQHsu.%s is not a valid tokenization scheme/analyzerN(RUtanalyzerRXR]R\RkR$(R2((RgR2R;Rhs>lib/python2.7/site-packages/sklearn/feature_extraction/text.pytbuild_analyzer5s    cC sf|j}|dk rYt|tr3t|}nt|tsi}xKt|D]=\}}|j|||krUd|}t|qUqUW|}ntt j |}t |t |krtdnxKt t |D]7}||krdt ||f}t|qqW|s>tdnt |_t||_n t|_dS(Nu Duplicate term in vocabulary: %ru%Vocabulary contains repeated indices.u/Vocabulary of size %d doesn't contain index %d.uempty vocabulary passed to fit(t vocabularyR%R"R`RetMappingt enumeratet setdefaultR$Rt itervaluesR7RtTruetfixed_vocabulary_tdictt vocabulary_tFalse(R2RntvocabRDtttmsgtindices((s>lib/python2.7/site-packages/sklearn/feature_extraction/text.pyt_validate_vocabularyOs0      cC sDd}t|dd|ft|jdkr@tdndS(u4Check if vocabulary is empty or missing (not fit-ed)u$%(name)s - Vocabulary wasn't fitted.u vocabulary_RziuVocabulary is emptyN(R R7RvR$(R2Rz((s>lib/python2.7/site-packages/sklearn/feature_extraction/text.pyt_check_vocabularykscC s;|j\}}||kr7tdt|jndS(u'Check validity of ngram_range parameteruOInvalid value for ngram_range=%s lower boundary larger than the upper boundary.N(R5R$tstr(R2R=tmax_m((s>lib/python2.7/site-packages/sklearn/feature_extraction/text.pyt_validate_paramsss  N(t__name__t __module__t__doc__RRRFRR%RERKRORXR\R]RkRmR|R}R(((s>lib/python2.7/site-packages/sklearn/feature_extraction/text.pyR)vs        %   tHashingVectorizercB seZdZddddeddddddded eeejd Zdd Z dd Z d Z ddZ dZ RS(uConvert a collection of text documents to a matrix of token occurrences It turns a collection of text documents into a scipy.sparse matrix holding token occurrence counts (or binary occurrence information), possibly normalized as token frequencies if norm='l1' or projected on the euclidean unit sphere if norm='l2'. This text vectorizer implementation uses the hashing trick to find the token string name to feature integer index mapping. This strategy has several advantages: - it is very low memory scalable to large datasets as there is no need to store a vocabulary dictionary in memory - it is fast to pickle and un-pickle as it holds no state besides the constructor parameters - it can be used in a streaming (partial fit) or parallel pipeline as there is no state computed during fit. There are also a couple of cons (vs using a CountVectorizer with an in-memory vocabulary): - there is no way to compute the inverse transform (from feature indices to string feature names) which can be a problem when trying to introspect which features are most important to a model. - there can be collisions: distinct tokens can be mapped to the same feature index. However in practice this is rarely an issue if n_features is large enough (e.g. 2 ** 18 for text classification problems). - no IDF weighting as this would render the transformer stateful. The hash function employed is the signed 32-bit version of Murmurhash3. Read more in the :ref:`User Guide `. Parameters ---------- input : string {'filename', 'file', 'content'} If 'filename', the sequence passed as an argument to fit is expected to be a list of filenames that need reading to fetch the raw content to analyze. If 'file', the sequence items must have a 'read' method (file-like object) that is called to fetch the bytes in memory. Otherwise the input is expected to be the sequence strings or bytes items are expected to be analyzed directly. encoding : string, default='utf-8' If bytes or files are given to analyze, this encoding is used to decode. decode_error : {'strict', 'ignore', 'replace'} Instruction on what to do if a byte sequence is given to analyze that contains characters not of the given `encoding`. By default, it is 'strict', meaning that a UnicodeDecodeError will be raised. Other values are 'ignore' and 'replace'. strip_accents : {'ascii', 'unicode', None} Remove accents and perform other character normalization during the preprocessing step. 'ascii' is a fast method that only works on characters that have an direct ASCII mapping. 'unicode' is a slightly slower method that works on any characters. None (default) does nothing. Both 'ascii' and 'unicode' use NFKD normalization from :func:`unicodedata.normalize`. lowercase : boolean, default=True Convert all characters to lowercase before tokenizing. preprocessor : callable or None (default) Override the preprocessing (string transformation) stage while preserving the tokenizing and n-grams generation steps. tokenizer : callable or None (default) Override the string tokenization step while preserving the preprocessing and n-grams generation steps. Only applies if ``analyzer == 'word'``. stop_words : string {'english'}, list, or None (default) If 'english', a built-in stop word list for English is used. There are several known issues with 'english' and you should consider an alternative (see :ref:`stop_words`). If a list, that list is assumed to contain stop words, all of which will be removed from the resulting tokens. Only applies if ``analyzer == 'word'``. token_pattern : string Regular expression denoting what constitutes a "token", only used if ``analyzer == 'word'``. The default regexp selects tokens of 2 or more alphanumeric characters (punctuation is completely ignored and always treated as a token separator). ngram_range : tuple (min_n, max_n), default=(1, 1) The lower and upper boundary of the range of n-values for different n-grams to be extracted. All values of n such that min_n <= n <= max_n will be used. analyzer : string, {'word', 'char', 'char_wb'} or callable Whether the feature should be made of word or character n-grams. Option 'char_wb' creates character n-grams only from text inside word boundaries; n-grams at the edges of words are padded with space. If a callable is passed it is used to extract the sequence of features out of the raw, unprocessed input. n_features : integer, default=(2 ** 20) The number of features (columns) in the output matrices. Small numbers of features are likely to cause hash collisions, but large numbers will cause larger coefficient dimensions in linear learners. binary : boolean, default=False. If True, all non zero counts are set to 1. This is useful for discrete probabilistic models that model binary events rather than integer counts. norm : 'l1', 'l2' or None, optional Norm used to normalize term vectors. None for no normalization. alternate_sign : boolean, optional, default True When True, an alternating sign is added to the features as to approximately conserve the inner product in the hashed space even for small n_features. This approach is similar to sparse random projection. .. versionadded:: 0.19 non_negative : boolean, optional, default False When True, an absolute value is applied to the features matrix prior to returning it. When used in conjunction with alternate_sign=True, this significantly reduces the inner product preservation property. .. deprecated:: 0.19 This option will be removed in 0.21. dtype : type, optional Type of the matrix returned by fit_transform() or transform(). Examples -------- >>> from sklearn.feature_extraction.text import HashingVectorizer >>> corpus = [ ... 'This is the first document.', ... 'This document is the second document.', ... 'And this is the third one.', ... 'Is this the first document?', ... ] >>> vectorizer = HashingVectorizer(n_features=2**4) >>> X = vectorizer.fit_transform(corpus) >>> print(X.shape) (4, 16) See also -------- CountVectorizer, TfidfVectorizer ucontentuutf-8ustrictu (?u)\b\w\w+\biuwordiiul2cC s||_||_||_||_||_||_| |_||_| |_||_ | |_ | |_ | |_ ||_ ||_||_||_dS(N(R*R.R/RSRTR[RlRVRZR;t n_featuresR5tbinarytnormtalternate_signt non_negativetdtype(R2R*R.R/RSRVRTR[R;RZR5RlRRRRRR((s>lib/python2.7/site-packages/sklearn/feature_extraction/text.pyt__init__ s"                cC s|S(uDoes nothing: this transformer is stateless. This method is just there to mark the fact that this transformer can work in a streaming setup. Parameters ---------- X : array-like, shape [n_samples, n_features] Training data. ((R2tXty((s>lib/python2.7/site-packages/sklearn/feature_extraction/text.pyt partial_fit9s cC sHt|tjr!tdn|j|jj|d||S(uDoes nothing: this transformer is stateless. Parameters ---------- X : array-like, shape [n_samples, n_features] Training data. uBIterable over raw text documents expected, string object received.R(R"RR#R$Rt _get_hashertfit(R2RR((s>lib/python2.7/site-packages/sklearn/feature_extraction/text.pyRFs   c st|tjr!tdn|j|j|jjfd|D}|jrx|j j dn|j dk rt |d|j dt}n|S(u Transform a sequence of documents to a document-term matrix. Parameters ---------- X : iterable over raw text documents, length = n_samples Samples. Each sample must be a text document (either bytes or unicode strings, file name or file object depending on the constructor argument) which will be tokenized and hashed. Returns ------- X : scipy.sparse matrix, shape = (n_samples, self.n_features) Document-term matrix. uBIterable over raw text documents expected, string object received.c3 s|]}|VqdS(N((t.0R3(Rl(s>lib/python2.7/site-packages/sklearn/feature_extraction/text.pys psiRtcopyN(R"RR#R$RRmRt transformRtdatatfillRR%RRw(R2R((Rls>lib/python2.7/site-packages/sklearn/feature_extraction/text.pyRYs   % cC s|j||j|S(uTransform a sequence of documents to a document-term matrix. Parameters ---------- X : iterable over raw text documents, length = n_samples Samples. Each sample must be a text document (either bytes or unicode strings, file name or file object depending on the constructor argument) which will be tokenized and hashed. y : any Ignored. This parameter exists only for compatibility with sklearn.pipeline.Pipeline. Returns ------- X : scipy.sparse matrix, shape = (n_samples, self.n_features) Document-term matrix. (RR(R2RR((s>lib/python2.7/site-packages/sklearn/feature_extraction/text.pyt fit_transformwsc C s1td|jddd|jd|jd|jS(NRt input_typeustringRRR(R RRRR(R2((s>lib/python2.7/site-packages/sklearn/feature_extraction/text.pyRs N(iii(RRRR%RsRwR0tfloat64RRRRRR(((s>lib/python2.7/site-packages/sklearn/feature_extraction/text.pyR}s      cC s@tj|r,tj|jd|jdStj|jSdS(uACount the number of non-zero values for each feature in sparse X.t minlengthiN(tsptisspmatrix_csrR0tbincountR{tshapetdifftindptr(R((s>lib/python2.7/site-packages/sklearn/feature_extraction/text.pyt_document_frequencystCountVectorizercB seZdZddddeddddddddddeejdZd Z dddd Z d Z dd Z dd Z dZdZdZRS(uConvert a collection of text documents to a matrix of token counts This implementation produces a sparse representation of the counts using scipy.sparse.csr_matrix. If you do not provide an a-priori dictionary and you do not use an analyzer that does some kind of feature selection then the number of features will be equal to the vocabulary size found by analyzing the data. Read more in the :ref:`User Guide `. Parameters ---------- input : string {'filename', 'file', 'content'} If 'filename', the sequence passed as an argument to fit is expected to be a list of filenames that need reading to fetch the raw content to analyze. If 'file', the sequence items must have a 'read' method (file-like object) that is called to fetch the bytes in memory. Otherwise the input is expected to be the sequence strings or bytes items are expected to be analyzed directly. encoding : string, 'utf-8' by default. If bytes or files are given to analyze, this encoding is used to decode. decode_error : {'strict', 'ignore', 'replace'} Instruction on what to do if a byte sequence is given to analyze that contains characters not of the given `encoding`. By default, it is 'strict', meaning that a UnicodeDecodeError will be raised. Other values are 'ignore' and 'replace'. strip_accents : {'ascii', 'unicode', None} Remove accents and perform other character normalization during the preprocessing step. 'ascii' is a fast method that only works on characters that have an direct ASCII mapping. 'unicode' is a slightly slower method that works on any characters. None (default) does nothing. Both 'ascii' and 'unicode' use NFKD normalization from :func:`unicodedata.normalize`. lowercase : boolean, True by default Convert all characters to lowercase before tokenizing. preprocessor : callable or None (default) Override the preprocessing (string transformation) stage while preserving the tokenizing and n-grams generation steps. tokenizer : callable or None (default) Override the string tokenization step while preserving the preprocessing and n-grams generation steps. Only applies if ``analyzer == 'word'``. stop_words : string {'english'}, list, or None (default) If 'english', a built-in stop word list for English is used. There are several known issues with 'english' and you should consider an alternative (see :ref:`stop_words`). If a list, that list is assumed to contain stop words, all of which will be removed from the resulting tokens. Only applies if ``analyzer == 'word'``. If None, no stop words will be used. max_df can be set to a value in the range [0.7, 1.0) to automatically detect and filter stop words based on intra corpus document frequency of terms. token_pattern : string Regular expression denoting what constitutes a "token", only used if ``analyzer == 'word'``. The default regexp select tokens of 2 or more alphanumeric characters (punctuation is completely ignored and always treated as a token separator). ngram_range : tuple (min_n, max_n) The lower and upper boundary of the range of n-values for different n-grams to be extracted. All values of n such that min_n <= n <= max_n will be used. analyzer : string, {'word', 'char', 'char_wb'} or callable Whether the feature should be made of word or character n-grams. Option 'char_wb' creates character n-grams only from text inside word boundaries; n-grams at the edges of words are padded with space. If a callable is passed it is used to extract the sequence of features out of the raw, unprocessed input. max_df : float in range [0.0, 1.0] or int, default=1.0 When building the vocabulary ignore terms that have a document frequency strictly higher than the given threshold (corpus-specific stop words). If float, the parameter represents a proportion of documents, integer absolute counts. This parameter is ignored if vocabulary is not None. min_df : float in range [0.0, 1.0] or int, default=1 When building the vocabulary ignore terms that have a document frequency strictly lower than the given threshold. This value is also called cut-off in the literature. If float, the parameter represents a proportion of documents, integer absolute counts. This parameter is ignored if vocabulary is not None. max_features : int or None, default=None If not None, build a vocabulary that only consider the top max_features ordered by term frequency across the corpus. This parameter is ignored if vocabulary is not None. vocabulary : Mapping or iterable, optional Either a Mapping (e.g., a dict) where keys are terms and values are indices in the feature matrix, or an iterable over terms. If not given, a vocabulary is determined from the input documents. Indices in the mapping should not be repeated and should not have any gap between 0 and the largest index. binary : boolean, default=False If True, all non zero counts are set to 1. This is useful for discrete probabilistic models that model binary events rather than integer counts. dtype : type, optional Type of the matrix returned by fit_transform() or transform(). Attributes ---------- vocabulary_ : dict A mapping of terms to feature indices. stop_words_ : set Terms that were ignored because they either: - occurred in too many documents (`max_df`) - occurred in too few documents (`min_df`) - were cut off by feature selection (`max_features`). This is only available if no vocabulary was given. Examples -------- >>> from sklearn.feature_extraction.text import CountVectorizer >>> corpus = [ ... 'This is the first document.', ... 'This document is the second document.', ... 'And this is the third one.', ... 'Is this the first document?', ... ] >>> vectorizer = CountVectorizer() >>> X = vectorizer.fit_transform(corpus) >>> print(vectorizer.get_feature_names()) ['and', 'document', 'first', 'is', 'one', 'second', 'the', 'third', 'this'] >>> print(X.toarray()) # doctest: +NORMALIZE_WHITESPACE [[0 1 1 1 0 0 1 0 1] [0 2 0 1 0 1 1 0 1] [1 0 0 1 1 0 1 1 1] [0 1 1 1 0 0 1 0 1]] See also -------- HashingVectorizer, TfidfVectorizer Notes ----- The ``stop_words_`` attribute can get large and increase the model size when pickling. This attribute is provided only for introspection and can be safely removed using delattr or set to None before pickling. ucontentuutf-8ustrictu (?u)\b\w\w+\biuwordg?cC s||_||_||_||_||_||_| |_||_| |_||_ | |_ | |_ | dks| dkrt dn||_ |dk rt|tj s|dkrt d|qn| |_||_||_||_dS(Niu#negative value for max_df or min_dfu4max_features=%r, neither a positive integer nor None(R*R.R/RSRTR[RlRVRZR;tmax_dftmin_dfR$t max_featuresR%R"tnumberstIntegralR5RnRR(R2R*R.R/RSRVRTR[R;RZR5RlRRRRnRR((s>lib/python2.7/site-packages/sklearn/feature_extraction/text.pyREs2                  cC sttj|}tjt|d|jj}x4t|D]&\}\}}|||<|||lib/python2.7/site-packages/sklearn/feature_extraction/text.pyt_sort_featuresfs! cC s|dkr1|dkr1|dkr1|tfSt|}tj|jddj}tjt|dt }|dk r|||kM}n|dk r|||kM}n|dk r)|j|kr)|| j | } tj t|dt } t | tj |d| <| }ntj|d} t} xRttj|D];\} }||r| ||| lib/python2.7/site-packages/sklearn/feature_extraction/text.pyt_limit_featurests2 $  !    " cC s|r|j}nt}|j|_|j}g}g}t}|jdx|D]}i} xb||D]T} y7|| } | | krd| | lib/python2.7/site-packages/sklearn/feature_extraction/text.pyt _count_vocabsP                  cC s|j||S(u Learn a vocabulary dictionary of all tokens in the raw documents. Parameters ---------- raw_documents : iterable An iterable which yields either str, unicode or file objects. Returns ------- self (R(R2RR((s>lib/python2.7/site-packages/sklearn/feature_extraction/text.pyRs c C sBt|tjr!tdn|j|j|j}|j}|j}|j ||j \}}|j r|j j dn|j s>|j||}|jd}t|tjr|n||} t|tjr|n||} | | krtdn|j||| | |\}|_||_n|S(uLearn the vocabulary dictionary and return term-document matrix. This is equivalent to fit followed by transform, but more efficiently implemented. Parameters ---------- raw_documents : iterable An iterable which yields either str, unicode or file objects. Returns ------- X : array, [n_samples, n_features] Document-term matrix. uBIterable over raw text documents expected, string object received.iiu-max_df corresponds to < documents than min_df(R"RR#R$RR|RRRRRtRRRRRRRRt stop_words_Rv( R2RRRRRRnRtn_doct max_doc_countt min_doc_count((s>lib/python2.7/site-packages/sklearn/feature_extraction/text.pyRs8                cC st|tjr!tdnt|ds=|jn|j|j|dt\}}|j r~|j j dn|S(uTransform documents to document-term matrix. Extract token counts out of raw text documents using the vocabulary fitted with fit or the one provided to the constructor. Parameters ---------- raw_documents : iterable An iterable which yields either str, unicode or file objects. Returns ------- X : sparse matrix, [n_samples, n_features] Document-term matrix. uBIterable over raw text documents expected, string object received.u vocabulary_Ri( R"RR#R$thasattrR|R}RRsRRR(R2Rt_R((s>lib/python2.7/site-packages/sklearn/feature_extraction/text.pyR"s    cC s|jtj|r(|j}ntj|}|jd}tjt|j j }tjt|j j }|tj |}gt |D]0}|||ddfjdj^qS(uReturn terms per document with nonzero entries in X. Parameters ---------- X : {array, sparse matrix}, shape = [n_samples, n_features] Returns ------- X_inv : list of arrays, len = n_samples List of arrays of terms. iNi(R}RtissparsettocsrR0tasmatrixRtarrayR6RvRRRtrangetnonzeroR(R2Rt n_samplesttermsR{tinverse_vocabularyRD((s>lib/python2.7/site-packages/sklearn/feature_extraction/text.pytinverse_transformBs  cC sat|ds|jn|jgttj|jdtdD]\}}|^qKS(u:Array mapping from feature integer indices to feature nameu vocabulary_tkeyi(RR|R}ReRRRvR(R2RyRD((s>lib/python2.7/site-packages/sklearn/feature_extraction/text.pytget_feature_names`s   N(ii(RRRR%RsRwR0RRRRRRRRRR(((s>lib/python2.7/site-packages/sklearn/feature_extraction/text.pyRs"    ( <  : cC stjtdS(uEConstruct an array.array of a type suitable for scipy.sparse indices.ui(RR~(((s>lib/python2.7/site-packages/sklearn/feature_extraction/text.pyRkstTfidfTransformercB s\eZdZdeeedZddZedZe dZ e j dZ RS(u Transform a count matrix to a normalized tf or tf-idf representation Tf means term-frequency while tf-idf means term-frequency times inverse document-frequency. This is a common term weighting scheme in information retrieval, that has also found good use in document classification. The goal of using tf-idf instead of the raw frequencies of occurrence of a token in a given document is to scale down the impact of tokens that occur very frequently in a given corpus and that are hence empirically less informative than features that occur in a small fraction of the training corpus. The formula that is used to compute the tf-idf for a term t of a document d in a document set is tf-idf(t, d) = tf(t, d) * idf(t), and the idf is computed as idf(t) = log [ n / df(t) ] + 1 (if ``smooth_idf=False``), where n is the total number of documents in the document set and df(t) is the document frequency of t; the document frequency is the number of documents in the document set that contain the term t. The effect of adding "1" to the idf in the equation above is that terms with zero idf, i.e., terms that occur in all documents in a training set, will not be entirely ignored. (Note that the idf formula above differs from the standard textbook notation that defines the idf as idf(t) = log [ n / (df(t) + 1) ]). If ``smooth_idf=True`` (the default), the constant "1" is added to the numerator and denominator of the idf as if an extra document was seen containing every term in the collection exactly once, which prevents zero divisions: idf(d, t) = log [ (1 + n) / (1 + df(d, t)) ] + 1. Furthermore, the formulas used to compute tf and idf depend on parameter settings that correspond to the SMART notation used in IR as follows: Tf is "n" (natural) by default, "l" (logarithmic) when ``sublinear_tf=True``. Idf is "t" when use_idf is given, "n" (none) otherwise. Normalization is "c" (cosine) when ``norm='l2'``, "n" (none) when ``norm=None``. Read more in the :ref:`User Guide `. Parameters ---------- norm : 'l1', 'l2' or None, optional (default='l2') Each output row will have unit norm, either: * 'l2': Sum of squares of vector elements is 1. The cosine similarity between two vectors is their dot product when l2 norm has been applied. * 'l1': Sum of absolute values of vector elements is 1. See :func:`preprocessing.normalize` use_idf : boolean (default=True) Enable inverse-document-frequency reweighting. smooth_idf : boolean (default=True) Smooth idf weights by adding one to document frequencies, as if an extra document was seen containing every term in the collection exactly once. Prevents zero divisions. sublinear_tf : boolean (default=False) Apply sublinear tf scaling, i.e. replace tf with 1 + log(tf). Attributes ---------- idf_ : array, shape (n_features) The inverse document frequency (IDF) vector; only defined if ``use_idf`` is True. References ---------- .. [Yates2011] `R. Baeza-Yates and B. Ribeiro-Neto (2011). Modern Information Retrieval. Addison Wesley, pp. 68-74.` .. [MRS2008] `C.D. Manning, P. Raghavan and H. Schütze (2008). Introduction to Information Retrieval. Cambridge University Press, pp. 118-120.` ul2cC s(||_||_||_||_dS(N(Rtuse_idft smooth_idft sublinear_tf(R2RRRR((s>lib/python2.7/site-packages/sklearn/feature_extraction/text.pyRs   c C st|dd }tj|s3tj|}n|jtkrK|jntj}|jr|j \}}t |j |}|t |j 7}|t |j 7}tj||d}tj|ddd||fddd ||_n|S( uLearn the idf vector (global term weights) Parameters ---------- X : sparse matrix, [n_samples, n_features] a matrix of term/token counts t accept_sparseucsrucscitoffsetsiRRR(ucsrucsc(R RRRRR R0RRRRtastypetintRtlogtdiagst _idf_diag(R2RRRRRtdftidf((s>lib/python2.7/site-packages/sklearn/feature_extraction/text.pyRs!  cC st|dddtd|}tj|sHtj|dtj}n|j\}}|jrtj |j |j |j d7_ n|j rt |dd|j jd}||krtd ||fn||j }n|jr t|d |jdt}n|S( uTransform a count matrix to a tf or tf-idf representation Parameters ---------- X : sparse matrix, [n_samples, n_features] a matrix of term/token counts copy : boolean, default True Whether to copy X and operate on the copy or perform in-place operations. Returns ------- vectors : sparse matrix, [n_samples, n_features] RucsrRRiu _idf_diaguidf vector is not fittediuKInput has n_features=%d while the model has been trained with n_features=%dR(R R RRRR0RRRRRRR RR$RRRw(R2RRRRtexpected_n_features((s>lib/python2.7/site-packages/sklearn/feature_extraction/text.pyRs"    cC stj|jjddS(NRi(R0RRR(R2((s>lib/python2.7/site-packages/sklearn/feature_extraction/text.pytidf_sc C sStj|dtj}|jd}tj|ddd|d|dd|_dS(NRiRtmRCRucsr(R0RRRRtspdiagsR(R2tvalueR((s>lib/python2.7/site-packages/sklearn/feature_extraction/text.pyRs N( RRRRsRwRR%RRtpropertyRtsetter(((s>lib/python2.7/site-packages/sklearn/feature_extraction/text.pyRpsO    *tTfidfVectorizercB s+eZdZddddeddddddddddeejdeeed Ze d Z e j d Z e d Z e j d Z e dZ e j dZ e dZej dZe dZej dZdZddZddZedZRS(u8Convert a collection of raw documents to a matrix of TF-IDF features. Equivalent to :class:`CountVectorizer` followed by :class:`TfidfTransformer`. Read more in the :ref:`User Guide `. Parameters ---------- input : string {'filename', 'file', 'content'} If 'filename', the sequence passed as an argument to fit is expected to be a list of filenames that need reading to fetch the raw content to analyze. If 'file', the sequence items must have a 'read' method (file-like object) that is called to fetch the bytes in memory. Otherwise the input is expected to be the sequence strings or bytes items are expected to be analyzed directly. encoding : string, 'utf-8' by default. If bytes or files are given to analyze, this encoding is used to decode. decode_error : {'strict', 'ignore', 'replace'} (default='strict') Instruction on what to do if a byte sequence is given to analyze that contains characters not of the given `encoding`. By default, it is 'strict', meaning that a UnicodeDecodeError will be raised. Other values are 'ignore' and 'replace'. strip_accents : {'ascii', 'unicode', None} (default=None) Remove accents and perform other character normalization during the preprocessing step. 'ascii' is a fast method that only works on characters that have an direct ASCII mapping. 'unicode' is a slightly slower method that works on any characters. None (default) does nothing. Both 'ascii' and 'unicode' use NFKD normalization from :func:`unicodedata.normalize`. lowercase : boolean (default=True) Convert all characters to lowercase before tokenizing. preprocessor : callable or None (default=None) Override the preprocessing (string transformation) stage while preserving the tokenizing and n-grams generation steps. tokenizer : callable or None (default=None) Override the string tokenization step while preserving the preprocessing and n-grams generation steps. Only applies if ``analyzer == 'word'``. analyzer : string, {'word', 'char', 'char_wb'} or callable Whether the feature should be made of word or character n-grams. Option 'char_wb' creates character n-grams only from text inside word boundaries; n-grams at the edges of words are padded with space. If a callable is passed it is used to extract the sequence of features out of the raw, unprocessed input. stop_words : string {'english'}, list, or None (default=None) If a string, it is passed to _check_stop_list and the appropriate stop list is returned. 'english' is currently the only supported string value. There are several known issues with 'english' and you should consider an alternative (see :ref:`stop_words`). If a list, that list is assumed to contain stop words, all of which will be removed from the resulting tokens. Only applies if ``analyzer == 'word'``. If None, no stop words will be used. max_df can be set to a value in the range [0.7, 1.0) to automatically detect and filter stop words based on intra corpus document frequency of terms. token_pattern : string Regular expression denoting what constitutes a "token", only used if ``analyzer == 'word'``. The default regexp selects tokens of 2 or more alphanumeric characters (punctuation is completely ignored and always treated as a token separator). ngram_range : tuple (min_n, max_n) (default=(1, 1)) The lower and upper boundary of the range of n-values for different n-grams to be extracted. All values of n such that min_n <= n <= max_n will be used. max_df : float in range [0.0, 1.0] or int (default=1.0) When building the vocabulary ignore terms that have a document frequency strictly higher than the given threshold (corpus-specific stop words). If float, the parameter represents a proportion of documents, integer absolute counts. This parameter is ignored if vocabulary is not None. min_df : float in range [0.0, 1.0] or int (default=1) When building the vocabulary ignore terms that have a document frequency strictly lower than the given threshold. This value is also called cut-off in the literature. If float, the parameter represents a proportion of documents, integer absolute counts. This parameter is ignored if vocabulary is not None. max_features : int or None (default=None) If not None, build a vocabulary that only consider the top max_features ordered by term frequency across the corpus. This parameter is ignored if vocabulary is not None. vocabulary : Mapping or iterable, optional (default=None) Either a Mapping (e.g., a dict) where keys are terms and values are indices in the feature matrix, or an iterable over terms. If not given, a vocabulary is determined from the input documents. binary : boolean (default=False) If True, all non-zero term counts are set to 1. This does not mean outputs will have only 0/1 values, only that the tf term in tf-idf is binary. (Set idf and normalization to False to get 0/1 outputs.) dtype : type, optional (default=float64) Type of the matrix returned by fit_transform() or transform(). norm : 'l1', 'l2' or None, optional (default='l2') Each output row will have unit norm, either: * 'l2': Sum of squares of vector elements is 1. The cosine similarity between two vectors is their dot product when l2 norm has been applied. * 'l1': Sum of absolute values of vector elements is 1. See :func:`preprocessing.normalize` use_idf : boolean (default=True) Enable inverse-document-frequency reweighting. smooth_idf : boolean (default=True) Smooth idf weights by adding one to document frequencies, as if an extra document was seen containing every term in the collection exactly once. Prevents zero divisions. sublinear_tf : boolean (default=False) Apply sublinear tf scaling, i.e. replace tf with 1 + log(tf). Attributes ---------- vocabulary_ : dict A mapping of terms to feature indices. idf_ : array, shape (n_features) The inverse document frequency (IDF) vector; only defined if ``use_idf`` is True. stop_words_ : set Terms that were ignored because they either: - occurred in too many documents (`max_df`) - occurred in too few documents (`min_df`) - were cut off by feature selection (`max_features`). This is only available if no vocabulary was given. Examples -------- >>> from sklearn.feature_extraction.text import TfidfVectorizer >>> corpus = [ ... 'This is the first document.', ... 'This document is the second document.', ... 'And this is the third one.', ... 'Is this the first document?', ... ] >>> vectorizer = TfidfVectorizer() >>> X = vectorizer.fit_transform(corpus) >>> print(vectorizer.get_feature_names()) ['and', 'document', 'first', 'is', 'one', 'second', 'the', 'third', 'this'] >>> print(X.shape) (4, 9) See also -------- CountVectorizer : Transforms text into a sparse matrix of n-gram counts. TfidfTransformer : Performs the TF-IDF transformation from a provided matrix of counts. Notes ----- The ``stop_words_`` attribute can get large and increase the model size when pickling. This attribute is provided only for introspection and can be safely removed using delattr or set to None before pickling. ucontentuutf-8ustrictuwordu (?u)\b\w\w+\big?ul2c#C stt|jd|d|d|d|d|d|d|d|d | d | d | d | d | d|d|d|d|td|d|d|d||_dS(NR*R.R/RSRVRTR[RlR;RZR5RRRRnRRRRRR(tsuperRRRt_tfidf(R2R*R.R/RSRVRTR[RlR;RZR5RRRRnRRRRRR((s>lib/python2.7/site-packages/sklearn/feature_extraction/text.pyRs   cC s |jjS(N(RR(R2((s>lib/python2.7/site-packages/sklearn/feature_extraction/text.pyRscC s||j_dS(N(RR(R2R((s>lib/python2.7/site-packages/sklearn/feature_extraction/text.pyRscC s |jjS(N(RR(R2((s>lib/python2.7/site-packages/sklearn/feature_extraction/text.pyRscC s||j_dS(N(RR(R2R((s>lib/python2.7/site-packages/sklearn/feature_extraction/text.pyRscC s |jjS(N(RR(R2((s>lib/python2.7/site-packages/sklearn/feature_extraction/text.pyRscC s||j_dS(N(RR(R2R((s>lib/python2.7/site-packages/sklearn/feature_extraction/text.pyR scC s |jjS(N(RR(R2((s>lib/python2.7/site-packages/sklearn/feature_extraction/text.pyRscC s||j_dS(N(RR(R2R((s>lib/python2.7/site-packages/sklearn/feature_extraction/text.pyRscC s |jjS(N(RR(R2((s>lib/python2.7/site-packages/sklearn/feature_extraction/text.pyRscC so|jt|dr_t|jt|kr_tdt|t|jfq_n||j_dS(Nu vocabulary_u5idf length = %d must be equal to vocabulary size = %d(R|RR7RvR$RnRR(R2R((s>lib/python2.7/site-packages/sklearn/feature_extraction/text.pyRs  %cC s5|jtkr1tjdjt|jtndS(NuKOnly {} 'dtype' should be used. {} 'dtype' will be converted to np.float64.(RR RcRdRt UserWarning(R2((s>lib/python2.7/site-packages/sklearn/feature_extraction/text.pyt _check_params$s  cC s6|jtt|j|}|jj||S(uLearn vocabulary and idf from training set. Parameters ---------- raw_documents : iterable an iterable which yields either str, unicode or file objects Returns ------- self : TfidfVectorizer (R RRRRR(R2RRR((s>lib/python2.7/site-packages/sklearn/feature_extraction/text.pyR+s cC sH|jtt|j|}|jj||jj|dtS(uLearn vocabulary and idf, return term-document matrix. This is equivalent to fit followed by transform, but more efficiently implemented. Parameters ---------- raw_documents : iterable an iterable which yields either str, unicode or file objects Returns ------- X : sparse matrix, [n_samples, n_features] Tf-idf-weighted document-term matrix. R(R RRRRRRRw(R2RRR((s>lib/python2.7/site-packages/sklearn/feature_extraction/text.pyR<s cC s>t|ddtt|j|}|jj|dtS(u:Transform documents to document-term matrix. Uses the vocabulary and document frequencies (df) learned by fit (or fit_transform). Parameters ---------- raw_documents : iterable an iterable which yields either str, unicode or file objects copy : boolean, default True Whether to copy X and operate on the copy or perform in-place operations. Returns ------- X : sparse matrix, [n_samples, n_features] Tf-idf-weighted document-term matrix. u_tfidfuThe tfidf vector is not fittedR(R RRRRRw(R2RRR((s>lib/python2.7/site-packages/sklearn/feature_extraction/text.pyRSsN(ii(RRRR%RsRwR0RRRRRRRRRR RRR(((s>lib/python2.7/site-packages/sklearn/feature_extraction/text.pyRs.        (6Rt __future__RRRt collectionsRRtoperatorRRRRctnumpyR0t scipy.sparsetsparseRtbaseRRt externalsRtexternals.six.movesRt preprocessingRthashingR R;R tutils.validationR R R t utils.fixesRRRotutilsRt__all__RRR!R(tobjectR)RRRRRR(((s>lib/python2.7/site-packages/sklearn/feature_extraction/text.pyt sR