ó ‡ˆ\c@sïdZddlZddlmZmZddlZddlZddlZddlZddl Z ddl Z ddl Z ddl jZddlmZddlmZddlmZddlmZdd lmZd d lmZd d lmZd d lmZd dlmZd dlmZmZeje ƒZ!eddddddƒZ"dZ#dZ$dZ%edƒd„ƒZ&d„Z'd„Z(e j)dƒZ*d„Z+d„Z,dd de.d!d$e.d"„Z/d d%de.e0d#„Z1dS(&sCaching loader for the 20 newsgroups text classification dataset The description of the dataset is available on the official website at: http://people.csail.mit.edu/jrennie/20Newsgroups/ Quoting the introduction: The 20 Newsgroups data set is a collection of approximately 20,000 newsgroup documents, partitioned (nearly) evenly across 20 different newsgroups. To the best of my knowledge, it was originally collected by Ken Lang, probably for his Newsweeder: Learning to filter netnews paper, though he does not explicitly mention this collection. The 20 newsgroups collection has become a popular data set for experiments in text applications of machine learning techniques, such as text classification and text clustering. This dataset loader will download the recommended "by date" variant of the dataset and which features a point in time split between the train and test sets. The compressed dataset size is around 14 Mb compressed. Once uncompressed the train set is 52 MB and the test set is 34 MB. iÿÿÿÿN(tdirnametjoini(t get_data_home(t load_files(t _pkl_filepath(t _fetch_remote(tRemoteFileMetadatai(tCountVectorizer(t normalize(t deprecated(t_joblib(tcheck_random_statetBunchtfilenames20news-bydate.tar.gzturls.https://ndownloader.figshare.com/files/5975967tchecksumt@8f1b2514ca22a5ade8fbb9cfa5727df95fa587f4c87b786e15c759fa66d95610s20news-bydate.pkzs20news-bydate-trains20news-bydate-tests}Function 'download_20newsgroups' was renamed to '_download_20newsgroups' in version 0.20 and will be removed in release 0.22.cCs t||ƒS(N(t_download_20newsgroups(t target_dirt cache_path((sAlib/python2.7/site-packages/sklearn/datasets/twenty_newsgroups.pytdownload_20newsgroupsAscCs(tjj|tƒ}tjj|tƒ}tjj|ƒsLtj|ƒntjdt j ƒt t d|ƒ}tj d|ƒt j|dƒjd|ƒtj|ƒtdt|ddƒd t|ddƒƒ}tjtj|ƒd ƒ}t|d ƒ}|j|ƒWd QXtj|ƒ|S( sADownload the 20 newsgroups data and stored it as a zipped pickle.s#Downloading dataset from %s (14 MB)RsDecompressing %ssr:gztpathttraintencodingtlatin1ttestt zlib_codectwbN(tosRRt TRAIN_FOLDERt TEST_FOLDERtexiststmakedirstloggertinfotARCHIVERRtdebugttarfiletopent extractalltremovetdictRtcodecstencodetpickletdumpstwritetshutiltrmtree(RRt train_patht test_patht archive_pathtcachetcompressed_contenttf((sAlib/python2.7/site-packages/sklearn/datasets/twenty_newsgroups.pyRHs   cCs|jdƒ\}}}|S(sÝ Given text in "news" format, strip the headers, by removing everything before the first blank line. Parameters ---------- text : string The text from which to remove the signature block. s (t partition(ttextt_beforet _blanklinetafter((sAlib/python2.7/site-packages/sklearn/datasets/twenty_newsgroups.pytstrip_newsgroup_headerbs sF(writes in|writes:|wrote:|says:|said:|^In article|^Quoted from|^\||^>)cCs>g|jdƒD]}tj|ƒs|^q}dj|ƒS(s= Given text in "news" format, strip lines beginning with the quote characters > or |, plus lines that often introduce a quoted section (for example, because they contain the string 'writes:'.) Parameters ---------- text : string The text from which to remove the signature block. s (tsplitt _QUOTE_REtsearchR(R8tlinet good_lines((sAlib/python2.7/site-packages/sklearn/datasets/twenty_newsgroups.pytstrip_newsgroup_quotingts cCsŠ|jƒjdƒ}xMtt|ƒdddƒD]/}||}|jƒjdƒdkr2Pq2q2W|dkr‚dj|| ƒS|SdS(s‰ Given text in "news" format, attempt to remove a signature block. As a rough heuristic, we assume that signatures are set apart by either a blank line or a line made of hyphens, and that it is the last such line in the file (disregarding blank lines at the end). Parameters ---------- text : string The text from which to remove the signature block. s iiÿÿÿÿt-tiN(tstripR=trangetlenR(R8tlinestline_numR@((sAlib/python2.7/site-packages/sklearn/datasets/twenty_newsgroups.pytstrip_newsgroup_footer„s #  Ri*cCs+td|ƒ}t|tƒ}tjj|dƒ}d} tjj|ƒrÉyIt|dƒ} | j ƒ} WdQXt j | dƒ} t j | ƒ} WqÉtk rÅ} ddGHdGHddGH| GHqÉXn| dkr|rtjd ƒtd |d |ƒ} qtd ƒ‚n|dkr(| |}n²|dkrÊtƒ}tƒ}tƒ}xHdD]@}| |}|j|jƒ|j|jƒ|j|jƒqVW||_tj|ƒ|_tj|ƒ|_ntd|ƒ‚ttƒ}tt|ddƒƒ}|j ƒ}WdQX||_d|krPg|jD]}t|ƒ^q2|_nd|kr„g|jD]}t|ƒ^qf|_nd|kr¸g|jD]}t |ƒ^qš|_n|dk r›g|D]}|j!j"|ƒ|f^qË}|j#ƒt$|Œ\}}tj%|j|ƒ}|j||_|j||_tj&||jƒ|_t|ƒ|_!tj|jdt'ƒ}||}|j(ƒ|_n|r't)|ƒ}tj*|jj+dƒ}|j,|ƒ|j||_|j||_tj|jdt'ƒ}||}|j(ƒ|_n|S(sr Load the filenames and data from the 20 newsgroups dataset (classification). Download it if necessary. ================= ========== Classes 20 Samples total 18846 Dimensionality 1 Features text ================= ========== Read more in the :ref:`User Guide <20newsgroups_dataset>`. Parameters ---------- data_home : optional, default: None Specify a download and cache folder for the datasets. If None, all scikit-learn data is stored in '~/scikit_learn_data' subfolders. subset : 'train' or 'test', 'all', optional Select the dataset to load: 'train' for the training set, 'test' for the test set, 'all' for both, with shuffled ordering. categories : None or collection of string or unicode If None (default), load all the categories. If not None, list of category names to load (other categories ignored). shuffle : bool, optional Whether or not to shuffle the data: might be important for models that make the assumption that the samples are independent and identically distributed (i.i.d.), such as stochastic gradient descent. random_state : int, RandomState instance or None (default) Determines random number generation for dataset shuffling. Pass an int for reproducible output across multiple function calls. See :term:`Glossary `. remove : tuple May contain any subset of ('headers', 'footers', 'quotes'). Each of these are kinds of text that will be detected and removed from the newsgroup posts, preventing classifiers from overfitting on metadata. 'headers' removes newsgroup headers, 'footers' removes blocks at the ends of posts that look like signatures, and 'quotes' removes lines that appear to be quoting another post. 'headers' follows an exact standard; the other filters are not always correct. download_if_missing : optional, True by default If False, raise an IOError if the data is not locally available instead of trying to download the data from the source site. Returns ------- bunch : Bunch object with the following attribute: - bunch.data: list, length [n_samples] - bunch.target: array, shape [n_samples] - bunch.filenames: list, length [n_samples] - bunch.DESCR: a description of the dataset. - bunch.target_names: a list of categories of the returned data, length [n_classes]. This depends on the `categories` parameter. t data_homet 20news_hometrbNRiPt_sCache loading faileds8Downloading 20news dataset. This may take a few minutes.RRs20Newsgroups dataset not foundRRtalls5subset can only be 'train', 'test' or 'all', got '%s'tdescrstwenty_newsgroups.rsttheaderstfooterstquotestdtypei(RR(RR(-RRt CACHE_NAMERRRtNoneRR&treadR*tdecodeR,tloadst ExceptionR!R"RtIOErrortlisttextendtdatattargett filenamestnptarrayt ValueErrorRt__file__tDESCRR<RJRBt target_namestindextsorttziptin1dt searchsortedtobjectttolistR tarangetshapetshuffle(RKtsubsett categoriesRpt random_stateR(tdownload_if_missingRt twenty_homeR4R6R5tuncompressed_contentteR^tdata_lstR_R`t module_pathtrst_filetfdescrR8tcattlabelstmasktindices((sAlib/python2.7/site-packages/sklearn/datasets/twenty_newsgroups.pytfetch_20newsgroupssŒG                     ( ( ( +     cCs|td|ƒ}d}|r5|ddj|ƒ7}nt||dƒ}td|ddddd td d d |d |ƒ}td|ddddd td d d |d |ƒ}tjj|ƒrØt j |ƒ\} } n^t dt j ƒ} | j|jƒjƒ} | j|jƒjƒ} t j| | f|ddƒ| jt jƒ} | jt jƒ} t| dtƒt| dtƒ|j} |dkr¡| } |j}ns|dkr¿| } |j}nU|dkrtj| | fƒjƒ} t j|j|jfƒ}ntd|ƒ‚ttƒ}tt|ddƒƒ}|j ƒ}WdQX|r]| |fSt!d| d|d| d|ƒS(s Load the 20 newsgroups dataset and vectorize it into token counts (classification). Download it if necessary. This is a convenience function; the transformation is done using the default settings for :class:`sklearn.feature_extraction.text.CountVectorizer`. For more advanced usage (stopword filtering, n-gram extraction, etc.), combine fetch_20newsgroups with a custom :class:`sklearn.feature_extraction.text.CountVectorizer`, :class:`sklearn.feature_extraction.text.HashingVectorizer`, :class:`sklearn.feature_extraction.text.TfidfTransformer` or :class:`sklearn.feature_extraction.text.TfidfVectorizer`. ================= ========== Classes 20 Samples total 18846 Dimensionality 130107 Features real ================= ========== Read more in the :ref:`User Guide <20newsgroups_dataset>`. Parameters ---------- subset : 'train' or 'test', 'all', optional Select the dataset to load: 'train' for the training set, 'test' for the test set, 'all' for both, with shuffled ordering. remove : tuple May contain any subset of ('headers', 'footers', 'quotes'). Each of these are kinds of text that will be detected and removed from the newsgroup posts, preventing classifiers from overfitting on metadata. 'headers' removes newsgroup headers, 'footers' removes blocks at the ends of posts that look like signatures, and 'quotes' removes lines that appear to be quoting another post. data_home : optional, default: None Specify an download and cache folder for the datasets. If None, all scikit-learn data is stored in '~/scikit_learn_data' subfolders. download_if_missing : optional, True by default If False, raise an IOError if the data is not locally available instead of trying to download the data from the source site. return_X_y : boolean, default=False. If True, returns ``(data.data, data.target)`` instead of a Bunch object. .. versionadded:: 0.20 Returns ------- bunch : Bunch object with the following attribute: - bunch.data: sparse matrix, shape [n_samples, n_features] - bunch.target: array, shape [n_samples] - bunch.target_names: a list of categories of the returned data, length [n_classes]. - bunch.DESCR: a description of the dataset. (data, target) : tuple if ``return_X_y`` is True .. versionadded:: 0.20 RKt20newsgroup_vectorizedsremove-RCs.pklRqRRrRpRsi R(RtRRTtcompressi tcopyROsC%r is not a valid subset: should be one of ['train', 'test', 'all']RPstwenty_newsgroups.rstNR^R_RfRe("RRRR€RVtTrueRRRR tloadRRatint16t fit_transformR^ttocsrt transformtdumptastypetfloat64RtFalseRfR_tsptvstackt concatenateRcRRdR&RWR (RqR(RKRtt return_X_ytfilebaset target_filet data_traint data_testtX_traintX_testt vectorizerRfR^R_RyRzR{((sAlib/python2.7/site-packages/sklearn/datasets/twenty_newsgroups.pytfetch_20newsgroups_vectorized<sdE              (((2t__doc__Rtos.pathRRtloggingR%R,R/treR*tnumpyRat scipy.sparsetsparseRŽtbaseRRRRRtfeature_extraction.textRt preprocessingRtutilsR R R R t getLoggert__name__R!R#RURRRRR<tcompileR>RBRJRVR„R€RR™(((sAlib/python2.7/site-packages/sklearn/datasets/twenty_newsgroups.pytsP               œ