ó ‡ˆ\c @ sÑdZddlmZddlmZddlZddlZddlZddlmZddl Z ddl j Z ddlmZmZddlmZmZmZdd lmZdd lmZmZdd lmZdd lmZmZdd lmZddl m!Z!ddl"m#Z#m$Z$ddl%m&Z&ddl'm(Z(ddl)m*Z*ddddddgZ+ddddddddde-dd„ Z.dddddddddd„ Z/e-e-e-e-e-dd „Z0e-d!„Z1d"„Z2dddddddd#d$„Z3d%„Z4d&„Z5d'„Z6ddd(ddddd)„Z7d*„Z8d+„Z9de j:d,d-d.ƒdde-dd/de-ddd0„ Z;d1„Z<d2„Z=ddddd/ddd3„Z>d4„Z?dS(5sm The :mod:`sklearn.model_selection._validation` module includes classes and functions to validate the model. iÿÿÿÿ(tprint_function(tdivisionN(tformat_exception_onlyi(t is_classifiertclone(t indexabletcheck_random_statet safe_indexing(tDeprecationDict(t _is_arrayliket _num_samples(t _safe_split(tParalleltdelayed(tlogger(tzip(t check_scoringt_check_multimetric_scoring(tFitFailedWarningi(tcheck_cv(t LabelEncodertcross_validatetcross_val_scoretcross_val_predicttpermutation_test_scoretlearning_curvetvalidation_curvetwarnis2*n_jobssraise-deprecatingc   sñtˆˆ|ƒ\‰‰}t|ˆdtˆƒƒ}tˆd|ƒ\‰} td|dˆd| ƒ}|‡‡‡‡‡‡‡‡‡f d†|jˆˆ|ƒDƒƒ}tt|Œƒ}ˆrâ|jdƒ}t |ƒ}nˆr÷|jƒ}n|\}}}t |ƒ}ˆdkr't ƒni}t j |ƒ|d `. Parameters ---------- estimator : estimator object implementing 'fit' The object to use to fit the data. X : array-like The data to fit. Can be for example a list, or an array. y : array-like, optional, default: None The target variable to try to predict in the case of supervised learning. groups : array-like, with shape (n_samples,), optional Group labels for the samples used while splitting the dataset into train/test set. scoring : string, callable, list/tuple, dict or None, default: None A single string (see :ref:`scoring_parameter`) or a callable (see :ref:`scoring`) to evaluate the predictions on the test set. For evaluating multiple metrics, either give a list of (unique) strings or a dict with names as keys and callables as values. NOTE that when using custom scorers, each scorer should return a single value. Metric functions returning a list/array of values can be wrapped into multiple scorers that return one value each. See :ref:`multimetric_grid_search` for an example. If None, the estimator's default scorer (if available) is used. cv : int, cross-validation generator or an iterable, optional Determines the cross-validation splitting strategy. Possible inputs for cv are: - None, to use the default 3-fold cross validation, - integer, to specify the number of folds in a `(Stratified)KFold`, - :term:`CV splitter`, - An iterable yielding (train, test) splits as arrays of indices. For integer/None inputs, if the estimator is a classifier and ``y`` is either binary or multiclass, :class:`StratifiedKFold` is used. In all other cases, :class:`KFold` is used. Refer :ref:`User Guide ` for the various cross-validation strategies that can be used here. .. versionchanged:: 0.20 ``cv`` default value if None will change from 3-fold to 5-fold in v0.22. n_jobs : int or None, optional (default=None) The number of CPUs to use to do the computation. ``None`` means 1 unless in a :obj:`joblib.parallel_backend` context. ``-1`` means using all processors. See :term:`Glossary ` for more details. verbose : integer, optional The verbosity level. fit_params : dict, optional Parameters to pass to the fit method of the estimator. pre_dispatch : int, or string, optional Controls the number of jobs that get dispatched during parallel execution. Reducing this number can be useful to avoid an explosion of memory consumption when more jobs get dispatched than CPUs can process. This parameter can be: - None, in which case all the jobs are immediately created and spawned. Use this for lightweight and fast-running jobs, to avoid delays due to on-demand spawning of the jobs - An int, giving the exact number of total jobs that are spawned - A string, giving an expression as a function of n_jobs, as in '2*n_jobs' return_train_score : boolean, optional Whether to include train scores. Current default is ``'warn'``, which behaves as ``True`` in addition to raising a warning when a training score is looked up. That default will be changed to ``False`` in 0.21. Computing training scores is used to get insights on how different parameter settings impact the overfitting/underfitting trade-off. However computing the scores on the training set can be computationally expensive and is not strictly required to select the parameters that yield the best generalization performance. return_estimator : boolean, default False Whether to return the estimators fitted on each split. error_score : 'raise' | 'raise-deprecating' or numeric Value to assign to the score if an error occurs in estimator fitting. If set to 'raise', the error is raised. If set to 'raise-deprecating', a FutureWarning is printed before the error is raised. If a numeric value is given, FitFailedWarning is raised. This parameter does not affect the refit step, which will always raise the error. Default is 'raise-deprecating' but from version 0.22 it will change to np.nan. Returns ------- scores : dict of float arrays of shape=(n_splits,) Array of scores of the estimator for each run of the cross validation. A dict of arrays containing the score/time arrays for each scorer is returned. The possible keys for this ``dict`` are: ``test_score`` The score array for test scores on each cv split. ``train_score`` The score array for train scores on each cv split. This is available only if ``return_train_score`` parameter is ``True``. ``fit_time`` The time for fitting the estimator on the train set for each cv split. ``score_time`` The time for scoring the estimator on the test set for each cv split. (Note time for scoring on the train set is not included even if ``return_train_score`` is set to ``True`` ``estimator`` The estimator objects for each cv split. This is available only if ``return_estimator`` parameter is set to ``True``. Examples -------- >>> from sklearn import datasets, linear_model >>> from sklearn.model_selection import cross_validate >>> from sklearn.metrics.scorer import make_scorer >>> from sklearn.metrics import confusion_matrix >>> from sklearn.svm import LinearSVC >>> diabetes = datasets.load_diabetes() >>> X = diabetes.data[:150] >>> y = diabetes.target[:150] >>> lasso = linear_model.Lasso() Single metric evaluation using ``cross_validate`` >>> cv_results = cross_validate(lasso, X, y, cv=3, ... return_train_score=False) >>> sorted(cv_results.keys()) # doctest: +ELLIPSIS ['fit_time', 'score_time', 'test_score'] >>> cv_results['test_score'] # doctest: +ELLIPSIS +NORMALIZE_WHITESPACE array([0.33150734, 0.08022311, 0.03531764]) Multiple metric evaluation using ``cross_validate`` (please refer the ``scoring`` parameter doc for more information) >>> scores = cross_validate(lasso, X, y, cv=3, ... scoring=('r2', 'neg_mean_squared_error'), ... return_train_score=True) >>> print(scores['test_neg_mean_squared_error']) # doctest: +ELLIPSIS [-3635.5... -3573.3... -6114.7...] >>> print(scores['train_r2']) # doctest: +ELLIPSIS [0.28010158 0.39088426 0.22784852] See Also --------- :func:`sklearn.model_selection.cross_val_score`: Run cross-validation for single metric evaluation. :func:`sklearn.model_selection.cross_val_predict`: Get predictions from each split of cross-validation for diagnostic purposes. :func:`sklearn.metrics.make_scorer`: Make a scorer from a performance metric or loss function. t classifiertscoringtn_jobstverboset pre_dispatchc3 s]|]S\}}ttƒtˆƒˆˆˆ||ˆdˆdˆdtdˆdˆƒ VqdS(treturn_train_scoret return_timestreturn_estimatort error_scoreN(R t_fit_and_scoreRtNonetTrue(t.0ttrainttest( tXR$t estimatort fit_paramsR#R!tscorersRty(sBlib/python2.7/site-packages/sklearn/model_selection/_validation.pys ësiRtfit_timet score_timeR,stest_%sstrain_%ss£You are accessing a training score ({!r}), which will not be available by default any more in 0.21. If you need training scores, please set return_train_score=True(RRRRR tsplittlistRtpopt_aggregate_score_dictsRtnptarraytformatt add_warningt FutureWarning(R,R+R/tgroupsRtcvRRR-R R!R#R$t_tparalleltscorest zipped_scorest train_scorestfitted_estimatorst test_scorest fit_timest score_timestrettnametkeytmessage(( R+R$R,R-R#R!R.RR/sBlib/python2.7/site-packages/sklearn/model_selection/_validation.pyR)s>¸ $      c C srt|d|ƒ} td|d|d|d|di| d6d|dtd |d |d |d | d | ƒ } | dS(s:Evaluate a score by cross-validation Read more in the :ref:`User Guide `. Parameters ---------- estimator : estimator object implementing 'fit' The object to use to fit the data. X : array-like The data to fit. Can be for example a list, or an array. y : array-like, optional, default: None The target variable to try to predict in the case of supervised learning. groups : array-like, with shape (n_samples,), optional Group labels for the samples used while splitting the dataset into train/test set. scoring : string, callable or None, optional, default: None A string (see model evaluation documentation) or a scorer callable object / function with signature ``scorer(estimator, X, y)``. cv : int, cross-validation generator or an iterable, optional Determines the cross-validation splitting strategy. Possible inputs for cv are: - None, to use the default 3-fold cross validation, - integer, to specify the number of folds in a `(Stratified)KFold`, - :term:`CV splitter`, - An iterable yielding (train, test) splits as arrays of indices. For integer/None inputs, if the estimator is a classifier and ``y`` is either binary or multiclass, :class:`StratifiedKFold` is used. In all other cases, :class:`KFold` is used. Refer :ref:`User Guide ` for the various cross-validation strategies that can be used here. .. versionchanged:: 0.20 ``cv`` default value if None will change from 3-fold to 5-fold in v0.22. n_jobs : int or None, optional (default=None) The number of CPUs to use to do the computation. ``None`` means 1 unless in a :obj:`joblib.parallel_backend` context. ``-1`` means using all processors. See :term:`Glossary ` for more details. verbose : integer, optional The verbosity level. fit_params : dict, optional Parameters to pass to the fit method of the estimator. pre_dispatch : int, or string, optional Controls the number of jobs that get dispatched during parallel execution. Reducing this number can be useful to avoid an explosion of memory consumption when more jobs get dispatched than CPUs can process. This parameter can be: - None, in which case all the jobs are immediately created and spawned. Use this for lightweight and fast-running jobs, to avoid delays due to on-demand spawning of the jobs - An int, giving the exact number of total jobs that are spawned - A string, giving an expression as a function of n_jobs, as in '2*n_jobs' error_score : 'raise' | 'raise-deprecating' or numeric Value to assign to the score if an error occurs in estimator fitting. If set to 'raise', the error is raised. If set to 'raise-deprecating', a FutureWarning is printed before the error is raised. If a numeric value is given, FitFailedWarning is raised. This parameter does not affect the refit step, which will always raise the error. Default is 'raise-deprecating' but from version 0.22 it will change to np.nan. Returns ------- scores : array of float, shape=(len(list(cv)),) Array of scores of the estimator for each run of the cross validation. Examples -------- >>> from sklearn import datasets, linear_model >>> from sklearn.model_selection import cross_val_score >>> diabetes = datasets.load_diabetes() >>> X = diabetes.data[:150] >>> y = diabetes.target[:150] >>> lasso = linear_model.Lasso() >>> print(cross_val_score(lasso, X, y, cv=3)) # doctest: +ELLIPSIS [0.33150734 0.08022311 0.03531764] See Also --------- :func:`sklearn.model_selection.cross_validate`: To run cross-validation on multiple metrics and also to return train scores, fit times and score times. :func:`sklearn.model_selection.cross_val_predict`: Get predictions from each split of cross-validation for diagnostic purposes. :func:`sklearn.metrics.make_scorer`: Make a scorer from a performance metric or loss function. RR,R+R/R;tscoreR<R!RRR-R R$t test_score(RRtFalse( R,R+R/R;RR<RRR-R R$tscorert cv_results((sBlib/python2.7/site-packages/sklearn/model_selection/_validation.pyRsv  c#C s*|dkri|dkr!d}n#ddjd„|jƒDƒƒ}td|dt|ƒdfƒn|dk r{|ni}tg|jƒD]$\}}|t|||ƒf^q‘ƒ}i}|dk rà|j|ntjƒ}t ||||ƒ\}}t |||||ƒ\}}t |ƒ }|rJt|j ƒƒnd}y6|dkrr|j ||n|j |||Wnt k rž}tjƒ|}d }|d krÀ‚qü|d krâtjd tƒ‚qüt|tjƒr|rJtt|j ƒ|g|ƒƒ}| r_tt|j ƒ|g|ƒƒ}q_n|}| r_|}ntjd |tt|ƒ|ƒdftƒqütdƒ‚n^Xtjƒ|}t|||||ƒ}tjƒ||}| rüt|||||ƒ}n|dkrS|rBx?|jƒD] \}}|d||f7}qWqS|d|7}n|dkr§||} d|tj| ƒf}!tddt|!ƒd|!fƒn| r¹||gn|g}"| rÞ|"jt|ƒƒn| rú|"j||gƒn| r|"j|ƒn| r&|"j|ƒn|"S(sg Fit estimator and compute scores for a given dataset split. Parameters ---------- estimator : estimator object implementing 'fit' The object to use to fit the data. X : array-like of shape at least 2D The data to fit. y : array-like, optional, default: None The target variable to try to predict in the case of supervised learning. scorer : A single callable or dict mapping scorer name to the callable If it is a single callable, the return value for ``train_scores`` and ``test_scores`` is a single float. For a dict, it should be one mapping the scorer name to the scorer callable object / function. The callable object / fn should have signature ``scorer(estimator, X, y)``. train : array-like, shape (n_train_samples,) Indices of training samples. test : array-like, shape (n_test_samples,) Indices of test samples. verbose : integer The verbosity level. error_score : 'raise' | 'raise-deprecating' or numeric Value to assign to the score if an error occurs in estimator fitting. If set to 'raise', the error is raised. If set to 'raise-deprecating', a FutureWarning is printed before the error is raised. If a numeric value is given, FitFailedWarning is raised. This parameter does not affect the refit step, which will always raise the error. Default is 'raise-deprecating' but from version 0.22 it will change to np.nan. parameters : dict or None Parameters to be set on the estimator. fit_params : dict or None Parameters that will be passed to ``estimator.fit``. return_train_score : boolean, optional, default: False Compute and return score on training set. return_parameters : boolean, optional, default: False Return parameters that has been used for the estimator. return_n_test_samples : boolean, optional, default: False Whether to return the ``n_test_samples`` return_times : boolean, optional, default: False Whether to return the fit/score times. return_estimator : boolean, optional, default: False Whether to return the fitted estimator. Returns ------- train_scores : dict of scorer name -> float, optional Score on training set (for all the scorers), returned only if `return_train_score` is `True`. test_scores : dict of scorer name -> float, optional Score on testing set (for all the scorers). n_test_samples : int Number of test samples. fit_time : float Time spent for fitting in seconds. score_time : float Time spent for scoring in seconds. parameters : dict or None, optional The parameters that have been evaluated. estimator : estimator object The fitted estimator its%ss, cs s%|]\}}d||fVqdS(s%s=%sN((R(tktv((sBlib/python2.7/site-packages/sklearn/model_selection/_validation.pys ÷ss [CV] %s %si@t.gtraisesraise-deprecatingsÖFrom version 0.22, errors during fit will result in a cross validation score of NaN by default. Use error_score='raise' if you want an exception raised or error_score=np.nan to adopt the behavior from version 0.22.sqEstimator fit failed. The score on this train-test partition for these parameters will be set to %f. Details: %sis‰error_score must be the string 'raise' or a numeric value. (Hint: if using 'raise', please make sure that it has been spelled correctly.)is, %s=%ss , score=%ss %s, total=%sN(R&tjointitemstprinttlentdictt_index_param_valuet set_paramsttimeR tcallabletkeystfitt ExceptiontwarningsRR:t isinstancetnumberstNumberRRttypeRt ValueErrort_scoreRtshort_format_timetappendR textend(#R,R+R/RMR)R*Rt parametersR-R!treturn_parameterstreturn_n_test_samplesR"R#R$tmsgRPRQRAt start_timetX_trainty_traintX_testty_testtis_multimetrict n_scorersteR0R1RCt scorer_nameRJt total_timetend_msgRF((sBlib/python2.7/site-packages/sklearn/model_selection/_validation.pyR%–sŠ]   %7              %cC s½|rt||||ƒS|dkr7|||ƒ}n||||ƒ}t|dƒry|jƒ}Wqtk r{qXnt|tjƒs¹tdt|ƒt |ƒ|fƒ‚n|S(s­Compute the score(s) of an estimator on a given test set. Will return a single float if is_multimetric is False and a dict of floats, if is_multimetric is True titems>scoring must return a number, got %s (%s) instead. (scorer=%r)N( t_multimetric_scoreR&thasattrRyReRaRbRctstrRd(R,RqRrRMRsRJ((sBlib/python2.7/site-packages/sklearn/model_selection/_validation.pyRfVs  "cC sÑi}xÄ|jƒD]¶\}}|dkr=|||ƒ}n||||ƒ}t|dƒr…y|jƒ}Wq…tk rq…Xn|||scoring must return a number, got %s (%s) instead. (scorer=%s)N( RUR&R{RyReRaRbRcR|Rd(R,RqRrR.R?RGRMRJ((sBlib/python2.7/site-packages/sklearn/model_selection/_validation.pyRzss   &tpredictc  sštˆˆ|ƒ\‰‰}t|ˆdtˆƒƒ}ˆd kr]tƒ} | jˆƒ‰ntd|dˆd|ƒ} | ‡‡‡‡‡‡fd†|jˆˆ|ƒDƒƒ} g| D]\} }| ^q¹}tjg| D]\}}|^qÞƒ}t |t ˆƒƒst d ƒ‚ntj t |ƒd tƒ}tjt |ƒƒ||`. Parameters ---------- estimator : estimator object implementing 'fit' and 'predict' The object to use to fit the data. X : array-like The data to fit. Can be, for example a list, or an array at least 2d. y : array-like, optional, default: None The target variable to try to predict in the case of supervised learning. groups : array-like, with shape (n_samples,), optional Group labels for the samples used while splitting the dataset into train/test set. cv : int, cross-validation generator or an iterable, optional Determines the cross-validation splitting strategy. Possible inputs for cv are: - None, to use the default 3-fold cross validation, - integer, to specify the number of folds in a `(Stratified)KFold`, - :term:`CV splitter`, - An iterable yielding (train, test) splits as arrays of indices. For integer/None inputs, if the estimator is a classifier and ``y`` is either binary or multiclass, :class:`StratifiedKFold` is used. In all other cases, :class:`KFold` is used. Refer :ref:`User Guide ` for the various cross-validation strategies that can be used here. .. versionchanged:: 0.20 ``cv`` default value if None will change from 3-fold to 5-fold in v0.22. n_jobs : int or None, optional (default=None) The number of CPUs to use to do the computation. ``None`` means 1 unless in a :obj:`joblib.parallel_backend` context. ``-1`` means using all processors. See :term:`Glossary ` for more details. verbose : integer, optional The verbosity level. fit_params : dict, optional Parameters to pass to the fit method of the estimator. pre_dispatch : int, or string, optional Controls the number of jobs that get dispatched during parallel execution. Reducing this number can be useful to avoid an explosion of memory consumption when more jobs get dispatched than CPUs can process. This parameter can be: - None, in which case all the jobs are immediately created and spawned. Use this for lightweight and fast-running jobs, to avoid delays due to on-demand spawning of the jobs - An int, giving the exact number of total jobs that are spawned - A string, giving an expression as a function of n_jobs, as in '2*n_jobs' method : string, optional, default: 'predict' Invokes the passed method name of the passed estimator. For method='predict_proba', the columns correspond to the classes in sorted order. Returns ------- predictions : ndarray This is the result of calling ``method`` See also -------- cross_val_score : calculate score for each CV split cross_validate : calculate one or more scores and timings for each CV split Notes ----- In the case that one or more classes are absent in a training portion, a default score needs to be assigned to all instances for that class if ``method`` produces columns per class, as in {'decision_function', 'predict_proba', 'predict_log_proba'}. For ``predict_proba`` this value is 0. In order to ensure finite output, we approximate negative infinity by the minimum finite float value for the dtype in other cases. Examples -------- >>> from sklearn import datasets, linear_model >>> from sklearn.model_selection import cross_val_predict >>> diabetes = datasets.load_diabetes() >>> X = diabetes.data[:150] >>> y = diabetes.target[:150] >>> lasso = linear_model.Lasso() >>> y_pred = cross_val_predict(lasso, X, y, cv=3) Rtdecision_functiont predict_probatpredict_log_probaRRR c 3 sB|]8\}}ttƒtˆƒˆˆ||ˆˆˆƒVqdS(N(R t_fit_and_predictR(R(R)R*(R+R,R-tmethodRR/(sBlib/python2.7/site-packages/sklearn/model_selection/_validation.pys ss+cross_val_predict only works for partitionstdtypeiR8(R~RR€(RRRRt fit_transformR R2R6t concatenatet_check_is_permutationR RetemptyRWtinttarangetsptissparsetvstackR8(R,R+R/R;R<RRR-R R‚tleR>tprediction_blockst pred_block_iR=t predictionst indices_it test_indicestinv_test_indices((R+R,R-R‚RR/sBlib/python2.7/site-packages/sklearn/model_selection/_validation.pyRs(n    cC s]|d k r|ni}tg|jƒD]$\}} |t|| |ƒf^q(ƒ}t||||ƒ\} } t|||||ƒ\} } | d kr­|j| |n|j| | |t||ƒ}|| ƒ}|d krStt|ƒƒ}|t|j ƒkrSd}t j dj t|j ƒ||ƒt ƒ|dkrÜ|jdkr |jdt|j ƒkr tdj |j|t|j ƒ|ƒƒ‚nt|j ƒdkrÜtd j t|j ƒ|ƒƒ‚qÜntj|jƒj}i|d6|d6d d6}tjt|ƒ|f||ƒ}||d d …|j f<|}qSn||fS( sFit estimator and predict values for a given dataset split. Read more in the :ref:`User Guide `. Parameters ---------- estimator : estimator object implementing 'fit' and 'predict' The object to use to fit the data. X : array-like of shape at least 2D The data to fit. y : array-like, optional, default: None The target variable to try to predict in the case of supervised learning. train : array-like, shape (n_train_samples,) Indices of training samples. test : array-like, shape (n_test_samples,) Indices of test samples. verbose : integer The verbosity level. fit_params : dict or None Parameters that will be passed to ``estimator.fit``. method : string Invokes the passed method name of the passed estimator. Returns ------- predictions : sequence Result of calling 'estimator.method' test : array-like This is the value of the test parameter R~RR€sTTo fix this, use a cross-validation technique resulting in properly stratified foldss‰Number of classes in training fold ({}) does not match total number of classes ({}). Results may not be appropriate for your use case. {}iis™Output shape {} of {} does not match number of classes ({}) in fold. Irregular decision_function outputs are not currently supported by cross_val_predictshOnly {} class/es in training fold, this is not supported for decision_function with imbalanced folds. {}iN(R~RR€(R&RXRURYR R^tgetattrRWtsettclasses_R`RR8tRuntimeWarningtndimtshapeReR6tfinfoRƒtmintfullR (R,R+R/R)R*RR-R‚RPRQRoRpRqR=tfuncRt n_classestrecommendationt float_mintdefault_valuestpredictions_for_all_classes((sBlib/python2.7/site-packages/sklearn/model_selection/_validation.pyRsN*7                cC sLt|ƒ|krtStj|dtƒ}t||`. Parameters ---------- estimator : estimator object implementing 'fit' The object to use to fit the data. X : array-like of shape at least 2D The data to fit. y : array-like The target variable to try to predict in the case of supervised learning. groups : array-like, with shape (n_samples,), optional Labels to constrain permutation within groups, i.e. ``y`` values are permuted among samples with the same group identifier. When not specified, ``y`` values are permuted among all samples. When a grouped cross-validator is used, the group labels are also passed on to the ``split`` method of the cross-validator. The cross-validator uses them for grouping the samples while splitting the dataset into train/test set. scoring : string, callable or None, optional, default: None A single string (see :ref:`scoring_parameter`) or a callable (see :ref:`scoring`) to evaluate the predictions on the test set. If None the estimator's default scorer, if available, is used. cv : int, cross-validation generator or an iterable, optional Determines the cross-validation splitting strategy. Possible inputs for cv are: - None, to use the default 3-fold cross validation, - integer, to specify the number of folds in a `(Stratified)KFold`, - :term:`CV splitter`, - An iterable yielding (train, test) splits as arrays of indices. For integer/None inputs, if the estimator is a classifier and ``y`` is either binary or multiclass, :class:`StratifiedKFold` is used. In all other cases, :class:`KFold` is used. Refer :ref:`User Guide ` for the various cross-validation strategies that can be used here. .. versionchanged:: 0.20 ``cv`` default value if None will change from 3-fold to 5-fold in v0.22. n_permutations : integer, optional Number of times to permute ``y``. n_jobs : int or None, optional (default=None) The number of CPUs to use to do the computation. ``None`` means 1 unless in a :obj:`joblib.parallel_backend` context. ``-1`` means using all processors. See :term:`Glossary ` for more details. random_state : int, RandomState instance or None, optional (default=0) If int, random_state is the seed used by the random number generator; If RandomState instance, random_state is the random number generator; If None, the random number generator is the RandomState instance used by `np.random`. verbose : integer, optional The verbosity level. Returns ------- score : float The true score without permuting targets. permutation_scores : array, shape (n_permutations,) The scores obtained for each permutations. pvalue : float The p-value, which approximates the probability that the score would be obtained by chance. This is calculated as: `(C + 1) / (n_permutations + 1)` Where C is the number of permutations whose score >= the true score. The best possible p-value is 1/(n_permutations + 1), the worst is 1.0. Notes ----- This function implements Test 1 in: Ojala and Garriga. Permutation Tests for Studying Classifier Performance. The Journal of Machine Learning Research (2010) vol. 11 RRRRc3 sB|]8}ttƒtˆƒˆtˆˆˆƒˆˆˆƒVqdS(N(R t_permutation_test_scoreRt_shuffle(R(R=(R+R<R,R;t random_stateRMR/(sBlib/python2.7/site-packages/sklearn/model_selection/_validation.pys sgð?i( RRRRRRªRR trangeR6R7tsum( R,R+R/R;R<tn_permutationsRR¬RRRJtpermutation_scorestpvalue((R+R<R,R;R¬RMR/sBlib/python2.7/site-packages/sklearn/model_selection/_validation.pyR§sd !!c C s›g}x…|j|||ƒD]n\}}t||||ƒ\} } t|||||ƒ\} } |j| | ƒ|j||| | ƒƒqWtj|ƒS(s-Auxiliary function for permutation_test_score(R2R R^RhR6tmean( R,R+R/R;R<RMt avg_scoreR)R*RoRpRqRr((sBlib/python2.7/site-packages/sklearn/model_selection/_validation.pyRªs"cC sƒ|dkr$|jt|ƒƒ}nRtjt|ƒƒ}x:tj|ƒD])}||k}|j||ƒ||`. Parameters ---------- estimator : object type that implements the "fit" and "predict" methods An object of that type which is cloned for each validation. X : array-like, shape (n_samples, n_features) Training vector, where n_samples is the number of samples and n_features is the number of features. y : array-like, shape (n_samples) or (n_samples, n_features), optional Target relative to X for classification or regression; None for unsupervised learning. groups : array-like, with shape (n_samples,), optional Group labels for the samples used while splitting the dataset into train/test set. train_sizes : array-like, shape (n_ticks,), dtype float or int Relative or absolute numbers of training examples that will be used to generate the learning curve. If the dtype is float, it is regarded as a fraction of the maximum size of the training set (that is determined by the selected validation method), i.e. it has to be within (0, 1]. Otherwise it is interpreted as absolute sizes of the training sets. Note that for classification the number of samples usually have to be big enough to contain at least one sample from each class. (default: np.linspace(0.1, 1.0, 5)) cv : int, cross-validation generator or an iterable, optional Determines the cross-validation splitting strategy. Possible inputs for cv are: - None, to use the default 3-fold cross validation, - integer, to specify the number of folds in a `(Stratified)KFold`, - :term:`CV splitter`, - An iterable yielding (train, test) splits as arrays of indices. For integer/None inputs, if the estimator is a classifier and ``y`` is either binary or multiclass, :class:`StratifiedKFold` is used. In all other cases, :class:`KFold` is used. Refer :ref:`User Guide ` for the various cross-validation strategies that can be used here. .. versionchanged:: 0.20 ``cv`` default value if None will change from 3-fold to 5-fold in v0.22. scoring : string, callable or None, optional, default: None A string (see model evaluation documentation) or a scorer callable object / function with signature ``scorer(estimator, X, y)``. exploit_incremental_learning : boolean, optional, default: False If the estimator supports incremental learning, this will be used to speed up fitting for different training set sizes. n_jobs : int or None, optional (default=None) Number of jobs to run in parallel. ``None`` means 1 unless in a :obj:`joblib.parallel_backend` context. ``-1`` means using all processors. See :term:`Glossary ` for more details. pre_dispatch : integer or string, optional Number of predispatched jobs for parallel execution (default is all). The option can reduce the allocated memory. The string can be an expression like '2*n_jobs'. verbose : integer, optional Controls the verbosity: the higher, the more messages. shuffle : boolean, optional Whether to shuffle training data before taking prefixes of it based on``train_sizes``. random_state : int, RandomState instance or None, optional (default=None) If int, random_state is the seed used by the random number generator; If RandomState instance, random_state is the random number generator; If None, the random number generator is the RandomState instance used by `np.random`. Used when ``shuffle`` is True. error_score : 'raise' | 'raise-deprecating' or numeric Value to assign to the score if an error occurs in estimator fitting. If set to 'raise', the error is raised. If set to 'raise-deprecating', a FutureWarning is printed before the error is raised. If a numeric value is given, FitFailedWarning is raised. This parameter does not affect the refit step, which will always raise the error. Default is 'raise-deprecating' but from version 0.22 it will change to np.nan. Returns ------- train_sizes_abs : array, shape (n_unique_ticks,), dtype int Numbers of training examples that has been used to generate the learning curve. Note that the number of ticks might be less than n_ticks because duplicate entries will be removed. train_scores : array, shape (n_ticks, n_cv_folds) Scores on training sets. test_scores : array, shape (n_ticks, n_cv_folds) Scores on test set. Notes ----- See :ref:`examples/model_selection/plot_learning_curve.py ` t partial_fitsSAn estimator must support the partial_fit interface to exploit incremental learningRRis%[learning_curve] Training set sizes: RR Rc3 s*|] \}}ˆj|ƒ|fVqdS(N(R´(R(R)R*(trng(sBlib/python2.7/site-packages/sklearn/model_selection/_validation.pys Ísc 3 sE|];\}}ttƒtˆƒˆˆˆ||ˆˆˆƒ VqdS(N(R t_incremental_fit_estimatorR(R(R)R*(R+tclassesR,RMttrain_sizes_absRR/(sBlib/python2.7/site-packages/sklearn/model_selection/_validation.pys Ñsc3 sW|]M\}}ttƒtˆƒˆˆˆ||ˆdddddtdˆƒVqdS(RjR-R!R$N(R R%RR&R'(R(R)R*(R+R$R,RMRR/(sBlib/python2.7/site-packages/sklearn/model_selection/_validation.pys ÚsiiN(iii(R{ReRRRR3R2RRWt_translate_train_sizesR™RVR|R RR6RµR&RhR7treshapetasarrayt transpose(R,R+R/R;t train_sizesR<Rtexploit_incremental_learningRR RtshuffleR¬R$tcv_itertn_max_training_samplestn_unique_ticksR>toutttrain_test_proportionsR)R*tn_train_samplest n_cv_folds(( R+R»R$R,R¹RMR¼RR/sBlib/python2.7/site-packages/sklearn/model_selection/_validation.pyR5s@~     !!  cC s8tj|ƒ}|jd}tj|ƒ}tj|ƒ}tj|jtjƒrº|dksj|dkrƒtd||fƒ‚n||j dtj dt ƒ}tj |d|ƒ}n4|dksÒ||krîtd|||fƒ‚ntj |ƒ}||jdkr4tjd |jd|ftƒn|S( s™Determine absolute sizes of training subsets and validate 'train_sizes'. Examples: _translate_train_sizes([0.5, 1.0], 10) -> [5, 10] _translate_train_sizes([5, 10], 10) -> [5, 10] Parameters ---------- train_sizes : array-like, shape (n_ticks,), dtype float or int Numbers of training examples that will be used to generate the learning curve. If the dtype is float, it is regarded as a fraction of 'n_max_training_samples', i.e. it has to be within (0, 1]. n_max_training_samples : int Maximum number of training samples (upper bound of 'train_sizes'). Returns ------- train_sizes_abs : array, shape (n_unique_ticks,), dtype int Numbers of training examples that will be used to generate the learning curve. Note that the number of ticks might be less than n_ticks because duplicate entries will be removed. iggð?sŠtrain_sizes has been interpreted as fractions of the maximum number of training samples and must be within (0, 1], but is within [%f, %f].Rƒtcopyis|train_sizes has been interpreted as absolute numbers of training samples and must be within (0, %d], but is within [%d, %d].s|Removed duplicate entries from 'train_sizes'. Number of ticks will be less than the size of 'train_sizes' %d instead of %d).(R6R¿R™R›tmaxt issubdtypeRƒtfloatingRetastypeRˆRLtclipRµR`RR—(RÁRÅR¼tn_tickstn_min_required_samplestn_max_required_samples((sBlib/python2.7/site-packages/sklearn/model_selection/_validation.pyR½ès0       c C s'gg} } t|tj||ƒd ƒ} xâ| D]Ú\} } || }t||||ƒ\}}t|||| ƒ\}}t|||||ƒ\}}|dkr¿|j|d|ƒn|j||d|ƒ| jt||||ƒƒ| jt||||ƒƒq3Wtj| | fƒj S(sETrain estimator on training subsets incrementally and compute scores.iÿÿÿÿR»N( RR6R2R R&R¸RhRfR7tT(R,R+R/R»R)R*RÁRMRRARCt partitionsRÉt partial_traint train_subsetRoRptX_partial_trainty_partial_trainRqRr((sBlib/python2.7/site-packages/sklearn/model_selection/_validation.pyRº#s     c   sÿtˆˆ|ƒ\‰‰}t|ˆdtˆƒƒ}tˆd|ƒ‰td|d| dˆƒ} | ‡‡‡‡‡‡‡‡fd†|jˆˆ|ƒDƒƒ} tj| ƒ} tˆƒ}| j d|}| j ||dƒj d ƒ} | d| d fS( sÃValidation curve. Determine training and test scores for varying parameter values. Compute scores for an estimator with different values of a specified parameter. This is similar to grid search with one parameter. However, this will also compute training scores and is merely a utility for plotting the results. Read more in the :ref:`User Guide `. Parameters ---------- estimator : object type that implements the "fit" and "predict" methods An object of that type which is cloned for each validation. X : array-like, shape (n_samples, n_features) Training vector, where n_samples is the number of samples and n_features is the number of features. y : array-like, shape (n_samples) or (n_samples, n_features), optional Target relative to X for classification or regression; None for unsupervised learning. param_name : string Name of the parameter that will be varied. param_range : array-like, shape (n_values,) The values of the parameter that will be evaluated. groups : array-like, with shape (n_samples,), optional Group labels for the samples used while splitting the dataset into train/test set. cv : int, cross-validation generator or an iterable, optional Determines the cross-validation splitting strategy. Possible inputs for cv are: - None, to use the default 3-fold cross validation, - integer, to specify the number of folds in a `(Stratified)KFold`, - :term:`CV splitter`, - An iterable yielding (train, test) splits as arrays of indices. For integer/None inputs, if the estimator is a classifier and ``y`` is either binary or multiclass, :class:`StratifiedKFold` is used. In all other cases, :class:`KFold` is used. Refer :ref:`User Guide ` for the various cross-validation strategies that can be used here. .. versionchanged:: 0.20 ``cv`` default value if None will change from 3-fold to 5-fold in v0.22. scoring : string, callable or None, optional, default: None A string (see model evaluation documentation) or a scorer callable object / function with signature ``scorer(estimator, X, y)``. n_jobs : int or None, optional (default=None) Number of jobs to run in parallel. ``None`` means 1 unless in a :obj:`joblib.parallel_backend` context. ``-1`` means using all processors. See :term:`Glossary ` for more details. pre_dispatch : integer or string, optional Number of predispatched jobs for parallel execution (default is all). The option can reduce the allocated memory. The string can be an expression like '2*n_jobs'. verbose : integer, optional Controls the verbosity: the higher, the more messages. error_score : 'raise' | 'raise-deprecating' or numeric Value to assign to the score if an error occurs in estimator fitting. If set to 'raise', the error is raised. If set to 'raise-deprecating', a FutureWarning is printed before the error is raised. If a numeric value is given, FitFailedWarning is raised. This parameter does not affect the refit step, which will always raise the error. Default is 'raise-deprecating' but from version 0.22 it will change to np.nan. Returns ------- train_scores : array, shape (n_ticks, n_cv_folds) Scores on training sets. test_scores : array, shape (n_ticks, n_cv_folds) Scores on test set. Notes ----- See :ref:`sphx_glr_auto_examples_model_selection_plot_validation_curve.py` RRRR Rc3 sk|]a\}}ˆD]N}ttƒtˆƒˆˆˆ||ˆdi|ˆ6dddtdˆƒVqqdS(RjR-R!R$N(R R%RR&R'(R(R)R*RQ(R+R$R,t param_namet param_rangeRMRR/(sBlib/python2.7/site-packages/sklearn/model_selection/_validation.pys ¢siii(iii( RRRRR R2R6R¿RWR™R¾RÀ(R,R+R/RÚRÛR;R<RRR RR$R>RÇtn_paramsRÊ((R+R$R,RÚRÛRMRR/sBlib/python2.7/site-packages/sklearn/model_selection/_validation.pyR8sc $ cC sIi}x<|dD]0}tjg|D]}||^q$ƒ||>> scores = [{'a': 1, 'b':10}, {'a': 2, 'b':2}, {'a': 3, 'b':3}, ... {'a': 10, 'b': 10}] # doctest: +SKIP >>> _aggregate_score_dicts(scores) # doctest: +SKIP {'a': array([1, 2, 3, 10]), 'b': array([10, 2, 3, 10])} i(R6R¿(R?RÇRHRJ((sBlib/python2.7/site-packages/sklearn/model_selection/_validation.pyR5°s.(@t__doc__t __future__RRR`RbR[t tracebackRtnumpyR6t scipy.sparsetsparseRŠtbaseRRtutilsRRRtutils.deprecationRtutils.validationR R tutils.metaestimatorsR t utils._joblibR R Rtexternals.six.movesRtmetrics.scorerRRt exceptionsRt_splitRt preprocessingRt__all__R&RLRRR%RfRzRRR†RYRRªR«tlinspaceRR½RºRR5(((sBlib/python2.7/site-packages/sklearn/model_selection/_validation.pytsr       è   ¼     g   u  ¯ ;  v