ó ‡ˆ\c@ s|dZddlmZddlZddlmZddlZddlm Z ddl m Z ddl m Z dd l mZdd l mZmZdd lmZdd lmZdd lmZddlmZddlmZddlmZdd„Zdddd„Zdddd„Z ddd„Z!ddd„Z"dde#d„Z$dd„Z%dd„Z&dd„Z'dS(sMetrics to assess performance on classification task given scores Functions named as ``*_score`` return a scalar value to maximize: the higher the better Function named as ``*_error`` or ``*_loss`` return a scalar value to minimize: the lower the better iÿÿÿÿ(tdivisionN(tpartial(t csr_matrix(trankdatai(tassert_all_finite(tcheck_consistent_length(t column_or_1dt check_array(ttype_of_target(t stable_cumsum(t count_nonzero(tUndefinedMetricWarning(tlabel_binarizei(t_average_binary_scoret deprecatedcC sGt||ƒt|ƒ}t|ƒ}|jddkrNtd|jƒ‚n|dkrmtjdtƒnd}|tkr¬tj ||fƒ}||||}}nZtj |ƒ}tj |dkƒrtj |dkƒrîd}qtdj |ƒƒ‚n|tj||ƒ}t|tjƒrC|jj|ƒ}n|S( sCompute Area Under the Curve (AUC) using the trapezoidal rule This is a general function, given points on a curve. For computing the area under the ROC-curve, see :func:`roc_auc_score`. For an alternative way to summarize a precision-recall curve, see :func:`average_precision_score`. Parameters ---------- x : array, shape = [n] x coordinates. These must be either monotonic increasing or monotonic decreasing. y : array, shape = [n] y coordinates. reorder : boolean, optional (default='deprecated') Whether to sort x before computing. If False, assume that x must be either monotonic increasing or monotonic decreasing. If True, y is used to break ties when sorting x. Make sure that y has a monotonic relation to x when setting reorder to True. .. deprecated:: 0.20 Parameter ``reorder`` has been deprecated in version 0.20 and will be removed in 0.22. It's introduced for roc_auc_score (not for general use) and is no longer used there. What's more, the result from auc will be significantly influenced if x is sorted unexpectedly due to slight floating point error (See issue #9786). Future (and default) behavior is equivalent to ``reorder=False``. Returns ------- auc : float Examples -------- >>> import numpy as np >>> from sklearn import metrics >>> y = np.array([1, 1, 2, 2]) >>> pred = np.array([0.1, 0.4, 0.35, 0.8]) >>> fpr, tpr, thresholds = metrics.roc_curve(y, pred, pos_label=2) >>> metrics.auc(fpr, tpr) 0.75 See also -------- roc_auc_score : Compute the area under the ROC curve average_precision_score : Compute average precision from prediction scores precision_recall_curve : Compute precision-recall pairs for different probability thresholds iisJAt least 2 points are needed to compute area under curve, but x.shape = %sRsÂThe 'reorder' parameter has been deprecated in version 0.20 and will be removed in 0.22. It is recommended not to set 'reorder' and ensure that x is monotonic increasing or monotonic decreasing.iiÿÿÿÿs,x is neither increasing nor decreasing : {}.(RRtshapet ValueErrortwarningstwarntDeprecationWarningtTruetnptlexsorttdifftanytalltformatttrapzt isinstancetmemmaptdtypettype(txtytreordert directiontordertdxtarea((s6lib/python2.7/site-packages/sklearn/metrics/ranking.pytauc)s.2         tmacroc C s¼dd d„}t|ƒ}|dkrB|dkrBtdƒ‚nO|dkr‘tj|ƒ}t|ƒdkr‘||kr‘td|ƒ‚q‘nt|d|ƒ}t||||d |ƒS( sþ Compute average precision (AP) from prediction scores AP summarizes a precision-recall curve as the weighted mean of precisions achieved at each threshold, with the increase in recall from the previous threshold used as the weight: .. math:: \text{AP} = \sum_n (R_n - R_{n-1}) P_n where :math:`P_n` and :math:`R_n` are the precision and recall at the nth threshold [1]_. This implementation is not interpolated and is different from computing the area under the precision-recall curve with the trapezoidal rule, which uses linear interpolation and can be too optimistic. Note: this implementation is restricted to the binary classification task or multilabel classification task. Read more in the :ref:`User Guide `. Parameters ---------- y_true : array, shape = [n_samples] or [n_samples, n_classes] True binary labels or binary label indicators. y_score : array, shape = [n_samples] or [n_samples, n_classes] Target scores, can either be probability estimates of the positive class, confidence values, or non-thresholded measure of decisions (as returned by "decision_function" on some classifiers). average : string, [None, 'micro', 'macro' (default), 'samples', 'weighted'] If ``None``, the scores for each class are returned. Otherwise, this determines the type of averaging performed on the data: ``'micro'``: Calculate metrics globally by considering each element of the label indicator matrix as a label. ``'macro'``: Calculate metrics for each label, and find their unweighted mean. This does not take label imbalance into account. ``'weighted'``: Calculate metrics for each label, and find their average, weighted by support (the number of true instances for each label). ``'samples'``: Calculate metrics for each instance, and find their average. Will be ignored when ``y_true`` is binary. pos_label : int or str (default=1) The label of the positive class. Only applied to binary ``y_true``. For multilabel-indicator ``y_true``, ``pos_label`` is fixed to 1. sample_weight : array-like of shape = [n_samples], optional Sample weights. Returns ------- average_precision : float References ---------- .. [1] `Wikipedia entry for the Average precision `_ See also -------- roc_auc_score : Compute the area under the ROC curve precision_recall_curve : Compute precision-recall pairs for different probability thresholds Examples -------- >>> import numpy as np >>> from sklearn.metrics import average_precision_score >>> y_true = np.array([0, 0, 1, 1]) >>> y_scores = np.array([0.1, 0.4, 0.35, 0.8]) >>> average_precision_score(y_true, y_scores) # doctest: +ELLIPSIS 0.83... Notes ----- .. versionchanged:: 0.19 Instead of linearly interpolating between operating points, precisions are weighted by the change in recall since the last operating point. icS sLt||d|d|ƒ\}}}tjtj|ƒtj|ƒd ƒ S(Nt pos_labelt sample_weightiÿÿÿÿ(tprecision_recall_curveRtsumRtarray(ty_truety_scoreR)R*t precisiontrecallt_((s6lib/python2.7/site-packages/sklearn/metrics/ranking.pyt(_binary_uninterpolated_average_precisionÛs!smultilabel-indicatorsnParameter pos_label is fixed to 1 for multilabel-indicator y_true. Do not set pos_label or set pos_label to 1.tbinaryis5pos_label=%r is invalid. Set it to a label in y_true.R)R*N(tNoneRRRtuniquetlenRR ( R.R/taverageR)R*R3ty_typetpresent_labelstaverage_precision((s6lib/python2.7/site-packages/sklearn/metrics/ranking.pytaverage_precision_score‚sZ     c std‡fd†}t|ƒ}|dkr[tj|ƒ}t||ƒdd…df}nt||||d|ƒS(sù Compute Area Under the Receiver Operating Characteristic Curve (ROC AUC) from prediction scores. Note: this implementation is restricted to the binary classification task or multilabel classification task in label indicator format. Read more in the :ref:`User Guide `. Parameters ---------- y_true : array, shape = [n_samples] or [n_samples, n_classes] True binary labels or binary label indicators. y_score : array, shape = [n_samples] or [n_samples, n_classes] Target scores, can either be probability estimates of the positive class, confidence values, or non-thresholded measure of decisions (as returned by "decision_function" on some classifiers). For binary y_true, y_score is supposed to be the score of the class with greater label. average : string, [None, 'micro', 'macro' (default), 'samples', 'weighted'] If ``None``, the scores for each class are returned. Otherwise, this determines the type of averaging performed on the data: ``'micro'``: Calculate metrics globally by considering each element of the label indicator matrix as a label. ``'macro'``: Calculate metrics for each label, and find their unweighted mean. This does not take label imbalance into account. ``'weighted'``: Calculate metrics for each label, and find their average, weighted by support (the number of true instances for each label). ``'samples'``: Calculate metrics for each instance, and find their average. Will be ignored when ``y_true`` is binary. sample_weight : array-like of shape = [n_samples], optional Sample weights. max_fpr : float > 0 and <= 1, optional If not ``None``, the standardized partial AUC [3]_ over the range [0, max_fpr] is returned. Returns ------- auc : float References ---------- .. [1] `Wikipedia entry for the Receiver operating characteristic `_ .. [2] Fawcett T. An introduction to ROC analysis[J]. Pattern Recognition Letters, 2006, 27(8):861-874. .. [3] `Analyzing a portion of the ROC curve. McClish, 1989 `_ See also -------- average_precision_score : Area under the precision-recall curve roc_curve : Compute Receiver operating characteristic (ROC) curve Examples -------- >>> import numpy as np >>> from sklearn.metrics import roc_auc_score >>> y_true = np.array([0, 0, 1, 1]) >>> y_scores = np.array([0.1, 0.4, 0.35, 0.8]) >>> roc_auc_score(y_true, y_scores) 0.75 c  sSttj|ƒƒdkr*tdƒ‚nt||d|ƒ\}}}ˆdks`ˆdkrmt||ƒSˆdks…ˆdkr˜tdˆƒ‚ntj|ˆdƒ}||d||g}||d||g}tj|| tj ˆ||ƒƒ}tj|| ˆƒ}t||ƒ} dˆd} ˆ} dd| | | | S( NisLOnly one class present in y_true. ROC AUC score is not defined in that case.R*iis)Expected max_frp in range ]0, 1], got: %rtrightgà?( R7RR6Rt roc_curveR5R't searchsortedtappendtinterp( R.R/R*tfprttprR2tstoptx_interpty_interpt partial_auctmin_areatmax_area(tmax_fpr(s6lib/python2.7/site-packages/sklearn/metrics/ranking.pyt_binary_roc_auc_scoreBs$   %R4NiR*(R5RRR6R R (R.R/R8R*RJRKR9tlabels((RJs6lib/python2.7/site-packages/sklearn/metrics/ranking.pyt roc_auc_scoreôsN  "c C s?t|ƒ}|dkp-|dko-|d k sHtdj|ƒƒ‚nt|||ƒt|ƒ}t|ƒ}t|ƒt|ƒ|d k rŸt|ƒ}ntj|ƒ}|d kr9tj |ddgƒp&tj |ddgƒp&tj |dgƒp&tj |dgƒp&tj |dgƒ r9tdƒ‚n|d krNd}n||k}tj |d d ƒd d d…}||}||}|d k r©||}nd}tj tj |ƒƒd}tj ||jdf} t||ƒ| } |d k r td||ƒ| } nd| | } | | || fS( s¡Calculate true and false positives per binary classification threshold. Parameters ---------- y_true : array, shape = [n_samples] True targets of binary classification y_score : array, shape = [n_samples] Estimated probabilities or decision function pos_label : int or str, default=None The label of the positive class sample_weight : array-like of shape = [n_samples], optional Sample weights. Returns ------- fps : array, shape = [n_thresholds] A count of false positives, at index i being the number of negative samples assigned a score >= thresholds[i]. The total number of negative samples is equal to fps[-1] (thus true negatives are given by fps[-1] - fps). tps : array, shape = [n_thresholds <= len(np.unique(y_score))] An increasing count of true positives, at index i being the number of positive samples assigned a score >= thresholds[i]. The total number of positive samples is equal to tps[-1] (thus false negatives are given by tps[-1] - tps). thresholds : array, shape = [n_thresholds] Decreasing score values. R4t multiclasss{0} format is not supportediiiÿÿÿÿs1Data is not binary and pos_label is not specifiedgð?tkindt mergesortN(RR5RRRRRRR6t array_equaltargsorttwhereRtr_tsizeR ( R.R/R)R*R9tclassestdesc_score_indicestweighttdistinct_value_indicestthreshold_idxsttpstfps((s6lib/python2.7/site-packages/sklearn/metrics/ranking.pyt_binary_clf_curvegsF#           "     c C s«t||d|d|ƒ\}}}|||}d|tj|ƒ<||d}|j|dƒ} t| ddƒ} tj|| dftj|| df|| fS(s Compute precision-recall pairs for different probability thresholds Note: this implementation is restricted to the binary classification task. The precision is the ratio ``tp / (tp + fp)`` where ``tp`` is the number of true positives and ``fp`` the number of false positives. The precision is intuitively the ability of the classifier not to label as positive a sample that is negative. The recall is the ratio ``tp / (tp + fn)`` where ``tp`` is the number of true positives and ``fn`` the number of false negatives. The recall is intuitively the ability of the classifier to find all the positive samples. The last precision and recall values are 1. and 0. respectively and do not have a corresponding threshold. This ensures that the graph starts on the y axis. Read more in the :ref:`User Guide `. Parameters ---------- y_true : array, shape = [n_samples] True targets of binary classification in range {-1, 1} or {0, 1}. probas_pred : array, shape = [n_samples] Estimated probabilities or decision function. pos_label : int or str, default=None The label of the positive class sample_weight : array-like of shape = [n_samples], optional Sample weights. Returns ------- precision : array, shape = [n_thresholds + 1] Precision values such that element i is the precision of predictions with score >= thresholds[i] and the last element is 1. recall : array, shape = [n_thresholds + 1] Decreasing recall values such that element i is the recall of predictions with score >= thresholds[i] and the last element is 0. thresholds : array, shape = [n_thresholds <= len(np.unique(probas_pred))] Increasing thresholds on the decision function used to compute precision and recall. See also -------- average_precision_score : Compute average precision from prediction scores roc_curve : Compute Receiver operating characteristic (ROC) curve Examples -------- >>> import numpy as np >>> from sklearn.metrics import precision_recall_curve >>> y_true = np.array([0, 0, 1, 1]) >>> y_scores = np.array([0.1, 0.4, 0.35, 0.8]) >>> precision, recall, thresholds = precision_recall_curve( ... y_true, y_scores) >>> precision # doctest: +ELLIPSIS array([0.66666667, 0.5 , 1. , 1. ]) >>> recall array([1. , 0.5, 0.5, 0. ]) >>> thresholds array([0.35, 0.4 , 0.8 ]) R)R*iiÿÿÿÿiN(R]RtisnanR?tsliceR5RT( R.t probas_predR)R*R\R[t thresholdsR0R1tlast_indtsl((s6lib/python2.7/site-packages/sklearn/metrics/ranking.pyR+ÁsG c C s¶t||d|d|ƒ\}}}|r¤t|ƒdkr¤tjtjttjtj|dƒtj|dƒƒtfƒd}||}||}||}n|jdksÓ|ddksÓ|ddkrtjd|f}tjd|f}tj|dd|f}n|ddkrRt j dt ƒtj tj |jƒ} n||d} |ddkr›t j dt ƒtj tj |jƒ} n||d} | | |fS( sç Compute Receiver operating characteristic (ROC) Note: this implementation is restricted to the binary classification task. Read more in the :ref:`User Guide `. Parameters ---------- y_true : array, shape = [n_samples] True binary labels. If labels are not either {-1, 1} or {0, 1}, then pos_label should be explicitly given. y_score : array, shape = [n_samples] Target scores, can either be probability estimates of the positive class, confidence values, or non-thresholded measure of decisions (as returned by "decision_function" on some classifiers). pos_label : int or str, default=None Label considered as positive and others are considered negative. sample_weight : array-like of shape = [n_samples], optional Sample weights. drop_intermediate : boolean, optional (default=True) Whether to drop some suboptimal thresholds which would not appear on a plotted ROC curve. This is useful in order to create lighter ROC curves. .. versionadded:: 0.17 parameter *drop_intermediate*. Returns ------- fpr : array, shape = [>2] Increasing false positive rates such that element i is the false positive rate of predictions with score >= thresholds[i]. tpr : array, shape = [>2] Increasing true positive rates such that element i is the true positive rate of predictions with score >= thresholds[i]. thresholds : array, shape = [n_thresholds] Decreasing thresholds on the decision function used to compute fpr and tpr. `thresholds[0]` represents no instances being predicted and is arbitrarily set to `max(y_score) + 1`. See also -------- roc_auc_score : Compute the area under the ROC curve Notes ----- Since the thresholds are sorted from low to high values, they are reversed upon returning them to ensure they correspond to both ``fpr`` and ``tpr``, which are sorted in reversed order during their calculation. References ---------- .. [1] `Wikipedia entry for the Receiver operating characteristic `_ .. [2] Fawcett T. An introduction to ROC analysis[J]. Pattern Recognition Letters, 2006, 27(8):861-874. Examples -------- >>> import numpy as np >>> from sklearn import metrics >>> y = np.array([1, 1, 2, 2]) >>> scores = np.array([0.1, 0.4, 0.35, 0.8]) >>> fpr, tpr, thresholds = metrics.roc_curve(y, scores, pos_label=2) >>> fpr array([0. , 0. , 0.5, 0.5, 1. ]) >>> tpr array([0. , 0.5, 0.5, 1. , 1. ]) >>> thresholds array([1.8 , 0.8 , 0.4 , 0.35, 0.1 ]) R)R*iiiiÿÿÿÿsINo negative samples in y_true, false positive value should be meaninglesssHNo positive samples in y_true, true positive value should be meaningless(R]R7RRSRTRt logical_orRRURRR trepeattnanR( R.R/R)R*tdrop_intermediateR\R[Rat optimal_idxsRBRC((s6lib/python2.7/site-packages/sklearn/metrics/ranking.pyR>s2R!    /  cC sÔt|||ƒt|dtƒ}t|dtƒ}|j|jkrUtdƒ‚nt|ƒ}|dkr¡|dko…|jdk r¡tdj|ƒƒ‚nt|ƒ}| }|j\}}d}xØt t |j |j dƒƒD]·\}\}} |j || !} | j d ks)| j |kr9|d 7}qén||} t| d ƒ| } t| | d ƒ} | | jƒ}|d k r–|||}n||7}qéW|d kr½||}n|tj|ƒ}|S( sŠCompute ranking-based average precision Label ranking average precision (LRAP) is the average over each ground truth label assigned to each sample, of the ratio of true vs. total labels with lower score. This metric is used in multilabel ranking problem, where the goal is to give better rank to the labels associated to each sample. The obtained score is always strictly greater than 0 and the best value is 1. Read more in the :ref:`User Guide `. Parameters ---------- y_true : array or sparse matrix, shape = [n_samples, n_labels] True binary labels in binary indicator format. y_score : array, shape = [n_samples, n_labels] Target scores, can either be probability estimates of the positive class, confidence values, or non-thresholded measure of decisions (as returned by "decision_function" on some classifiers). sample_weight : array-like of shape = [n_samples], optional Sample weights. Returns ------- score : float Examples -------- >>> import numpy as np >>> from sklearn.metrics import label_ranking_average_precision_score >>> y_true = np.array([[1, 0, 0], [0, 0, 1]]) >>> y_score = np.array([[0.75, 0.5, 1], [1, 0.2, 0.1]]) >>> label_ranking_average_precision_score(y_true, y_score) # doctest: +ELLIPSIS 0.416... t ensure_2ds'y_true and y_score have different shapesmultilabel-indicatorR4is{0} format is not supportedgiigð?tmaxN(RRtFalseRRRtndimRRt enumeratetziptindptrtindicesRURtmeanR5RR,(R.R/R*R9t n_samplestn_labelstouttitstartRDtrelevanttscores_itranktLtaux((s6lib/python2.7/site-packages/sklearn/metrics/ranking.pyt%label_ranking_average_precision_score˜s:+   2     cC sût|dtƒ}t|dtƒ}t|||ƒt|ƒ}|dkrdtdj|ƒƒ‚n|j|jkr…tdƒ‚ntjj |dtj |ƒƒ}|j ddƒj d ƒ}||kj ddƒ}|jd ƒ}tj|d |ƒS( s0Coverage error measure Compute how far we need to go through the ranked scores to cover all true labels. The best value is equal to the average number of labels in ``y_true`` per sample. Ties in ``y_scores`` are broken by giving maximal rank that would have been assigned to all tied values. Note: Our implementation's score is 1 greater than the one given in Tsoumakas et al., 2010. This extends it to handle the degenerate case in which an instance has 0 true labels. Read more in the :ref:`User Guide `. Parameters ---------- y_true : array, shape = [n_samples, n_labels] True binary labels in binary indicator format. y_score : array, shape = [n_samples, n_labels] Target scores, can either be probability estimates of the positive class, confidence values, or non-thresholded measure of decisions (as returned by "decision_function" on some classifiers). sample_weight : array-like of shape = [n_samples], optional Sample weights. Returns ------- coverage_error : float References ---------- .. [1] Tsoumakas, G., Katakis, I., & Vlahavas, I. (2010). Mining multi-label data. In Data mining and knowledge discovery handbook (pp. 667-685). Springer US. Rismultilabel-indicators{0} format is not supporteds'y_true and y_score have different shapetmasktaxisiiÿÿÿÿitweights(iÿÿÿÿi(RRkRRRRRRtmat masked_arrayt logical_nottmintreshapeR,tfilledR8(R.R/R*R9t y_score_maskty_min_relevanttcoverage((s6lib/python2.7/site-packages/sklearn/metrics/ranking.pytcoverage_errorïs(  !cC sìt|dtddƒ}t|dtƒ}t|||ƒt|ƒ}|dkrjtdj|ƒƒ‚n|j|jkr‹tdƒ‚n|j\}}t|ƒ}tj |ƒ}x¼t t |j |j dƒƒD]›\}\}} tj ||dtƒ\} } tj| |j|| !d t| ƒƒ} tj| d t| ƒƒ} | | }tj| jƒ|ƒ||`. .. versionadded:: 0.17 A function *label_ranking_loss* Parameters ---------- y_true : array or sparse matrix, shape = [n_samples, n_labels] True binary labels in binary indicator format. y_score : array, shape = [n_samples, n_labels] Target scores, can either be probability estimates of the positive class, confidence values, or non-thresholded measure of decisions (as returned by "decision_function" on some classifiers). sample_weight : array-like of shape = [n_samples], optional Sample weights. Returns ------- loss : float References ---------- .. [1] Tsoumakas, G., Katakis, I., & Vlahavas, I. (2010). Mining multi-label data. In Data mining and knowledge discovery handbook (pp. 667-685). Springer US. Rit accept_sparsetcsrsmultilabel-indicators{0} format is not supporteds'y_true and y_score have different shapeitreturn_inverset minlengthR~tdividetignoretinvalidNgiR(smultilabel-indicator(RRkRRRRRRRtzerosRmRnRoR6RtbincountRpR7tdottcumsumR terrstateRdR8(R.R/R*R9RrRstlossRuRvRDt unique_scorestunique_inversettrue_at_reversed_ranktall_at_reversed_ranktfalse_at_reversed_rankt n_positives((s6lib/python2.7/site-packages/sklearn/metrics/ranking.pytlabel_ranking_loss*s6(   2  "((t__doc__t __future__RRt functoolsRtnumpyRt scipy.sparseRt scipy.statsRtutilsRRRRtutils.multiclassRt utils.extmathR tutils.sparsefuncsR t exceptionsR t preprocessingR tbaseR R'R5R<RMR]R+RR>R|R‰R(((s6lib/python2.7/site-packages/sklearn/metrics/ranking.pyts8    Y q rZ U € W ;