\c@ s^ ddlmZmZddlmZddlmZddlmZddlZ ddl j Z ddl Z ddlmZddlmZddlmZdd lmZdd lmZdd lmZdd lmZdd lmZddlmZddlmZddlmZddlmZddlm Z ddl!m"Z"ddl!m#Z#ddl!m$Z$ddl!m%Z%ddl!m&Z&ddl!m'Z'ddl!m(Z(ddl!m)Z)ddl!m*Z*ddl!m+Z+ddl!m,Z,ddl!m-Z-ddl!m.Z.dd l!m/Z/dd!l!m0Z0dd"l!m1Z1dd#l!m2Z2dd$l!m3Z3dd%l!m4Z4dd&l!m5Z5dd'l!m6Z6dd(l!m7Z7dd)l!m8Z8dd*l!m9Z9dd+l!m:Z:dd,l!m;Z;dd-l!m<Z<dd.l=m>Z>ie3d/6e4d06e5d16e)d26ee8d3d4d56Z?i&e"d66e$d76ee$d8e@d96ee"d:eAd;6e'd<6d=d>6e,d?6e.d@6ee.d:eAdA6e<dB6ee<d:eAdC6e7dD6e9dE6e*dF6ee+dGdHdI6ee+dGdJdK6e2dL6ee+dMdNdGdJdO6ee*dMdNdP6ee+dMdNdGdHdQ6ee7dMdNdR6ee9dMdNdS6ee+dMdTdGdJdU6ee*dMdTdV6ee+dMdTdGdHdW6ee7dMdTdX6ee9dMdTdY6ee+dMdZdGdJd[6ee*dMdZd\6ee+dMdZdGdHd]6ee7dMdZd^6ee9dMdZd_6ee+dMd`dGdJda6ee*dMd`db6ee+dMd`dGdHdc6ee7dMd`dd6ee9dMd`de6e&df6ZBdgZCie;dh6eCdi6ZDie(dj6e0dk6e1dl6ee1d:eAdm6e-dn6e%do6e:dp6ee:dMdNdq6ee:dMd`dr6ee:dMdTds6ee:dtdJdu6e#dv6ee#dMdNdw6ee#dMd`dx6ee#dMdTdy6e/dz6ZEeFZGeGjHeEeGjHeBeGjHe?eGjHeDdadbdcdddedjdkdzhZIdodpdsdqdrdudvdwdydxdDdEdFdIdKdhdihZJeIjKeJZLdDdEdFdIdKhZMdpdvduhZNdhdidodDdEdFdIdKdvdwdydxdOdPdQdRdSdUdVdWdXdYd[d\d]d^d_hZOd<d>dhdid?dDdEdFdIdKdOdPdQdRdSdUdVdWdXdYd[d\d]d^d_dfhZPd6d@dBhZQdldmdpdqdrdsdudvdwdxdydjdkdzhZRd6d;d?d@dAdBdCdOdPdQdRdSd[d\d]d^d_dUdVdWdXdYdadbdcdddehZSd/d0d5d2hZTd6d;d?d@dAdBdCdFdVd\dSdUdVdWdXdYdLd/d0d1dfhZUd7d9d2d5d<d>dhdidDdEdIdKdOdPdQdRd[d]d^d_dldnhZVd1hZWe d{ZXe jYjZd|e[eGeLd}Z\e d~Z]e jYjZd|e[eGeLdZ^e jYjZd|e[eBeLdZ_e jYjZd|eEdZ`ddge jae jagfddge jbe jbgfddge jbe jagfgZce jYjZdeeEjde?jddZee jYjZdeBjddZfe dZge dZhe jYjZd|e[eGeLe[eEdZie jYjZd|eTeSBdZje jYjZd|eTdZke jYjZd|eTdZle dZme jYjZd|eSdZne jYjZd|eQdZoe jYjZd|eQdZpdZqe dZrdZse jYjZd|eMdZte jYjZd|eMeNBdZue jYjZd|eMdZvdZwe jYjZd|eMdZxe dZye jYjZd|e[eGjze[e?eWdZ{e jYjZd|e[eGe[e?eWeIdZ|e jYjZd|e[eGe[e?eWeLdZ}e jYjZd|eSeRBeTBeWdZ~e dZdS(i(tdivisiontprint_function(tpartial(tproduct(tchainN(tmake_multilabel_classification(tLabelBinarizer(ttype_of_target(t _num_samples(tcheck_random_state(tshuffle(tassert_allclose(tassert_array_equal(tassert_array_less(t assert_equal(tassert_raise_message(t assert_raises(tignore_warnings(taccuracy_score(taverage_precision_score(tbalanced_accuracy_score(tbrier_score_loss(tcohen_kappa_score(tconfusion_matrix(tcoverage_error(texplained_variance_score(tf1_score(t fbeta_score(t hamming_loss(t hinge_loss(tjaccard_similarity_score(t%label_ranking_average_precision_score(tlabel_ranking_loss(tlog_loss(tmatthews_corrcoef(tmean_absolute_error(tmean_squared_error(tmedian_absolute_error(tprecision_recall_curve(tprecision_score(tr2_score(t recall_score(t roc_auc_score(t roc_curve(t zero_one_loss(t_average_binary_scoreR#R$R%Rt multioutputtvariance_weightedR(RRtadjustedt adjusted_balanced_accuracy_scoret normalizetunnormalized_accuracy_scoretunnormalized_confusion_matrixcO sBt||jdt||jddddtjfS(Ntfloattaxisi(Rtastypetsumtnptnewaxis(targstkwargs((s@lib/python2.7/site-packages/sklearn/metrics/tests/test_common.pytostnormalized_confusion_matrixRRt%unnormalized_jaccard_similarity_scoreR,tunnormalized_zero_one_lossR'R)Rtbetaitf2_scoreg?s f0.5_scoretmatthews_corrcoef_scoretaveragetweightedsweighted_f0.5_scoretweighted_f1_scoretweighted_f2_scoretweighted_precision_scoretweighted_recall_scoretmicrosmicro_f0.5_scoretmicro_f1_scoretmicro_f2_scoretmicro_precision_scoretmicro_recall_scoretmacrosmacro_f0.5_scoretmacro_f1_scoretmacro_f2_scoretmacro_precision_scoretmacro_recall_scoretsamplesssamples_f0.5_scoretsamples_f1_scoretsamples_f2_scoretsamples_precision_scoretsamples_recall_scoreRc O skt||\}}}t|t|}tj||tj|dd|fdddtjggS(s' The dimensions of precision-recall pairs and the threshold array as returned by the precision_recall_curve do not match. See func:`sklearn.metrics.precision_recall_curve` This prevents implicit conversion of return value triple to an higher dimensional np.array of dtype('float64') (it will be of dtype('object) instead). This again is needed for assert_array_equal to work correctly. As a workaround we pad the threshold array with NaN values to match the dimension of precision and recall arrays respectively. t pad_widthitmodetconstanttconstant_values(R&tlenR9tarraytpadtnan(R;R<t precisiontrecallt thresholdstpad_threshholds((s@lib/python2.7/site-packages/sklearn/metrics/tests/test_common.pyt(precision_recall_curve_padded_thresholdss   R+R&RR R!tunnormalized_log_lossRRR*tweighted_roc_auctsamples_roc_auct micro_roc_auctmax_fprtpartial_roc_aucRt weighted_average_precision_scoretsamples_average_precision_scoretmicro_average_precision_scoreRc C s$td}|jdddd}|jdddd }ttjtttttt ttj ttgxAtD]9}t |}t ||||||dd|qWxYtD]Q}t |}t t 3}t||||||d||_WdQXqWdS( Niitsizeiterr_msgs%s is not symmetrics%s seems to be symmetric(i(i(R trandintRtSYMMETRIC_METRICStuniontNOT_SYMMETRIC_METRICStsettTHRESHOLDED_METRICSt"METRIC_UNDEFINED_BINARY_MULTICLASSt ALL_METRICSt intersectionR RtAssertionErrorR tmsg(t random_statety_truety_predtnametmetrictcm((s@lib/python2.7/site-packages/sklearn/metrics/tests/test_common.pyt test_symmetrys&          Rc C std}|jdddd}|jdddd }t||dd\}}t8t|}t||||||dd|WdQXdS( NiiRoiR|Rps %s is not sample order invariant(i(i(R RqR RRxR (RR|R}R~ty_true_shufflety_pred_shuffleR((s@lib/python2.7/site-packages/sklearn/metrics/tests/test_common.pyttest_sample_order_invariances   c C sktd}|jdddd }|jdddd }|jd|j}t|||dd\}}}xAtD]9}t|}t||||||dd|qyWxAtD]9}t|}t||||||dd|qWxjt D]b}t|}t||||||dd|t||||||dd|qWdS( NiiRoiiR|Rps %s is not sample order invariant(ii(ii( R RqtnormaltshapeR tMULTILABELS_METRICSRxR tTHRESHOLDED_MULTILABEL_METRICStMULTIOUTPUT_METRICS( R|R}R~ty_scoreRRty_score_shuffleRR((s@lib/python2.7/site-packages/sklearn/metrics/tests/test_common.pyt7test_sample_order_invariance_multilabel_and_multioutputs4        c C std}|jdddd}|jdddd}t|}t|}tj|tj|}}t|jdt|jdtj|d}tj|d} tj|d} tj|d} tt |} | ||} t | ||| dd|t | ||| dd |t | || | dd |t | ||| dd |t | ||| dd |t | || | dd |t | ||| dd |t | || | dd |t | ||| dd |t t | || t t | | |t t | || t t | | |t t | || t t | | | |t tBtBkrt t | | | nWdQXdS(NiiRoiiiRps,%s is not representation invariant with lists3%s is not representation invariant with np-array-1ds7%s is not representation invariant with np-array-columns@%s is not representation invariant with mix np-array-1d and listsK%s is not representation invariant with mix np-array-1d and np-array-columnsD%s is not representation invariant with mix list and np-array-column(i(i(ii(ii(ii(ii(R RqtlistR9R^R tndimtreshapeRRxR Rt ValueErrorRRR(RR|ty1ty2ty1_listty2_listty1_1dty2_1dt y1_columnt y2_columnty1_rowty2_rowRtmeasure((s@lib/python2.7/site-packages/sklearn/metrics/tests/test_common.pyt&test_format_invariance_with_1d_vectorssf       c C std}|jdddd}|jdddd}tjddg|}tjddg|}d}ddg}t6t|}|||} |} |tkrt| d|} n| ||} t| | dd j || |j d |j d } t| | dd j ||t krt| d |} | ||} t| | dd j || |j d |j d } t| | dd j |nWdQXdS(NiiRoiteggstspamt pos_labelRps+{0} failed string vs number invariance testtOs2{0} failed string object vs number invariance testtlabelss,{0} failed string vs number invariance test(i(i( R RqR9R^RtCLASSIFICATION_METRICStMETRICS_WITH_POS_LABELRR tformatR7tMETRICS_WITH_LABELS( RR|RRty1_strty2_strt pos_label_strt labels_strRtmeasure_with_numbert metric_strtmeasure_with_strtmeasure_with_strobj((s@lib/python2.7/site-packages/sklearn/metrics/tests/test_common.pyt7test_classification_invariance_string_vs_numbers_labelsTsB             c C sLtd}|jdddd }|jdddd }tjddg|}d}tt|}|tkr|}|tkrt|d|}n|||}|||} t || dd j |||j d |} t || dd j |n/t t |||t t ||j d |WdQXdS(NiiRoiRRRRps+{0} failed string vs number invariance testRs2{0} failed string object vs number invariance test(i(i(R RqR9R^RRvtMETRIC_UNDEFINED_BINARYRRR RR7RR( RR|RRRRRRRRR((s@lib/python2.7/site-packages/sklearn/metrics/tests/test_common.pyt4test_thresholded_invariance_string_vs_numbers_labelss,         iiRcC s1x*tD]"\}}ttd|||qWdS(Nscontains NaN, infinity(tinvalidsRR(RR}R((s@lib/python2.7/site-packages/sklearn/metrics/tests/test_common.pyt)test_regression_thresholded_inf_nan_inputscC s1x*tD]"\}}ttd|||qWdS(NsJClassification metrics can't handle a mix of binary and continuous targets(RRR(RR}R((s@lib/python2.7/site-packages/sklearn/metrics/tests/test_common.pyt!test_classification_inf_nan_inputscC sJt|}x9tddgddD]\}}||g|gq#WdS(Niitrepeati(RxR(RRtitj((s@lib/python2.7/site-packages/sklearn/metrics/tests/test_common.pytcheck_single_samples %cC snt|}x]tddgddD]C\}}}}|tj||ggtj||ggq#WdS(NiiRi(RxRR9R^(RRRRtktl((s@lib/python2.7/site-packages/sklearn/metrics/tests/test_common.pytcheck_single_sample_multioutputs +cC st|dS(N(R(R((s@lib/python2.7/site-packages/sklearn/metrics/tests/test_common.pyttest_single_samplescC st|dS(N(R(R((s@lib/python2.7/site-packages/sklearn/metrics/tests/test_common.pyttest_single_sample_multioutputscC stjddddgddddgddddgg}tjddgddgddgg}t|}tt|||dS(Nii(R9R^RxRR(RR}R~R((s@lib/python2.7/site-packages/sklearn/metrics/tests/test_common.pyt(test_multioutput_number_of_output_differs<* cC std}|jdddd }|jdddd }t|}|||}xjtdD]\}|j|jd}t||dd|f|dd|f|dd |qbWdS( NiiRoiiiiRps'%s is not dimension shuffling invariant(ii(ii(R tuniformRxtranget permutationRR (RR|R}R~Rterrort_tperm((s@lib/python2.7/site-packages/sklearn/metrics/tests/test_common.pyt=test_multioutput_regression_invariance_to_dimension_shufflings  /c C s4d}d}tddd|ddd|d t\}}tddd|ddd|d t\}}tj|dg|gg}tj|dg|gg}tj|}tj|}xktD]c}t|}t|t rd |_ ||_ n|||} t |||| d d |qWdS( Nii2t n_featuresit n_classesR|it n_samplestallow_unlabeledttmpRpsO%s failed representation invariance between dense and sparse indicator formats.( RtTrueR9tvstacktspt coo_matrixRRxt isinstanceRt __module__t__name__R ( RRRRRty1_sparse_indicatorty2_sparse_indicatorRRR((s@lib/python2.7/site-packages/sklearn/metrics/tests/test_common.pyt)test_multilabel_representation_invariances,      c C sddggdgdgddggdddgggdgtjgddggddg}t|}x!|D]}tt|||qqWdS( Niiitdtypetobject((ii((R9R^RxRR(Rtmultilabel_sequencesRtseq((s@lib/python2.7/site-packages/sklearn/metrics/tests/test_common.pyt+test_raise_value_error_multilabel_sequences's  $  cC sd}td}|jddd|f}|jddd|f}t|}|||dt}td|dddt|||dt||dS( NiiiRoR2gRps0We failed to test correctly the normalize option(R RqRxRR R tFalse(RRR|R}R~tmetricsR((s@lib/python2.7/site-packages/sklearn/metrics/tests/test_common.pyt+test_normalize_option_binary_classification7s  cC std}|jdddd }|jdddd }|jd}t|}|||dt}td|dddt|||dt||dS( NiiRoiR2gRps0We failed to test correctly the normalize option(i(i(R RqRRxRR R R(RR|R}R~RRR((s@lib/python2.7/site-packages/sklearn/metrics/tests/test_common.pyt/test_normalize_option_multiclass_classificationHs   c C sd}d}tddd|dddtd |\}}tddd|dddtd |\}}|dg|7}|dg|7}xntD]f}t|}|||d t}td |dd d t|||d t||d d|qWdS(NiidRiRR|iRRR2gRps0We failed to test correctly the normalize optionsFailed with %s(RRtMETRICS_WITH_NORMALIZE_OPTIONRxR R R(RRRR}R~RRR((s@lib/python2.7/site-packages/sklearn/metrics/tests/test_common.pyt/test_normalize_option_multilabel_classificationYs*    c C s|j\}}|||dd}t|gt|D]5} ||dd| f|dd| f^q7|||dd} t| ||j|j|||dd} t| tj|tj|dddt} tj| dkr9|||dd} t| tj |d| n"|||dd} t| d|r|||dd }t|tjgt|D]} ||| || ^qnt t |||dd t t |||dd dS( NRDRJROR6iRREtweightsRTtunknowntgarbage( RtNoneR RtravelR9tmeanR8tintRDRR(RR}R~ty_true_binarizety_pred_binarizet is_multilabelRRt label_measureRt micro_measuret macro_measureRtweighted_measuretsample_measure((s@lib/python2.7/site-packages/sklearn/metrics/tests/test_common.pyt_check_averagingys0F  4cC st|jd}t|}|tkrGt||||||n4|tkrot||||||n tddS(Nt multilabels2Metric is not recorded as having an average option(Rt startswithRxtMETRICS_WITH_AVERAGINGRt"THRESHOLDED_METRICS_WITH_AVERAGINGR(RR}RR~RRRR((s@lib/python2.7/site-packages/sklearn/metrics/tests/test_common.pytcheck_averagings     c C sd\}}td}|jd|d|f}|jd|d|f}|jd||f}tj|}|j|}|j|} t||||| |dS(Ni2iiRo(i2i(R RqRRtfitt transformR( RRRR|R}R~RtlbRR((s@lib/python2.7/site-packages/sklearn/metrics/tests/test_common.pyttest_averaging_multiclasss   c C sd \}}tddd|ddd|dt\}}|d }|d }td jd d |f}|}|} t||||| |dS( Ni(iRiRR|RRiiRo(i(i(RRR RR( RRRRtyR}R~RRR((s@lib/python2.7/site-packages/sklearn/metrics/tests/test_common.pyttest_averaging_multilabels     cC sVtjd}tjd}tjd}|}|}t||||||dS(Nii(ii(ii(ii(R9tzerosR(RR}R~RRR((s@lib/python2.7/site-packages/sklearn/metrics/tests/test_common.pyt$test_averaging_multilabel_all_zeroess cC sVtjd}tjd}|}|}dd}t|||||dtdS(NiiROcS stt|||S(N(R-R'(R}RRD((s@lib/python2.7/site-packages/sklearn/metrics/tests/test_common.pyR=sR(ii(ii(R9RRR(R}R~RRt binary_metric((s@lib/python2.7/site-packages/sklearn/metrics/tests/test_common.pyt+test_averaging_binary_multilabel_all_zeroess cC sVtjd}tjd}tjd}|}|}t||||||dS(Nii(ii(ii(ii(R9tonesR(RR}R~RRR((s@lib/python2.7/site-packages/sklearn/metrics/tests/test_common.pyt"test_averaging_multilabel_all_oness c C stjjd}|jdddt|}|||dd}t||||dtjdt|dd||||d|}tt *}t||d |||f|_ WdQX|||d|j } t|| dd || |f|tj ||d dtj ||d ddd} t|| dd ||ddd } tj |} d| ddd <|ddd } |ddd }|| |d| }|||d| }t||dd|||f|jdsMxAd dgD]0}t||||d||dd|qWndjt|t|t|d }tt||||dtj||gdS(Niii Rot sample_weightRRpsAFor %s sample_weight=None is not equivalent to sample_weight=onessQUnweighted and weighted scores are unexpectedly almost equal (%s) and (%s) for %ssVWeighted scores for array and list sample_weight input are not equal (%s != %s) for %sR6s.Weighting %s is not equal to repeating samplesiseZeroing weights does not give the same result as removing the corresponding samples (%s != %s) for %st unnormalizedg333333?s/%s sample_weight is not invariant under scalingsHFound input variables with inconsistent numbers of samples: [{}, {}, {}](R9trandomt RandomStateRqR]RR RRRzR{ttolistRtcopyRRRRRthstack(RRRRtrngRtunweighted_scoretweighted_scoreRtweighted_score_listtrepeat_weighted_scoretsample_weight_subsettsample_weight_zeroedt y1_subsett y2_subsettweighted_score_subsettweighted_score_zeroedtscalingt error_message((s@lib/python2.7/site-packages/sklearn/metrics/tests/test_common.pytcheck_sample_weight_invariancesh'   !        cC s]d}td}|jd|f}|jd|f}t|}t||||dS(Ni2iRo(R t random_sampleRxR(RRR|R}R~R((s@lib/python2.7/site-packages/sklearn/metrics/tests/test_common.pyt(test_regression_sample_weight_invarianceHs   cC sd}td}|jddd|f}|jddd|f}|jd|f}t|}|tkrt||||nt||||dS(Ni2iiRo(R RqRRxRvR(RRR|R}R~RR((s@lib/python2.7/site-packages/sklearn/metrics/tests/test_common.pyt$test_binary_sample_weight_invarianceVs   cC sd}td}|jddd|f}|jddd|f}|jd|df}t|}|tkrt||||nt||||dS(Ni2iiRo(R RqRRxRvR(RRR|R}R~RR((s@lib/python2.7/site-packages/sklearn/metrics/tests/test_common.pyt(test_multiclass_sample_weight_invariancehs   c C std}tddddddddd t\}}tddddddddd t\}}tj||g}tj||g}|jdd d |j}t|}|tkrt ||||nt ||||dS( NiRiRiR|RidRiRo( R RRR9RRqRRxRvR( RR|RtyatybR}R~RR((s@lib/python2.7/site-packages/sklearn/metrics/tests/test_common.pyt(test_multilabel_sample_weight_invariance{s     c C sjtjddddgddddgg}tjddddgddddgg}tjdddg}tjdddg}tjddddg}tj|dt\}}xtD]}x||g||ggD]x\}} |tkr| jdkrqnt|} | || d|dd} | || dd} t | | |qWqWdS(Niiiitreturn_inverseRRD( R9R^tuniqueRRRRRxRR ( ty_true_multilabelty_pred_multilabelty_true_multiclassty_pred_multiclassRRtinverse_labelsRR}R~Rt score_labelstscore((s@lib/python2.7/site-packages/sklearn/metrics/tests/test_common.pyttest_no_averaging_labelss--   (t __future__RRt functoolsRt itertoolsRRtnumpyR9t scipy.sparsetsparseRtpytesttsklearn.datasetsRtsklearn.preprocessingRtsklearn.utils.multiclassRtsklearn.utils.validationRR t sklearn.utilsR tsklearn.utils.testingR R R RRRRtsklearn.metricsRRRRRRRRRRRRRRR R!R"R#R$R%R&R'R(R)R*R+R,tsklearn.metrics.baseR-tREGRESSION_METRICSRRRRet CURVE_METRICSRvtdictRxtupdateRtMETRIC_UNDEFINED_MULTICLASSRsRwRRRRRRRRRrRttMETRICS_WITHOUT_SAMPLE_WEIGHTRtmarkt parametrizeRuRRRRRtinfR`RtvaluesRRRRRRRRRRRRRRRRRRRRRRyRRRRR&(((s@lib/python2.7/site-packages/sklearn/metrics/tests/test_common.pyts2  $                                         ! $ J 0#! ' $   &" % *    O  * *