ó ù`]c @ s=dZddlmZmZmZddlZddlmZddlm Z ddl m Z ddl m Z mZddlmZdd lmZdd lmZejeƒZed efd „ƒYƒZed kr9ddlZddlZddlmZejƒZej ddddddddƒej ddddddddƒej dddddd dgdd!ƒej d"d#dd$dd dgdd%ƒej d&d'dd(dd)ƒej d*d+dd,dd-dd.ƒej d/d0dd1dd2dd3ƒej d4d5dd6dd7dd8ƒej d9d:dd;de!dd<ƒej d=d>dd?de"dd@ddAƒej#ƒ\Z$Z%e$j& rŸej'ƒe(ƒnej)dBdCdDe*e$j+ƒƒgZ,e-e$j&dEƒâZ.xØe.D]ÐZ/e/j0e$j1ƒZ2e2dFe3e2dGd!ƒe4e2dj5ƒj0e$j6ƒƒf\Z7Z8Z9e$j:e$j;kp–e<e$j:ƒdFkore7e$j:kp–e<e$j;ƒdFko–e7e$j;kr²e,j=e7e8e9fƒnqâWWdQXe$j>rìee,e?ee$jƒe$j>ƒƒZ@nee,e?ee$jƒƒZ@e$jArneBe?e@e$jCƒƒƒejDƒndS(HuN Implementations of inter-annotator agreement coefficients surveyed by Artstein and Poesio (2007), Inter-Coder Agreement for Computational Linguistics. An agreement coefficient calculates the amount that annotators agreed on label assignments beyond what is expected by chance. In defining the AnnotationTask class, we use naming conventions similar to the paper's terminology. There are three types of objects in an annotation task: the coders (variables "c" and "C") the items to be annotated (variables "i" and "I") the potential categories to be assigned (variables "k" and "K") Additionally, it is often the case that we don't want to treat two different labels as complete disagreement, and so the AnnotationTask constructor can also take a distance metric as a final argument. Distance metrics are simply functions that take two arguments, and return a value between 0.0 and 1.0 indicating the distance between them. If not supplied, the default is binary comparison between the arguments. The simplest way to initialize an AnnotationTask is with a list of triples, each containing a coder's assignment for one object in the task: task = AnnotationTask(data=[('c1', '1', 'v1'),('c2', '1', 'v1'),...]) Note that the data list needs to contain the same number of triples for each individual coder, containing category values for the same set of items. Alpha (Krippendorff 1980) Kappa (Cohen 1960) S (Bennet, Albert and Goldstein 1954) Pi (Scott 1955) TODO: Describe handling of multiple coders and missing data Expected results from the Artstein and Poesio survey paper: >>> from nltk.metrics.agreement import AnnotationTask >>> import os.path >>> t = AnnotationTask(data=[x.split() for x in open(os.path.join(os.path.dirname(__file__), "artstein_poesio_example.txt"))]) >>> t.avg_Ao() 0.88 >>> t.pi() 0.7995322418977615... >>> t.S() 0.8199999999999998... This would have returned a wrong value (0.0) in @785fb79 as coders are in the wrong order. Subsequently, all values for pi(), S(), and kappa() would have been wrong as they are computed with avg_Ao(). >>> t2 = AnnotationTask(data=[('b','1','stat'),('a','1','stat')]) >>> t2.avg_Ao() 1.0 The following, of course, also works. >>> t3 = AnnotationTask(data=[('a','1','othr'),('b','1','othr')]) >>> t3.avg_Ao() 1.0 iÿÿÿÿ(tprint_functiontunicode_literalstdivisionN(tgroupby(t itemgetter(t iteritems(tFreqDisttConditionalFreqDist(t deprecated(tpython_2_unicode_compatible(tbinary_distancetAnnotationTaskcB seZdZded„Zd„Zd„Zdd„Zd„Z d„Z d„Z e dƒdddd „ƒZ dd „Zd „Zd „Zd „Zdd„Zdd„Zd„Zd„Zd„Zd„Zd„Zd„Zd„Zd„Zdd„Zdd„ZRS(u/Represents an annotation task, i.e. people assign labels to items. Notation tries to match notation in Artstein and Poesio (2007). In general, coders and items can be represented as any hashable object. Integers, for example, are fine, though strings are more readable. Labels must support the distance functions applied to them, so e.g. a string-edit-distance makes no sense if your labels are integers, whereas interval distance needs numeric values. A notable case of this is the MASI metric, which requires Python sets. cC sV||_tƒ|_tƒ|_tƒ|_g|_|dk rR|j|ƒndS(u¯Initialize an annotation task. The data argument can be None (to create an empty annotation task) or a sequence of 3-tuples, each representing a coder's labeling of an item: (coder,item,label) The distance argument is a function taking two arguments (labels) and producing a numerical distance. The distance from a label to itself should be zero: distance(l,l) = 0 N(tdistancetsettItKtCtdatatNonet load_array(tselfRR ((s5lib/python2.7/site-packages/nltk/metrics/agreement.pyt__init__fs      cC sdjtd„|jƒƒS(Nu cS s2d|d|djddƒdj|dƒfS(Nu%s %s %sucoderuitemu_u u,ulabels(treplacetjoin(tx((s5lib/python2.7/site-packages/nltk/metrics/agreement.pyt|s(RtmapR(R((s5lib/python2.7/site-packages/nltk/metrics/agreement.pyt__str__yscC ssxl|D]d\}}}|jj|ƒ|jj|ƒ|jj|ƒ|jji|d6|d6|d6ƒqWdS(u×Load an sequence of annotation results, appending to any data already loaded. The argument is a sequence of 3-tuples, each representing a coder's labeling of an item: (coder,item,label) ucoderulabelsuitemN(RtaddRRRtappend(Rtarraytcodertitemtlabels((s5lib/python2.7/site-packages/nltk/metrics/agreement.pyR‚s c så|p |j}t‡‡‡fd†|Dƒƒ}|dˆkrct‡‡fd†|Dƒƒ}nt‡‡fd†|Dƒƒ}dt|j|d|dƒƒ}tjdˆˆˆ|ƒtjd|d|dd|ƒ|S( u6Agreement between two coders on a given item c3 s;|]1}|dˆˆfkr|dˆkr|VqdS(ucoderuitemN((t.0R(tcAtcBti(s5lib/python2.7/site-packages/nltk/metrics/agreement.pys –sucoderc3 s5|]+}|dˆkr|dˆkr|VqdS(ucoderuitemN((R"R(R$R%(s5lib/python2.7/site-packages/nltk/metrics/agreement.pys ˜sc3 s5|]+}|dˆkr|dˆkr|VqdS(ucoderuitemN((R"R(R#R%(s5lib/python2.7/site-packages/nltk/metrics/agreement.pys šsgð?ulabelsu.Observed agreement between %s and %s on %s: %fu"Distance between "%r" and "%r": %f(RtnexttfloatR tlogtdebug(RR#R$R%Rtk1tk2tret((R#R$R%s5lib/python2.7/site-packages/nltk/metrics/agreement.pytagrŽs""$ c s#tt‡fd†|jDƒƒƒS(Nc3 s%|]}|dˆkrdVqdS(ulabelsiN((R"R(tk(s5lib/python2.7/site-packages/nltk/metrics/agreement.pys §s(R'tsumR(RR.((R.s5lib/python2.7/site-packages/nltk/metrics/agreement.pytNk¦sc s&tt‡‡fd†|jDƒƒƒS(Nc3 s5|]+}|dˆkr|dˆkrdVqdS(uitemulabelsiN((R"R(R%R.(s5lib/python2.7/site-packages/nltk/metrics/agreement.pys ªs(R'R/R(RR%R.((R%R.s5lib/python2.7/site-packages/nltk/metrics/agreement.pytNik©sc s&tt‡‡fd†|jDƒƒƒS(Nc3 s5|]+}|dˆkr|dˆkrdVqdS(ucoderulabelsiN((R"R(tcR.(s5lib/python2.7/site-packages/nltk/metrics/agreement.pys ­s(R'R/R(RR2R.((R2R.s5lib/python2.7/site-packages/nltk/metrics/agreement.pytNck¬suUse Nk, Nik or Nck insteadcC sÞ|dk r6|dkr6|dkr6|j|ƒ}n‹|dk ro|dk ro|dkro|j||ƒ}nR|dk r¨|dk r¨|dkr¨|j||ƒ}ntd|||fƒ‚tjd||||ƒ|S(uHImplements the "n-notation" used in Artstein and Poesio (2007) u7You must pass either i or c, not both! (k=%r,i=%r,c=%r)uCount on N[%s,%s,%s]: %dN(RR0R1R3t ValueErrorR(R)(RR.R%R2R,((s5lib/python2.7/site-packages/nltk/metrics/agreement.pytN¯s$$$cC s4|p |j}tt|dt|ƒƒt|ƒƒS(Ntkey(RRtsortedR(RtfieldR((s5lib/python2.7/site-packages/nltk/metrics/agreement.pyt _grouped_dataÁsc sqˆjd‡‡fd†ˆjDƒƒ}t‡‡‡fd†|Dƒƒtˆjƒ}tjdˆˆ|ƒ|S(u=Observed agreement between two coders on all items. uitemc3 s+|]!}|dˆˆfkr|VqdS(ucoderN((R"R(R#R$(s5lib/python2.7/site-packages/nltk/metrics/agreement.pys Êsc3 s-|]#\}}ˆjˆˆ||ƒVqdS(N(R-(R"R t item_data(R#R$R(s5lib/python2.7/site-packages/nltk/metrics/agreement.pys Ìsu(Observed agreement between %s and %s: %f(R9RR/tlenRR(R)(RR#R$RR,((R#R$Rs5lib/python2.7/site-packages/nltk/metrics/agreement.pytAoÅs "" cC sxd}d}|jjƒ}xL|jD]A}|j|ƒx+|D]#}||||ƒ7}|d7}q?Wq%W||}|S(uP Calculates the average of function results for each coder pair ii(Rtcopytremove(RtfunctionttotaltntsR#R$R,((s5lib/python2.7/site-packages/nltk/metrics/agreement.pyt_pairwise_averageÒs   cC s&|j|jƒ}tjd|ƒ|S(uAAverage observed agreement across all coders and items. uAverage observed agreement: %f(RCR<R(R)(RR,((s5lib/python2.7/site-packages/nltk/metrics/agreement.pytavg_Aoásgð?c  s d}‡‡fd†|jDƒ}xJ|jd|ƒD]6\}}||jt|ƒdt|ƒdƒ7}q5W|t|jƒ|}tjdˆˆ|ƒ|S(uGThe observed disagreement for the weighted kappa coefficient. gc3 s+|]!}|dˆˆfkr|VqdS(ucoderN((R"R(R#R$(s5lib/python2.7/site-packages/nltk/metrics/agreement.pys îsuitemulabelsu+Observed disagreement between %s and %s: %f(RR9R R&R;RR(R)( RR#R$t max_distanceR@RR%titemdataR,((R#R$s5lib/python2.7/site-packages/nltk/metrics/agreement.pytDo_Kw_pairwiseés.c s/ˆj‡‡fd†ƒ}tjd|ƒ|S(u$Averaged over all labelers c sˆj||ˆƒS(N(RG(R#R$(RER(s5lib/python2.7/site-packages/nltk/metrics/agreement.pyRütuObserved disagreement: %f(RCR(R)(RRER,((RERs5lib/python2.7/site-packages/nltk/metrics/agreement.pytDo_Kw÷scC s/dt|jƒ}|jƒ|d|}|S(u,Bennett, Albert and Goldstein 1954 gð?(R;RRD(RtAeR,((s5lib/python2.7/site-packages/nltk/metrics/agreement.pytSscC s„d}td„|jDƒƒ}x(t|ƒD]\}}||d7}q,W|t|jƒt|jƒd}|jƒ|d|S(u_Scott 1955; here, multi-pi. Equivalent to K from Siegel and Castellan (1988). gcs s|]}|dVqdS(ulabelsN((R"R((s5lib/python2.7/site-packages/nltk/metrics/agreement.pys sii(RRRR;RRRD(RR@t label_freqsR.tfRJ((s5lib/python2.7/site-packages/nltk/metrics/agreement.pytpi s $cC sud}tt|jƒƒ}td„|jDƒƒ}x:|jƒD],}|||||||||7}qAW|S(Ngcs s#|]}|d|dfVqdS(ulabelsucoderN((R"R((s5lib/python2.7/site-packages/nltk/metrics/agreement.pys s(R'R;RRRt conditions(RR#R$RJtnitemsRLR.((s5lib/python2.7/site-packages/nltk/metrics/agreement.pytAe_kappas *cC sJ|j||ƒ}|j||ƒ|d|}tjd|||ƒ|S(u gð?u(Expected agreement between %s and %s: %f(RQR<R(R)(RR#R$RJR,((s5lib/python2.7/site-packages/nltk/metrics/agreement.pytkappa_pairwisescC s|j|jƒS(uNCohen 1960 Averages naively over kappas for each coder pair. (RCRR(R((s5lib/python2.7/site-packages/nltk/metrics/agreement.pytkappa'scC s(|j|jƒ}|jƒ|d|S(ulDavies and Fleiss 1982 Averages over observed and expected agreements for each coder pair. gð?(RCRQRD(RRJ((s5lib/python2.7/site-packages/nltk/metrics/agreement.pyt multi_kappa.scC sŠt|jƒƒ}d}x[t|ƒD]M\}}x>t|ƒD]0\}}|t||ƒ|j||ƒ7}q>Wq%Wd|||dS(Nggð?i(R/tvaluesRR'R (RRLt total_labelstpairstjtnjtltnl((s5lib/python2.7/site-packages/nltk/metrics/agreement.pyt Disagreement6s ,c C sSt|jƒdkr$tdƒ‚nt|jƒdkrJtjdƒdSt|jƒdkrƒt|jƒdkrƒtdƒ‚nd}d}tgƒ}d}xx|jdƒD]g\}}td„|Dƒƒ}t |j ƒƒ}|d kr÷q±n||7}||j |ƒ|7}q±W|t |j ƒƒ} |j |ƒ} d | | } | S( uKrippendorff 1980 iu(Cannot calculate alpha, no data present!iu.Only one annotation value, allpha returning 1.u8Cannot calculate alpha, only one coder and item present!guitemcs s|]}|dVqdS(ulabelsN((R"R((s5lib/python2.7/site-packages/nltk/metrics/agreement.pys Qsigð?( R;RR4R(R)RRRR9R/RUR\( Rttotal_disagreementt total_ratingstall_valid_labels_freqttotal_doR%RFRLt labels_counttdotdetk_alpha((s5lib/python2.7/site-packages/nltk/metrics/agreement.pytalpha>s, *   c  sØd}t‡‡fd†|jDƒƒ}xS|jD]H}x?|jD]4}||ˆ||ˆ||j||ƒ7}qBWq2W||tt|jƒdƒ}tjdˆˆ|ƒ|j ˆˆƒ} d| |} | S(uCohen 1968 gc3 s9|]/}|dˆˆfkr|d|dfVqdS(ucoderulabelsN((R"R(R#R$(s5lib/python2.7/site-packages/nltk/metrics/agreement.pys fsiu+Expected disagreement between %s and %s: %fgð?( RRRR tpowR;RR(R)RG( RR#R$RER@RLRXRZtDetDoR,((R#R$s5lib/python2.7/site-packages/nltk/metrics/agreement.pytweighted_kappa_pairwise`s6 c sˆj‡‡fd†ƒS(uCohen 1968 c sˆj||ˆƒS(N(Ri(R#R$(RER(s5lib/python2.7/site-packages/nltk/metrics/agreement.pyRvRH(RC(RRE((RERs5lib/python2.7/site-packages/nltk/metrics/agreement.pytweighted_kappaqsN(t__name__t __module__t__doc__RR RRRR-R0R1R3RR5R9R<RCRDRGRIRKRNRQRRRSRTR\ReRiRj(((s5lib/python2.7/site-packages/nltk/metrics/agreement.pyR Xs4                " u__main__(R u-du --distancetdestudistancetdefaultubinary_distancethelpudistance metric to useu-au --agreementu agreementukappau"agreement coefficient to calculateu-eu --excludeuexcludetactionuappendu8coder names to exclude (may be specified multiple times)u-iu --includeuincludeu.coder names to include, same format as excludeu-fu--fileufileuPfile to read labelings from, each line with three columns: 'labeler item labels'u-vu --verboseuverboseu0u+how much debugging to print on stderr (0-4)u-cu --columnsepu columnsepu uIchar/string that separates the three columns in the file, defaults to tabu-lu --labelsepulabelsepu,u[char/string that separates labels (if labelers can assign more than one), defaults to commau-pu --presenceupresenceu=convert each labeling into 1 or 0, based on presence of LABELu-Tu --thoroughuthoroughu store_trueu6calculate agreement for every subset of the annotatorstleveli2i urii(ERmt __future__RRRtloggingt itertoolsRtoperatorRtsixRtnltk.probabilityRRtnltk.internalsRt nltk.compatR tnltk.metrics.distanceR t getLoggerRkR(tobjectR tretoptparset nltk.metricsR t OptionParsertparsert add_optionRtFalset parse_argstoptionst remaindertfilet print_helptexitt basicConfigtinttverboseRtopentinfileRZtsplitt columnsepttokststrt frozensettstriptlabelsepRtobject_R!tincludetexcludeR;Rtpresencetgetattrttasktthoroughtprintt agreementtshutdown(((s5lib/python2.7/site-packages/nltk/metrics/agreement.pytFsÐ ÿ"       ! .$$# $