ó ų`]c@sddlmZd„ZdS(iĸĸĸĸ(t defaultdictc seg|jƒD]$}ttt|jdƒƒƒ^q }g|jƒD]$}ttt|jdƒƒƒ^qD}d d d d d dddg‰t|ƒjt|ƒƒ‰t|ƒjt|ƒƒ‰ttƒ‰x6ˆD].\}}ˆdj|ƒˆdj|ƒqÕW‡‡‡‡‡‡fd†}‡‡‡‡‡fd†}|ƒ||ƒ||ƒt ˆƒS(s This module symmetrisatizes the source-to-target and target-to-source word alignment output and produces, aka. GDFA algorithm (Koehn, 2005). Step 1: Find the intersection of the bidirectional alignment. Step 2: Search for additional neighbor alignment points to be added, given these criteria: (i) neighbor alignments points are not in the intersection and (ii) neighbor alignments are in the union. Step 3: Add all other alignment points thats not in the intersection, not in the neighboring alignments that met the criteria but in the original foward/backward alignment outputs. >>> forw = ('0-0 2-1 9-2 21-3 10-4 7-5 11-6 9-7 12-8 1-9 3-10 ' ... '4-11 17-12 17-13 25-14 13-15 24-16 11-17 28-18') >>> back = ('0-0 1-9 2-9 3-10 4-11 5-12 6-6 7-5 8-6 9-7 10-4 ' ... '11-6 12-8 13-12 15-12 17-13 18-13 19-12 20-13 ' ... '21-3 22-12 23-14 24-17 25-15 26-17 27-18 28-18') >>> srctext = ("こãŪ よう お ハロマ į™―č‰ē わい 星 ãŪ  é–Ē数 " ... "ãŊ  ãĻ å…ą ãŦ äļé€Ģįķš ãŦ åĒ—加 する こãĻ が " ... "期åū… さ れる こãĻ を įĪšã— た 。") >>> trgtext = ("Therefore , we expect that the luminosity function " ... "of such halo white dwarfs increases discontinuously " ... "with the luminosity .") >>> srclen = len(srctext.split()) >>> trglen = len(trgtext.split()) >>> >>> gdfa = grow_diag_final_and(srclen, trglen, forw, back) >>> gdfa == sorted(set([(28, 18), (6, 6), (24, 17), (2, 1), (15, 12), (13, 12), ... (2, 9), (3, 10), (26, 17), (25, 15), (8, 6), (9, 7), (20, ... 13), (18, 13), (0, 0), (10, 4), (13, 15), (23, 14), (7, 5), ... (25, 14), (1, 9), (17, 13), (4, 11), (11, 17), (9, 2), (22, ... 12), (27, 18), (24, 16), (21, 3), (19, 12), (17, 12), (5, ... 12), (11, 6), (12, 8)])) True References: Koehn, P., A. Axelrod, A. Birch, C. Callison, M. Osborne, and D. Talbot. 2005. Edinburgh System Description for the 2005 IWSLT Speech Translation Evaluation. In MT Eval Workshop. :type srclen: int :param srclen: the number of tokens in the source language :type trglen: int :param trglen: the number of tokens in the target language :type e2f: str :param e2f: the forward word alignment outputs from source-to-target language (in pharaoh output format) :type f2e: str :param f2e: the backward word alignment outputs from target-to-source language (in pharaoh output format) :rtype: set(tuple(int)) :return: the symmetrized alignment points from the GDFA algorithm t-iĸĸĸĸiitetfcs(tˆƒd}x|tˆƒkr#t}xčtˆƒD]Ú}xŅtˆƒD]Ã}||fˆkrKxĻˆD]}td„t||f|ƒDƒƒ}|\}}|ˆkrj|ˆkrj|ˆkrjˆj|ƒˆdj|ƒˆdj|ƒ|d7}t}qjqjWqKqKWq8W|rPqqWdS(sz Search for the neighbor points and them to the intersected alignment points if criteria are met. icss|]\}}||VqdS(N((t.0titj((s2lib/python2.7/site-packages/nltk/translate/gdfa.pys dsRRN(tlentTruetrangettupletziptaddtFalse(tprev_lent no_new_pointsRRtneighborte_newtf_new(talignedt alignmentt neighborstsrclenttrglentunion(s2lib/python2.7/site-packages/nltk/translate/gdfa.pyt grow_diagSs$ %    cs”xtˆƒD]}xvtˆƒD]h}|ˆkr |ˆkr ||fˆkr ˆj||fƒˆdj|ƒˆdj|ƒq q Wq WdS(sĒ Adds remaining points that are not in the intersection, not in the neighboring alignments but in the original *e2f* and *f2e* alignments RRN(R R (taRR(RRRRR(s2lib/python2.7/site-packages/nltk/translate/gdfa.pyt final_andts  (iĸĸĸĸi(iiĸĸĸĸ(ii(ii(iĸĸĸĸiĸĸĸĸ(iĸĸĸĸi(iiĸĸĸĸ(ii( tsplitR tmaptinttsett intersectionRRR tsorted( RRte2ftf2eRRRRR((RRRRRRs2lib/python2.7/site-packages/nltk/translate/gdfa.pytgrow_diag_final_and s:77 !  N(t collectionsRR$(((s2lib/python2.7/site-packages/nltk/translate/gdfa.pyt s