ó ù`]c@ s²dZddlmZddlZddlmZddlmZddlmZddlm Z ddlm Z dd l m Z d e fd „ƒYZ d e fd „ƒYZdS(s‘ Lexical translation model that considers word order. IBM Model 2 improves on Model 1 by accounting for word order. An alignment probability is introduced, a(i | j,l,m), which predicts a source word position, given its aligned target word's position. The EM algorithm used in Model 2 is: E step - In the training data, collect counts, weighted by prior probabilities. (a) count how many times a source language word is translated into a target language word (b) count how many times a particular position in the source sentence is aligned to a particular position in the target sentence M step - Estimate new probabilities based on the counts from the E step Notations: i: Position in the source sentence Valid values are 0 (for NULL), 1, 2, ..., length of source sentence j: Position in the target sentence Valid values are 1, 2, ..., length of target sentence l: Number of words in the source sentence, excluding NULL m: Number of words in the target sentence s: A word in the source language t: A word in the target language References: Philipp Koehn. 2010. Statistical Machine Translation. Cambridge University Press, New York. Peter E Brown, Stephen A. Della Pietra, Vincent J. Della Pietra, and Robert L. Mercer. 1993. The Mathematics of Statistical Machine Translation: Parameter Estimation. Computational Linguistics, 19 (2), 263-311. iÿÿÿÿ(tdivisionN(t defaultdict(t AlignedSent(t Alignment(tIBMModel(t IBMModel1(tCountst IBMModel2cB sbeZdZd d„Zd„Zd„Zd„Zd„Zd„Z d„Z d„Z d „Z RS( sY Lexical translation model that considers word order >>> bitext = [] >>> bitext.append(AlignedSent(['klein', 'ist', 'das', 'haus'], ['the', 'house', 'is', 'small'])) >>> bitext.append(AlignedSent(['das', 'haus', 'ist', 'ja', 'groß'], ['the', 'house', 'is', 'big'])) >>> bitext.append(AlignedSent(['das', 'buch', 'ist', 'ja', 'klein'], ['the', 'book', 'is', 'small'])) >>> bitext.append(AlignedSent(['das', 'haus'], ['the', 'house'])) >>> bitext.append(AlignedSent(['das', 'buch'], ['the', 'book'])) >>> bitext.append(AlignedSent(['ein', 'buch'], ['a', 'book'])) >>> ibm2 = IBMModel2(bitext, 5) >>> print(round(ibm2.translation_table['buch']['book'], 3)) 1.0 >>> print(round(ibm2.translation_table['das']['book'], 3)) 0.0 >>> print(round(ibm2.translation_table['buch'][None], 3)) 0.0 >>> print(round(ibm2.translation_table['ja'][None], 3)) 0.0 >>> print(ibm2.alignment_table[1][1][2][2]) 0.938... >>> print(round(ibm2.alignment_table[1][2][2][2], 3)) 0.0 >>> print(round(ibm2.alignment_table[2][2][4][5], 3)) 1.0 >>> test_sentence = bitext[2] >>> test_sentence.words ['das', 'buch', 'ist', 'ja', 'klein'] >>> test_sentence.mots ['the', 'book', 'is', 'small'] >>> test_sentence.alignment Alignment([(0, 0), (1, 1), (2, 2), (3, 2), (4, 3)]) cC s£tt|ƒj|ƒ|dkrQt|d|ƒ}|j|_|j|ƒn|d|_|d|_x$td|ƒD]}|j |ƒq{W|j |ƒdS(s™ Train on ``sentence_aligned_corpus`` and create a lexical translation model and an alignment model. Translation direction is from ``AlignedSent.mots`` to ``AlignedSent.words``. :param sentence_aligned_corpus: Sentence-aligned parallel corpus :type sentence_aligned_corpus: list(AlignedSent) :param iterations: Number of iterations to run training algorithm :type iterations: int :param probability_tables: Optional. Use this to pass in custom probability values. If not specified, probabilities will be set to a uniform distribution, or some other sensible value. If specified, all the following entries must be present: ``translation_table``, ``alignment_table``. See ``IBMModel`` for the type and purpose of these tables. :type probability_tables: dict[str]: object ittranslation_tabletalignment_tableiN( tsuperRt__init__tNoneRRtset_uniform_probabilitiesR trangettraint align_all(tselftsentence_aligned_corpust iterationstprobability_tablestibm1tn((s2lib/python2.7/site-packages/nltk/translate/ibm2.pyR es    c C sôtƒ}xä|D]Ü}t|jƒ}t|jƒ}||f|kr|j||fƒd|d}|tjkr”tjdt |ƒdƒnxUt d|dƒD]=}x4t d|dƒD]}||j ||||Ötii(RRRR*(RR1R2talignment_prob_for_tR&R4R%((s2lib/python2.7/site-packages/nltk/translate/ibm2.pyR)Ès c C sZt|ƒd}t|ƒd}||}||}|j|||j||||S(sz Probability that position j in ``trg_sentence`` is aligned to position i in the ``src_sentence`` i(RRR ( RR%R&R1R2R"R#R5R4((s2lib/python2.7/site-packages/nltk/translate/ibm2.pyR*ßs   c C s´d}t|jƒd}t|jƒd}xut|jƒD]d\}}|dkrZq<n|j|}|j|}||j|||j||||9}q<Wt|tj ƒS(sc Probability of target sentence and an alignment given the source sentence gð?ii( RR1R2t enumerateR8RR R;RR( Rtalignment_infotprobR"R#R&R%ttrg_wordtsrc_word((s2lib/python2.7/site-packages/nltk/translate/ibm2.pytprob_t_a_given_sês   cC s"x|D]}|j|ƒqWdS(N(talign(RR/t sentence_pair((s2lib/python2.7/site-packages/nltk/translate/ibm2.pyRÿs c C sg}t|jƒ}t|jƒ}xßt|jƒD]Î\}}|j|d|jd|d||}t|tj ƒ}d}xht|jƒD]W\} } |j|| |j| d|d||} | |kr”| }| }q”q”W|j ||fƒq4Wt |ƒ|_ dS(s Determines the best word alignment for one sentence pair from the corpus that the model was trained on. The best alignment will be set in ``sentence_pair`` when the method returns. In contrast with the internal implementation of IBM models, the word indices in the ``Alignment`` are zero- indexed, not one-indexed. :param sentence_pair: A sentence in the source language and its counterpart sentence in the target language :type sentence_pair: AlignedSent iiN( RRRRCRR R R;RRtappendRR8( RRJtbest_alignmentR"R#R&RFt best_probtbest_alignment_pointR%RGt align_prob((s2lib/python2.7/site-packages/nltk/translate/ibm2.pyRIs "  N( t__name__t __module__t__doc__R R R RR.R)R*RHRRI(((s2lib/python2.7/site-packages/nltk/translate/ibm2.pyR=s& )     R(cB s)eZdZd„Zd„Zd„ZRS(so Data object to store counts of various parameters during training. Includes counts for alignment. cC s;tt|ƒjƒtd„ƒ|_td„ƒ|_dS(NcS s td„ƒS(NcS s td„ƒS(NcS s td„ƒS(NcS sdS(Ng((((s2lib/python2.7/site-packages/nltk/translate/ibm2.pyR@5RA(R(((s2lib/python2.7/site-packages/nltk/translate/ibm2.pyR@5RA(R(((s2lib/python2.7/site-packages/nltk/translate/ibm2.pyR@5RA(R(((s2lib/python2.7/site-packages/nltk/translate/ibm2.pyR@5RAcS s td„ƒS(NcS s td„ƒS(NcS sdS(Ng((((s2lib/python2.7/site-packages/nltk/translate/ibm2.pyR@8RA(R(((s2lib/python2.7/site-packages/nltk/translate/ibm2.pyR@8RA(R(((s2lib/python2.7/site-packages/nltk/translate/ibm2.pyR@8RA(R R(R RR8R:(R((s2lib/python2.7/site-packages/nltk/translate/ibm2.pyR 2s cC s.|j||c|7<|j|c|7|j||||c|7<|j|||c|7/s ï