`]c@ sdZddlmZddlZy$ddlmZddlmZWn,ek rtdZ dZ dZnXej d Z d e fd YZd Zd ZedZedZdZdZdS(s A port of the Gale-Church Aligner. Gale & Church (1993), A Program for Aligning Sentences in Bilingual Corpora. http://aclweb.org/anthology/J93-1004.pdf i(tdivisionN(tnorm(tlogsfcC st|}ddd|}|tj| |d|d|d|d|d|d|d |d |d |d }|d kr|Sd|SdS(sComplementary error function.ig?gś??g5?g`yg?gƸ?gꪂIǿg#v?g9)gS?gޅ1Ogv(?gg@N(tabstmathtexp(txtztttr((s9lib/python2.7/site-packages/nltk/translate/gale_church.pyterfccs(  ( cC sddt|tjdS(s>Return the area under the normal distribution from M{-∞..x}.ig?i(R Rtsqrt(R((s9lib/python2.7/site-packages/nltk/translate/gale_church.pytnorm_cdfCscC s:ytjdt|SWntk r5tdSXdS(Nis-inf(RtlogR t ValueErrortfloat(R((s9lib/python2.7/site-packages/nltk/translate/gale_church.pyt norm_logsfGs itLanguageIndependentcB sDeZidd6dd 6dd 6dd 6dd 6dd 6ZdZdZRS(gׁsF?iig{Gz?gbX9ȶ?igI +?g333333@(ii(ii(ii(ii(ii(ii(t__name__t __module__tPRIORStAVERAGE_CHARACTERStVARIANCE_CHARACTERS(((s9lib/python2.7/site-packages/nltk/translate/gale_church.pyRQs c C sg}t|t|f}x|dkrtd|Dry||\}}Wn0tk r|dd|ddf}q!nXxVt|D]H}x?t|D]1}|j|d|d|d|dfqWqW|d||d|f}q!W|dddS(s Traverse the alignment cost from the tracebacks and retrieves appropriate sentence pairs. :param backlinks: A dictionary where the key is the alignment points and value is the cost (referencing the LanguageIndependent.PRIORS) :type backlinks: dict :param source_sents_lens: A list of target sentences' lengths :type source_sents_lens: list(int) :param target_sents_lens: A list of target sentences' lengths :type target_sents_lens: list(int) ics s|]}|dkVqdS(iN((t.0tp((s9lib/python2.7/site-packages/nltk/translate/gale_church.pys qsiNi(ii(tlentallt TypeErrortrangetappend( t backlinkstsource_sents_lensttarget_sents_lenstlinkstpositiontsRtitj((s9lib/python2.7/site-packages/nltk/translate/gale_church.pyttracecs % 3 c  stfdt|dD}tfdt|dD}y>|||jd}||j|tj||j} Wntk rtdSXtt t | tj |j | S(sPReturns the log probability of the two sentences C{source_sents[i]}, C{target_sents[j]} being aligned with a specific C{alignment}. @param i: The offset of the source sentence. @param j: The offset of the target sentence. @param source_sents: The list of source sentence lengths. @param target_sents: The list of target sentence lengths. @param alignment: The alignment type, a tuple of two integers. @param params: The sentence alignment parameters. @returns: The log probability of a specific alignment between the two sentences, given the parameters. c3 s!|]}|dVqdS(iN((Rtoffset(R$t source_sents(s9lib/python2.7/site-packages/nltk/translate/gale_church.pys sic3 s!|]}|dVqdS(iN((RR'(R%t target_sents(s9lib/python2.7/site-packages/nltk/translate/gale_church.pys siis-inf( tsumRRRR RtZeroDivisionErrorRtLOG2RRR R( R$R%R(R)t alignmenttparamstl_stl_ttmtdelta((R$R%R(R)s9lib/python2.7/site-packages/nltk/translate/gale_church.pytalign_log_probs ))  c C st|jj}gg}i}xPtt|dD]8}xtt|dD]}td}d} x|D]} d| d} || d} | t| ksw| dkrqwn|| | t||||| |} | |krw| }| } qwqwW|tdkrd}n| |||f<|dj|qXWt|dkrf|j dn|jgq;Wt |||S(sReturn the sentence alignment of two text blocks (usually paragraphs). >>> align_blocks([5,5,5], [7,7,7]) [(0, 0), (1, 1), (2, 2)] >>> align_blocks([10,5,5], [12,20]) [(0, 0), (1, 1), (2, 1)] >>> align_blocks([12,20], [10,5,5]) [(0, 0), (1, 1), (1, 2)] >>> align_blocks([10,2,10,10,2,10], [12,3,20,3,12]) [(0, 0), (1, 1), (2, 2), (3, 2), (4, 3), (5, 4)] @param source_sents_lens: The list of source sentence lengths. @param target_sents_lens: The list of target sentence lengths. @param params: the sentence alignment parameters. @return: The sentence alignments, a list of index pairs. itinfiiiN( tlistRtkeysRRRtNoneR3RtpopR&(RR R.talignment_typestDRR$R%tmin_distt min_aligntatprev_itprev_jR((s9lib/python2.7/site-packages/nltk/translate/gale_church.pyt align_blockss2      cC sYt|t|kr'tdngt||D]\}}t|||^q7S(sCreates the sentence alignment of two texts. Texts can consist of several blocks. Block boundaries cannot be crossed by sentence alignment links. Each block consists of a list that contains the lengths (in characters) of the sentences in this block. @param source_blocks: The list of blocks in the source text. @param target_blocks: The list of blocks in the target text. @param params: the sentence alignment parameters. @returns: A list of sentence alignment lists s>Source and target texts do not have the same number of blocks.(RRtzipR@(t source_blockst target_blocksR.t source_blockt target_block((s9lib/python2.7/site-packages/nltk/translate/gale_church.pyt align_textss  c# s4fd}xtr/|jVqWdS(sSplits an iterator C{it} at values of C{split_value}. Each instance of C{split_value} is swallowed. The iterator produces subiterators which need to be consumed fully before the next subiterator can be used. c3 s.|}x!|kr)|Vj}q WdS(N(tnext(tfirsttv(titt split_value(s9lib/python2.7/site-packages/nltk/translate/gale_church.pyt_chunk_iteratorsN(tTrueRG(RJRKRL((RJRKs9lib/python2.7/site-packages/nltk/translate/gale_church.pytsplit_ats cC sLgt||D]8}gt||D]}td|D^q&^qS(sParses a stream of tokens and splits it into sentences (using C{soft_delimiter} tokens) and blocks (using C{hard_delimiter} tokens) for use with the L{align_texts} function. cs s|]}t|VqdS(N(R(Rttoken((s9lib/python2.7/site-packages/nltk/translate/gale_church.pys s(RNR*(tstreamtsoft_delimiterthard_delimitertblock_itt sentence_it((s9lib/python2.7/site-packages/nltk/translate/gale_church.pytparse_token_streams(t__doc__t __future__RRt scipy.statsRRRt ImportErrorR R R R,tobjectRR&R3R@RFRNRU(((s9lib/python2.7/site-packages/nltk/translate/gale_church.pyts"   '     6