B >?["@sdZddlmZddlZyddlmZddlmZWn,ek r`ddZ dd Z d d ZYnXe d Z Gd dde ZddZddZefddZefddZddZddZdS)z A port of the Gale-Church Aligner. Gale & Church (1993), A Program for Aligning Sentences in Bilingual Corpora. http://aclweb.org/anthology/J93-1004.pdf )divisionN)norm)logsfcCst|}ddd|}|t| |d|d|d|d|d|d|d |d |d |d }|d kr|Sd|SdS)zComplementary error function.g?gś??g5?g`yg?gƸ?gꪂIǿg#v?g9)gS?gޅ1Ogv(?gg@N)absmathZexp)xztrr 9lib/python3.7/site-packages/nltk/translate/gale_church.pyerfccs(4rcCsddt|tdS)u>Return the area under the normal distribution from M{-∞..x}.rg?)rrsqrt)rr r r norm_cdfCsrcCs0ytdt|Stk r*tdSXdS)Nrz-inf)rlogr ValueErrorfloat)rr r r norm_logsfGsrrc@s&eZdZdddddddZdZdZdS) LanguageIndependentgׁsF?g{Gz?gbX9ȶ?gI +?))rr)rr)rr)rr)rr)rrrg333333@N)__name__ __module__ __qualname__PRIORSAVERAGE_CHARACTERSVARIANCE_CHARACTERSr r r r rQsrc Csg}t|t|f}x|dkrtdd|Dry||\}}Wn.tk rn|dd|ddf}wYnXxHt|D]<}x6t|D]*}||d|d|d|dfqWqzW|d||d|f}qW|dddS)a Traverse the alignment cost from the tracebacks and retrieves appropriate sentence pairs. :param backlinks: A dictionary where the key is the alignment points and value is the cost (referencing the LanguageIndependent.PRIORS) :type backlinks: dict :param source_sents_lens: A list of target sentences' lengths :type source_sents_lens: list(int) :param target_sents_lens: A list of target sentences' lengths :type target_sents_lens: list(int) )rrcss|]}|dkVqdS)rNr ).0pr r r qsztrace..rrN)lenall TypeErrorrangeappend) backlinkssource_sents_lenstarget_sents_lensZlinksZpositionsr ijr r r tracecs .r,c stfddt|dD}tfddt|dD}y4|||jd}||j|t||j} Wntk rtdSXtt t | t |j | S)aPReturns the log probability of the two sentences C{source_sents[i]}, C{target_sents[j]} being aligned with a specific C{alignment}. @param i: The offset of the source sentence. @param j: The offset of the target sentence. @param source_sents: The list of source sentence lengths. @param target_sents: The list of target sentence lengths. @param alignment: The alignment type, a tuple of two integers. @param params: The sentence alignment parameters. @returns: The log probability of a specific alignment between the two sentences, given the parameters. c3s|]}|dVqdS)rNr )roffset)r* source_sentsr r rsz!align_log_prob..rc3s|]}|dVqdS)rNr )rr-)r+ target_sentsr r rsrrz-inf) sumr$rrrrZeroDivisionErrorrLOG2rrrr) r*r+r.r/Z alignmentparamsZl_sZl_tmZdeltar )r*r+r.r/r align_log_probs   r5c Cst|j}gg}i}xtt|dD]}xtt|dD]}td}d} xj|D]b} d| d} || d} | t| ksV| dkrqV|| | t||||| |} | |krV| }| } qVW|tdkrd}| |||f<|d|q@Wt|dkr|d|gq*Wt |||S)aReturn the sentence alignment of two text blocks (usually paragraphs). >>> align_blocks([5,5,5], [7,7,7]) [(0, 0), (1, 1), (2, 2)] >>> align_blocks([10,5,5], [12,20]) [(0, 0), (1, 1), (2, 1)] >>> align_blocks([12,20], [10,5,5]) [(0, 0), (1, 1), (1, 2)] >>> align_blocks([10,2,10,10,2,10], [12,3,20,3,12]) [(0, 0), (1, 1), (2, 2), (3, 2), (4, 3), (5, 4)] @param source_sents_lens: The list of source sentence lengths. @param target_sents_lens: The list of target sentence lengths. @param params: the sentence alignment parameters. @return: The sentence alignments, a list of index pairs. rinfNr rr) listrkeysr$r!rr5r%popr,)r'r(r3Zalignment_typesDr&r*r+Zmin_distZ min_alignaZprev_iZprev_jrr r r align_blockss2       r<cs0t|t|krtdfddt||DS)aCreates the sentence alignment of two texts. Texts can consist of several blocks. Block boundaries cannot be crossed by sentence alignment links. Each block consists of a list that contains the lengths (in characters) of the sentences in this block. @param source_blocks: The list of blocks in the source text. @param target_blocks: The list of blocks in the target text. @param params: the sentence alignment parameters. @returns: A list of sentence alignment lists z>Source and target texts do not have the same number of blocks.csg|]\}}t||qSr )r<)rZ source_blockZ target_block)r3r r szalign_texts..)r!rzip)Z source_blocksZ target_blocksr3r )r3r align_textss  r?c#s&fdd}x|VqWdS)zSplits an iterator C{it} at values of C{split_value}. Each instance of C{split_value} is swallowed. The iterator produces subiterators which need to be consumed fully before the next subiterator can be used. c3s$|}x|kr|V}qWdS)N)next)firstv)it split_valuer r _chunk_iterators z!split_at.._chunk_iteratorN)r@)rCrDrEr )rCrDr split_atsrFcsfddt||DS)zParses a stream of tokens and splits it into sentences (using C{soft_delimiter} tokens) and blocks (using C{hard_delimiter} tokens) for use with the L{align_texts} function. cs g|]}ddt|DqS)cSsg|]}tdd|DqS)css|]}t|VqdS)N)r!)rtokenr r r rsz;parse_token_stream....)r0)rZ sentence_itr r r r=sz1parse_token_stream...)rF)rZblock_it)soft_delimiterr r r=sz&parse_token_stream..)rF)streamrHZhard_delimiterr )rHr parse_token_streams rJ)__doc__Z __future__rrZ scipy.statsrrr ImportErrorrrrr2objectrr,r5r<r?rFrJr r r r s"  '  6