ó
ù `]c
@ s‚ d d l m Z d d l m Z d d l m Z m Z e j Z e e d e d d d e d d ƒZ
d
e
_ d „ Z d „ Z
d
S( iÿÿÿÿ( t partial( t chain( t
everygramst pad_sequencet pad_leftt left_pad_symbols t pad_rightt right_pad_symbols s‘ Pads both ends of a sentence to length specified by ngram order.
Following convention pads the start of sentence pads its end.
c C s" t t t | d | ƒƒ d | ƒS( sp Helper with some useful defaults.
Applies pad_both_ends to sentence and follows it up with everygrams.
t nt max_len( R t listt
pad_both_ends( t ordert sentence( ( s4 lib/python2.7/site-packages/nltk/lm/preprocessing.pyt padded_everygrams s c s> t t d ˆ ƒ‰ ‡ ‡ f d † | Dƒ t t ˆ | ƒ ƒ f S( sí Default preprocessing for a sequence of sentences.
Creates two iterators:
- sentences padded and turned into sequences of `nltk.util.everygrams`
- sentences padded as above and chained together for a flat stream of words
:param order: Largest ngram length produced by `everygrams`.
:param text: Text to iterate over. Expected to be an iterable of sentences:
Iterable[Iterable[str]]
:return: iterator over text as ngrams, iterator over text as vocabulary data
R c 3 s- | ]# } t t ˆ | ƒ ƒ d ˆ ƒVq d S( R N( R R
( t .0t sent( R t
padding_fn( s4 lib/python2.7/site-packages/nltk/lm/preprocessing.pys 1 s ( R R t flattent map( R t text( ( R R s4 lib/python2.7/site-packages/nltk/lm/preprocessing.pyt padded_everygram_pipeline# s N( t functoolsR t itertoolsR t nltk.utilR R t
from_iterableR t TrueR t __doc__R R ( ( ( s4 lib/python2.7/site-packages/nltk/lm/preprocessing.pyt s