ó ù`]c@s½dZddlmZddlmZmZmZddlmZm Z ddl m Z m Z ddl mZddlmZddlmZd Zed efd „ƒYƒZd „Zd S(uV A general interface to the SENNA pipeline that supports any of the operations specified in SUPPORTED_OPERATIONS. Applying multiple operations at once has the speed advantage. For example, Senna will automatically determine POS tags if you are extracting named entities. Applying both of the operations will cost only the time of extracting the named entities. The SENNA pipeline has a fixed maximum size of the sentences that it can read. By default it is 1024 token/sentence. If you have larger sentences, changing the MAX_SENTENCE_SIZE value in SENNA_main.c should be considered and your system specific binary should be rebuilt. Otherwise this could introduce misalignment errors. The input is: - path to the directory that contains SENNA executables. If the path is incorrect, Senna will automatically search for executable file specified in SENNA environment variable - List of the operations needed to be performed. - (optionally) the encoding of the input data (default:utf-8) Note: Unit tests for this module can be found in test/unit/test_senna.py >>> from __future__ import unicode_literals >>> from nltk.classify import Senna >>> pipeline = Senna('/usr/share/senna-v3.0', ['pos', 'chk', 'ner']) >>> sent = 'Dusseldorf is an international business center'.split() >>> [(token['word'], token['chk'], token['ner'], token['pos']) for token in pipeline.tag(sent)] # doctest: +SKIP [('Dusseldorf', 'B-NP', 'B-LOC', 'NNP'), ('is', 'B-VP', 'O', 'VBZ'), ('an', 'B-NP', 'O', 'DT'), ('international', 'I-NP', 'O', 'JJ'), ('business', 'I-NP', 'O', 'NN'), ('center', 'I-NP', 'O', 'NN')] iÿÿÿÿ(tunicode_literals(tpathtseptenviron(tPopentPIPE(t architecturetsystem(t text_type(tTaggerI(tpython_2_unicode_compatibleuhttp://ml.nec-labs.com/senna/tSennacBsGeZdddgZdd„Zd„Zd„Zd„Zd„ZRS( uposuchkuneruutf-8cCs³||_tj|ƒt|_|j|jƒ}tj|ƒs¦dtkr¦tjtdƒt|_|j|jƒ}tj|ƒs£td||fƒ‚q£q¦n||_ dS(NuSENNAu3Senna executable expected at %s or %s but not found( t _encodingRtnormpathRt_patht executabletisfileRtOSErrort operations(tselft senna_pathRtencodingt exe_file_1t exe_file_2((s2lib/python2.7/site-packages/nltk/classify/senna.pyt__init__<s  cCs–tƒ}|dkrNtƒd}|dkr>tj|dƒStj|dƒS|dkrjtj|dƒS|dkr†tj|d ƒStj|d ƒS( uÆ The function that determines the system specific binary that should be used in the pipeline. In case, the system is not known the default senna binary will be used. uLinuxiu64bitu senna-linux64u senna-linux32uWindowsusenna-win32.exeuDarwinu senna-osxusenna(RRRtjoin(Rt base_pathtos_nametbits((s2lib/python2.7/site-packages/nltk/classify/senna.pyRQs      cCsJi}d}x7tjD],}||jkr|||<|d7}qqW|S(u¨ A method that calculates the order of the columns that SENNA pipeline will output the tags into. This depends on the operations being ordered. i(R tSUPPORTED_OPERATIONSR(Rt_maptit operation((s2lib/python2.7/site-packages/nltk/classify/senna.pyRcs cCs|j|gƒdS(uI Applies the specified operation(s) on a list of tokens. i(t tag_sents(Rttokens((s2lib/python2.7/site-packages/nltk/classify/senna.pyttagpscCsI|j}tj|j|jƒƒsCtd|j|jƒƒ‚n|j|jƒd|jddg}|jg|jD]}d|^qwƒdjd„|Dƒƒd}t |t ƒrÒ|rÒ|j |ƒ}nt |dt d t d t ƒ}|jd |ƒ\}}|} |jd kr0td |ƒ‚n|rH|j|ƒ} n|jƒ} gg} d } d } xÙ| jƒjdƒD]Â}|s®| jgƒ| d7} d } qn|jdƒ}i}x&| D]}|| |jƒ||ststdintstdouttstderrtinputiu!Senna command failed! Details: %siu uwordu´Misalignment error occurred at sentence number %d. Possible reason is that the sentence size exceeded the maximum size. Check the documentation of Senna class for more information.iÿÿÿÿ(R RRRRRtextendRRt isinstanceRtencodeRRt communicatet returncodet RuntimeErrortdecodeRtstriptsplittappendt IndexError(Rt sentencesRt _senna_cmdtopt_inputtpR'R(t senna_outputtmap_ttagged_sentencestsentence_indext token_indext tagged_wordttagstresultR#((s2lib/python2.7/site-packages/nltk/classify/senna.pyR!vsX  '      (t__name__t __module__RRRRR#R!(((s2lib/python2.7/site-packages/nltk/classify/senna.pyR 7s    cCsPddlm}ytddddgƒ}Wntk rK|dƒ‚nXdS(Niÿÿÿÿ(tSkipTestu/usr/share/senna-v3.0uposuchkuneruSenna executable not found(tnoseRDR R(tmoduleRDttagger((s2lib/python2.7/site-packages/nltk/classify/senna.pyt setup_module½s  N(t__doc__t __future__RtosRRRt subprocessRRtplatformRRtsixRt nltk.tag.apiR t nltk.compatR t _senna_urlR RH(((s2lib/python2.7/site-packages/nltk/classify/senna.pyt's…