B P?[*!@sdZddlmZddlmZmZmZddlmZddl m Z m Z ddl m Z ddlmZmZdd d Zd d Zd dZGdddeZdddZde_dS)z Utility functions for parsers. )print_function)CFGFeatureGrammarPCFG)load)Chart ChartParser)InsideChartParser) FeatureChartFeatureChartParserNcKst|f|}t|tstdt|trB|dkr4t}||||dSt|trr|dkrXt}|dkrdt}||||dS|dkr~t }|dkrt }||||dSdS)a Load a grammar from a file, and build a parser based on that grammar. The parser depends on the grammar format, and might also depend on properties of the grammar itself. The following grammar formats are currently supported: - ``'cfg'`` (CFGs: ``CFG``) - ``'pcfg'`` (probabilistic CFGs: ``PCFG``) - ``'fcfg'`` (feature-based CFGs: ``FeatureGrammar``) :type grammar_url: str :param grammar_url: A URL specifying where the grammar is located. The default protocol is ``"nltk:"``, which searches for the file in the the NLTK data package. :type trace: int :param trace: The level of tracing that should be used when parsing a text. ``0`` will generate no tracing output; and higher numbers will produce more verbose tracing output. :param parser: The class used for parsing; should be ``ChartParser`` or a subclass. If None, the class depends on the grammar format. :param chart_class: The class used for storing the chart; should be ``Chart`` or a subclass. Only used for CFGs and feature CFGs. If None, the chart class depends on the grammar format. :type beam_size: int :param beam_size: The maximum length for the parser's edge queue. Only used for probabilistic CFGs. :param load_args: Keyword parameters used when loading the grammar. See ``data.load`` for more information. z1The grammar must be a CFG, or a subclass thereof.N)trace beam_size)r chart_class) r isinstancer ValueErrorrr rr r rr)Z grammar_urlr parserrr Z load_argsgrammarr.lib/python3.7/site-packages/nltk/parse/util.py load_parsers$"    rc csRxLt|ddD]<\}\}}t||d||dddddg }d|d}|VqWdS) a A module to convert a single POS tagged sentence into CONLL format. >>> from nltk import word_tokenize, pos_tag >>> text = "This is a foobar sentence." >>> for line in taggedsent_to_conll(pos_tag(word_tokenize(text))): ... print(line, end="") 1 This _ DT DT _ 0 a _ _ 2 is _ VBZ VBZ _ 0 a _ _ 3 a _ DT DT _ 0 a _ _ 4 foobar _ JJ JJ _ 0 a _ _ 5 sentence _ NN NN _ 0 a _ _ 6 . _ . . _ 0 a _ _ :param sentence: A single input sentence to parse :type sentence: list(tuple(str, str)) :rtype: iter(str) :return: a generator yielding a single sentence in CONLL format. )start_0a  N) enumeratestrjoin)sentenceiZwordtag input_strrrrtaggedsent_to_conllPsr$ccs0x*|D]"}xt|D] }|VqWdVqWdS)a5 A module to convert the a POS tagged document stream (i.e. list of list of tuples, a list of sentences) and yield lines in CONLL format. This module yields one line per word and two newlines for end of sentence. >>> from nltk import word_tokenize, sent_tokenize, pos_tag >>> text = "This is a foobar sentence. Is that right?" >>> sentences = [pos_tag(word_tokenize(sent)) for sent in sent_tokenize(text)] >>> for line in taggedsents_to_conll(sentences): ... if line: ... print(line, end="") 1 This _ DT DT _ 0 a _ _ 2 is _ VBZ VBZ _ 0 a _ _ 3 a _ DT DT _ 0 a _ _ 4 foobar _ JJ JJ _ 0 a _ _ 5 sentence _ NN NN _ 0 a _ _ 6 . _ . . _ 0 a _ _ 1 Is _ VBZ VBZ _ 0 a _ _ 2 that _ IN IN _ 0 a _ _ 3 right _ NN NN _ 0 a _ _ 4 ? _ . . _ 0 a _ _ :param sentences: Input sentences to parse :type sentence: list(list(tuple(str, str))) :rtype: iter(str) :return: a generator yielding sentences in CONLL format. z N)r$) sentencesr r#rrrtaggedsents_to_conlljs!  r&c@s$eZdZdZdddZd ddZdS) TestGrammarz Unit tests for CFG. NcCs*||_t|dd|_||_||_||_dS)Nr)r )Z test_grammarrcpsuiteZ_acceptZ_reject)selfrr)acceptrejectrrr__init__s zTestGrammar.__init__Fc Csx|jD]}t|ddddxdD]}x||D]}|}t|j|}|r||r|tt|x|D] }t|qlW|dkr|gkrtd|qd}q4|rtd |q4d} q4Wq&W|r| rtd qWd S) a| Sentences in the test suite are divided into two classes: - grammatical (``accept``) and - ungrammatical (``reject``). If a sentence should parse accordng to the grammar, the value of ``trees`` will be a non-empty list. If a sentence should be rejected according to the grammar, then the value of ``trees`` will be None. doc: )end)r+r,r+zSentence '%s' failed to parse'TzSentence '%s' received a parse'zAll tests passed!N)r)printsplitlistr(parser) r*Z show_treesZtestkeyZsenttokensZtreesZtreeZacceptedZrejectedrrrruns(     zTestGrammar.run)NN)F)__name__ __module__ __qualname____doc__r-r8rrrrr's r'#%;cCs|dk r||}g}x|dD]}|dks"|d|kr s    8,/ '