B P?[%;@s2ddlmZmZmZddlZddlZddlZddlZddlm Z ddl m Z m Z ddl mZmZddlmZmZmZddZd d Zd d Zd dZddZddZddZddZddZddZddZddZd7d%d&Z d'd(Z!d8d)d*Z"ed+d,gZ#ed+d-d.d/d0d1d2d3d,g Z$d4d5Z%e&d6kr.edS)9)print_functionabsolute_importdivisionN)treebank) error_listTemplate)WordPos)BrillTaggerTrainer RegexpTagger UnigramTaggercCs tdS)z Run a demo with defaults. See source comments for details, or docstrings of any of the more specific demo_* functions. N)postagrr,lib/python3.7/site-packages/nltk/tbl/demo.pydemosrcCstdddS)zN Exemplify repr(Rule) (see also str(Rule) and Rule.format("verbose")) repr) ruleformatN)r rrrrdemo_repr_rule_format!srcCstdddS)zN Exemplify repr(Rule) (see also str(Rule) and Rule.format("verbose")) str)rN)r rrrrdemo_str_rule_format(srcCstdddS)z* Exemplify Rule.format("verbose") verbose)rN)r rrrrdemo_verbose_rule_format/srcCstttdddggddS)a The feature/s of a template takes a list of positions relative to the current word where the feature should be looked for, conceptually joined by logical OR. For instance, Pos([-1, 1]), given a value V, will hold whenever V is found one step to the left and/or one step to the right. For contiguous ranges, a 2-arg form giving inclusive end points can also be used: Pos(-3, -1) is the same as the arg below. ) templatesN)r rr rrrrdemo_multiposition_feature6s rcCs$tttdgtddggddS)z8 Templates can have more than a single feature. rrr)rN)r rrr rrrrdemo_multifeature_templateEsrcCstddddS)ah Show aggregate statistics per template. Little used templates are candidates for deletion, much used templates may possibly be refined. Deleting unused templates is mostly about saving time and/or space: training is basically O(T) in the number of templates T (also in terms of memory usage, which often will be the limiting factor). T)incremental_statstemplate_statsN)r rrrrdemo_template_statisticsLs r cCsptjdddgddgdd}tjddddgddgdd}ttj||gd d }td t|t|ddd d S)a  Template.expand and Feature.expand are class methods facilitating generating large amounts of templates. See their documentation for details. Note: training with 500 templates can easily fill all available even on relatively small corpora rrF)Z excludezerorT)r!) combinationsz9Generated {0} templates for transformation-based learning)rrrN) rexpandr listrprintformatlenr )ZwordtplsZtagtplsrrrrdemo_generated_templatesXs  r*cCstdddddS)z Plot a learning curve -- the contribution on tagging accuracy of the individual rules. Note: requires matplotlib Tzlearningcurve.png)rseparate_baseline_datalearning_curve_outputN)r rrrrdemo_learning_curvelsr-cCstdddS)zW Writes a file with context for each erroneous word after tagging testing data z errors.txt) error_outputN)r rrrrdemo_error_analysisysr/cCstdddS)zm Serializes the learned tagger to a file in pickle format; reloads it and validates the process. z tagger.pcl)serialize_outputN)r rrrrdemo_serialize_taggersr1cCstdddddS)z Discard rules with low accuracy. This may hurt performance a bit, but will often produce rules which are more interesting read to a human. i gQ? ) num_sentsmin_acc min_scoreN)r rrrrdemo_high_accuracy_rulessr6,r#皙?Frc& Cs|pt}|dkr&ddlm}m}|}t|||||\}}}}|rtj|st||d}t |d}t ||WdQRXt d |t |d}t |}t d |WdQRXnt||d}t d |rt d ||t}t|||| d }t d |||||}t d t||rDt d|||dkrt dx2t|dD] \}}t d || | qfW| rt d|||\} }!t d|st d|}"| r||!|rt||!|"|dt d |n t d||} | r|| dk rxt | d4}#|#d| |#dt|| ddWdQRXt d | | dk r||} t | d}t ||WdQRXt d | t | d}t |}$WdQRXt d | ||}%| |%krt dnt ddS)a Brill Tagger Demonstration :param templates: how many sentences of training and testing data to use :type templates: list of Template :param tagged_data: maximum number of rule instances to create :type tagged_data: C{int} :param num_sents: how many sentences of training and testing data to use :type num_sents: C{int} :param max_rules: maximum number of rule instances to create :type max_rules: C{int} :param min_score: the minimum score for a rule in order for it to be considered :type min_score: C{int} :param min_acc: the minimum score for a rule in order for it to be considered :type min_acc: C{float} :param train: the fraction of the the corpus to be used for training (1=all) :type train: C{float} :param trace: the level of diagnostic tracing output to produce (0-4) :type trace: C{int} :param randomize: whether the training data should be a random subset of the corpus :type randomize: C{bool} :param ruleformat: rule output format, one of "str", "repr", "verbose" :type ruleformat: C{str} :param incremental_stats: if true, will tag incrementally and collect stats for each rule (rather slow) :type incremental_stats: C{bool} :param template_stats: if true, will print per-template statistics collected in training and (optionally) testing :type template_stats: C{bool} :param error_output: the file where errors will be saved :type error_output: C{string} :param serialize_output: the file where the learned tbl tagger will be saved :type serialize_output: C{string} :param learning_curve_output: filename of plot of learning curve(s) (train and also test, if available) :type learning_curve_output: C{string} :param learning_curve_take: how many rules plotted :type learning_curve_take: C{int} :param baseline_backoff_tagger: the file where rules will be saved :type baseline_backoff_tagger: tagger :param separate_baseline_data: use a fraction of the training data exclusively for training baseline :type separate_baseline_data: C{bool} :param cache_baseline_tagger: cache baseline tagger to this file (only interesting as a temporary workaround to get deterministic output from the baseline unigram tagger between python versions) :type cache_baseline_tagger: C{string} Note on separate_baseline_data: if True, reuse training data both for baseline and rule learner. This is fast and fine for a demo, but is likely to generalize worse on unseen data. Also cannot be sensibly used for learning curves on training data (the baseline will be artificially high). Nr)describe_template_setsbrill24)Zbackoffwz*Trained baseline tagger, pickled it to {0}rz Reloaded pickled tagger from {0}zTrained baseline taggerz" Accuracy on test set: {0:0.4f})rzTraining tbl tagger...z&Trained tbl tagger in {0:0.2f} secondsz Accuracy on test set: %.4fr!z Learned rules: z {0:4d} {1:s}zJIncrementally tagging the test data, collecting individual rule statisticsz Rule statistics collectedzbWARNING: train_stats asked for separate_baseline_data=True; the baseline will be artificially high)takez#Wrote plot of learning curve to {0}zTagging the test datazErrors for Brill Tagger %r  zutf-8z,Wrote tagger errors including context to {0}zWrote pickled tagger to {0}z4Reloaded tagger tried on test set, results identicalz;PROBLEM: Reloaded tagger gave different results on test set) REGEXP_TAGGERnltk.tag.brillr:r;_demo_prepare_dataospathexistsr openpickledumpr'r(loadZevaluatetimer train enumerateZrulesZbatch_tag_incrementalZ train_statsZprint_template_statistics _demo_plotZ tag_sentswritejoinrencode)&r tagged_datar3Z max_rulesr5r4rKZtrace randomizerrrr.r0r,Zlearning_curve_takeZbaseline_backoff_taggerr+Zcache_baseline_taggerr:r; training_data baseline_data gold_data testing_dataZbaseline_taggerZ print_rulesZtbrillZtrainerZ brill_taggerZrulenoZruleZ taggedtest teststats trainstatsfZbrill_tagger_reloadedZtaggedtest_reloadedrrrr sX            &       r cCs|dkrtdt}|dks,t||kr4t|}|rPtt|t|t||}|d|}|||}dd|D}|s|} n&t|d} |d| || d} }t|\} } t|\} }t| \}}td | |td | | td |||r dnd || ||fS) Nz%Loading tagged data from treebank... cSsg|]}dd|DqS)cSsg|] }|dqS)rr).0trrr gsz1_demo_prepare_data...r)rZZsentrrrr\gsz&_demo_prepare_data..r#z)Read testing data ({0:d} sents/{1:d} wds)z*Read training data ({0:d} sents/{1:d} wds)z0Read baseline data ({0:d} sents/{1:d} wds) {2:s}z[reused the training set]) r'rZ tagged_sentsr)randomZseedZshuffleint corpus_sizer()rQrKr3rRr+cutoffrSrUrVrTZ bl_cutoffZ trainseqsZ traintokensZtestseqsZ testtokensZ bltrainseqsZ bltraintokensrrrrBWs8         rBc sdg}x"dD]}||d|qWfdd|d|D}dg}x"dD]}||d|q\Wfdd|d|D}ddlm}ttt|}||||||ddddg||dS) NZ initialerrorsZ rulescoresrcsg|]}d|dqS)r! tokencountr)rZx)rWrrr\sz_demo_plot..csg|]}d|dqS)r!rbr)rZrc)rXrrr\srg?) appendZmatplotlib.pyplotZpyplotr&ranger)ZplotZaxisZsavefig) r,rWrXr>Z testcurveZ rulescoreZ traincurveZpltr=r)rWrXrrMs   rM)z^-?[0-9]+(.[0-9]+)?$ZCD)z.*NN)z(The|the|A|a|An|an)$AT)z.*able$ZJJ)z.*ness$rf)z.*ly$ZRB)z.*s$ZNNS)z.*ing$ZVBG)z.*ed$ZVBDcCst|tdd|DfS)Ncss|]}t|VqdS)N)r))rZrcrrr szcorpus_size..)r)sum)Zseqsrrrr`sr`__main__)NNr7r8r#Nr9r#FrFFNNNr8NFN)NN)'Z __future__rrrrCrGr^rJZ nltk.corpusrZnltk.tblrrrArr Znltk.tagr r r rrrrrrr r*r-r/r1r6r rBrMZ NN_CD_TAGGERr@r`__name__rrrr sp     5(