B >?[@sdZddlmZmZy ddlZWnek r4YnXddlZddlZddlm Z ddl m Z ddl m Z ddlmZddlmZdd lmZdd lmZdd lmZmZmZdd lmZmZmZdd lmZm Z m!Z!dZ"e j#GdddeZ$e$Z%Gddde&Z'Gddde'Z(Gddde'Z)Gddde)Z*Gddde)Z+Gddde'Z,d1ddZ-d d!Z.d"d#Z/d2d$d%Z0d&d'Z1d(d)Z2d3d*d+Z3Gd,d-d-e$Z4d.d/Z5e6d0kre5dS)4a A classifier model based on maximum entropy modeling framework. This framework considers all of the probability distributions that are empirically consistent with the training data; and chooses the distribution with the highest entropy. A probability distribution is "empirically consistent" with a set of training data if its estimated frequency with which a class and a feature vector value co-occur is equal to the actual frequency in the data. Terminology: 'feature' ====================== The term *feature* is usually used to refer to some property of an unlabeled token. For example, when performing word sense disambiguation, we might define a ``'prevword'`` feature whose value is the word preceding the target word. However, in the context of maxent modeling, the term *feature* is typically used to refer to a property of a "labeled" token. In order to prevent confusion, we will introduce two distinct terms to disambiguate these two different concepts: - An "input-feature" is a property of an unlabeled token. - A "joint-feature" is a property of a labeled token. In the rest of the ``nltk.classify`` module, the term "features" is used to refer to what we will call "input-features" in this module. In literature that describes and discusses maximum entropy models, input-features are typically called "contexts", and joint-features are simply referred to as "features". Converting Input-Features to Joint-Features ------------------------------------------- In maximum entropy models, joint-features are required to have numeric values. Typically, each input-feature ``input_feat`` is mapped to a set of joint-features of the form: | joint_feat(token, label) = { 1 if input_feat(token) == feat_val | { and label == some_label | { | { 0 otherwise For all values of ``feat_val`` and ``some_label``. This mapping is performed by classes that implement the ``MaxentFeatureEncodingI`` interface. )print_functionunicode_literalsN) defaultdict) integer_types)compat)gzip_open_unicode) OrderedDict)DictionaryProbDist) ClassifierI) CutoffCheckeraccuracylog_likelihood) call_megamwrite_megam_fileparse_megam_weights) call_tadmwrite_tadm_fileparse_tadm_weightsz epytext enc@seZdZdZd#ddZddZddZd d Zd d Zd dZ d$ddZ d%ddZ d&ddZ ddZ ddddgZed'd!d"ZdS)(MaxentClassifiera A maximum entropy classifier (also known as a "conditional exponential classifier"). This classifier is parameterized by a set of "weights", which are used to combine the joint-features that are generated from a featureset by an "encoding". In particular, the encoding maps each ``(featureset, label)`` pair to a vector. The probability of each label is then computed using the following equation:: dotprod(weights, encode(fs,label)) prob(fs|label) = --------------------------------------------------- sum(dotprod(weights, encode(fs,l)) for l in labels) Where ``dotprod`` is the dot product:: dotprod(a,b) = sum(x*y for (x,y) in zip(a,b)) TcCs*||_||_||_|t|ks&tdS)a{ Construct a new maxent classifier model. Typically, new classifier models are created using the ``train()`` method. :type encoding: MaxentFeatureEncodingI :param encoding: An encoding that is used to convert the featuresets that are given to the ``classify`` method into joint-feature vectors, which are used by the maxent classifier model. :type weights: list of float :param weights: The feature weight vector for this classifier. :type logarithmic: bool :param logarithmic: If false, then use non-logarithmic weights. N) _encoding_weights _logarithmiclengthlenAssertionError)selfencodingweightsZ logarithmicr3lib/python3.7/site-packages/nltk/classify/maxent.py__init__hszMaxentClassifier.__init__cCs |jS)N)rlabels)rrrrr!szMaxentClassifier.labelscCs ||_|jt|kstdS)z Set the feature weight vector for this classifier. :param new_weights: The new feature weight vector. :type new_weights: list of float N)rrrrr)rZ new_weightsrrr set_weightsszMaxentClassifier.set_weightscCs|jS)zg :return: The feature weight vector for this classifier. :rtype: list of float )r)rrrrrszMaxentClassifier.weightscCs||S)N) prob_classifymax)r featuresetrrrclassifyszMaxentClassifier.classifyc Csi}x|jD]z}|j||}|jrZd}x"|D]\}}||j||7}q2W|||<qd}x"|D]\}}||j||9}qdW|||<qWt||jddS)Ngg?T)logZ normalize)rr!encoderrr ) rr%Z prob_dictlabelfeature_vectortotalf_idf_valZprodrrrr#s  zMaxentClassifier.prob_classifyc sd}dt|dd}|tjdd}|d|}td|d d d |Dtd d |ddt|t t xt |D]\}}j ||}|jfddddx|D]\} } jrj| | } nj| | } j | } | dd} | d| 7} t| dkr8| ddd} t|| |dd| f|| 7<qWqWtd d |ddt|td|d fdd |Dtd|d fdd |DdS)z Print a table showing the effect of each of the features in the given feature set, and how they combine to determine the probabilities of each label for that featureset. 2z %-zs%s%8.3fT)keyreverseNz Featurecss"|]}dd|ddVqdS)z%8sz%sNr).0lrrr sz+MaxentClassifier.explain..z -cstj|dS)Nr)absr)Zfid__)rrrsz*MaxentClassifier.explain..z and label is rz (%s)/,z... z TOTAL:c3s|]}d|VqdS)z%8.3fNr)r5r6)sumsrrr7sz PROBS:c3s|]}d|VqdS)z%8.3fN)prob)r5r6)pdistrrr7s)strr#sortedsamplesrAprintljustjoinrrint enumeraterr(sortrrdescribesplit) rr%columnsZ descr_widthTEMPLATEr!ir)r*r,r-ZscoreZdescrr)rBrr@rexplains>      $zMaxentClassifier.explain csPtdrjd|Sttttjfdddd_jd|SdS)zW Generates the ranked list of informative features from most to least. _most_informative_featuresNcstj|S)N)r:r)fid)rrrr;sz.T)r1r2)hasattrrSrDlistrangerr)rnr)rrmost_informative_featuress   z*MaxentClassifier.most_informative_featuresallcsxd}|dkr&fdd|D}n|dkr@fdd|D}x2|d|D]"}tdj|j|fqNWdS)z :param show: all, neg, or pos (for negative-only or positive-only) :type show: str :param n: The no. of top features :type n: int Nposcsg|]}j|dkr|qS)r)r)r5rT)rrr szCMaxentClassifier.show_most_informative_features..negcsg|]}j|dkr|qS)r)r)r5rT)rrrr\sz%8.3f %s)rYrFrrrL)rrXZshowZfidsrTr)rrshow_most_informative_featuress z/MaxentClassifier.show_most_informative_featurescCsdt|j|jfS)Nz:)rrr!r)rrrr__repr__s zMaxentClassifier.__repr__ZGISZIISZMEGAMZTADMNrc Ks|dkr d}x |D]}|dkrtd|qW|}|dkrPt||||f|S|dkrjt||||f|S|dkrt|||||f|S|dkr|} || d<|| d <|| d <|| d <tj|f| Std |dS) a Train a new maxent classifier based on the given corpus of training samples. This classifier will have its weights chosen to maximize entropy while remaining empirically consistent with the training corpus. :rtype: MaxentClassifier :return: The new maxent classifier :type train_toks: list :param train_toks: Training data, represented as a list of pairs, the first member of which is a featureset, and the second of which is a classification label. :type algorithm: str :param algorithm: A case-insensitive string, specifying which algorithm should be used to train the classifier. The following algorithms are currently available. - Iterative Scaling Methods: Generalized Iterative Scaling (``'GIS'``), Improved Iterative Scaling (``'IIS'``) - External Libraries (requiring megam): LM-BFGS algorithm, with training performed by Megam (``'megam'``) The default algorithm is ``'IIS'``. :type trace: int :param trace: The level of diagnostic tracing output to produce. Higher values produce more verbose output. :type encoding: MaxentFeatureEncodingI :param encoding: A feature encoding, used to convert featuresets into feature vectors. If none is specified, then a ``BinaryMaxentFeatureEncoding`` will be built based on the features that are attested in the training corpus. :type labels: list(str) :param labels: The set of possible labels. If none is given, then the set of all labels attested in the training data will be used instead. :param gaussian_prior_sigma: The sigma value for a gaussian prior on model weights. Currently, this is supported by ``megam``. For other algorithms, its value is ignored. :param cutoffs: Arguments specifying various conditions under which the training should be halted. (Some of the cutoff conditions are not supported by some algorithms.) - ``max_iter=v``: Terminate after ``v`` iterations. - ``min_ll=v``: Terminate after the negative average log-likelihood drops under ``v``. - ``min_lldelta=v``: Terminate if a single iteration improves log likelihood by less than ``v``. NZiis) max_iterZmin_ll min_lldeltaZmax_accZ min_accdelta count_cutoffZnormexplicit bernoullizUnexpected keyword arg %rZgisZmegamZtadmtracerr!gaussian_prior_sigmazUnknown algorithm %s) TypeErrorlower train_maxent_classifier_with_iis train_maxent_classifier_with_gis"train_maxent_classifier_with_megamTadmMaxentClassifiertrain ValueError) cls train_toks algorithmrfrr!rgcutoffsr1kwargsrrrrns0>  zMaxentClassifier.train)T)r.)rR)rRrZ)Nr`NNr)__name__ __module__ __qualname____doc__r r!r"rr&r#rQrYr^r_Z ALGORITHMS classmethodrnrrrrrTs$   ,   rc@s8eZdZdZddZddZddZdd Zd d Zd S) MaxentFeatureEncodingIa A mapping that converts a set of input-feature values to a vector of joint-feature values, given a label. This conversion is necessary to translate featuresets into a format that can be used by maximum entropy models. The set of joint-features used by a given encoding is fixed, and each index in the generated joint-feature vectors corresponds to a single joint-feature. The length of the generated joint-feature vectors is therefore constant (for a given encoding). Because the joint-feature vectors generated by ``MaxentFeatureEncodingI`` are typically very sparse, they are represented as a list of ``(index, value)`` tuples, specifying the value of each non-zero joint-feature. Feature encodings are generally created using the ``train()`` method, which generates an appropriate encoding based on the input-feature values and labels that are present in a given corpus. cCs tdS)aC Given a (featureset, label) pair, return the corresponding vector of joint-feature values. This vector is represented as a list of ``(index, value)`` tuples, specifying the value of each non-zero joint-feature. :type featureset: dict :rtype: list(tuple(int, int)) N)NotImplementedError)rr%r)rrrr(s zMaxentFeatureEncodingI.encodecCs tdS)z :return: The size of the fixed-length joint-feature vectors that are generated by this encoding. :rtype: int N)r{)rrrrrszMaxentFeatureEncodingI.lengthcCs tdS)z :return: A list of the "known labels" -- i.e., all labels ``l`` such that ``self.encode(fs,l)`` can be a nonzero joint-feature vector for some value of ``fs``. :rtype: list N)r{)rrrrr!szMaxentFeatureEncodingI.labelscCs tdS)z :return: A string describing the value of the joint-feature whose index in the generated feature vectors is ``fid``. :rtype: str N)r{)rrTrrrrLszMaxentFeatureEncodingI.describecCs tdS)ao Construct and return new feature encoding, based on a given training corpus ``train_toks``. :type train_toks: list(tuple(dict, str)) :param train_toks: Training data, represented as a list of pairs, the first member of which is a feature dictionary, and the second of which is a classification label. N)r{)rprqrrrrns zMaxentFeatureEncodingI.trainN) rurvrwrxr(rr!rLrnrrrrrzks   rzc@s8eZdZdZddZddZddZdd Zd d Zd S) #FunctionBackedMaxentFeatureEncodingz A feature encoding that calls a user-supplied function to map a given featureset/label pair to a sparse joint-feature vector. cCs||_||_||_dS)ag Construct a new feature encoding based on the given function. :type func: (callable) :param func: A function that takes two arguments, a featureset and a label, and returns the sparse joint feature vector that encodes them:: func(featureset, label) -> feature_vector This sparse joint feature vector (``feature_vector``) is a list of ``(index,value)`` tuples. :type length: int :param length: The size of the fixed-length joint-feature vectors that are generated by this encoding. :type labels: list :param labels: A list of the "known labels" for this encoding -- i.e., all labels ``l`` such that ``self.encode(fs,l)`` can be a nonzero joint-feature vector for some value of ``fs``. N)_length_func_labels)rfuncrr!rrrr sz,FunctionBackedMaxentFeatureEncoding.__init__cCs |||S)N)r~)rr%r)rrrr(sz*FunctionBackedMaxentFeatureEncoding.encodecCs|jS)N)r})rrrrrsz*FunctionBackedMaxentFeatureEncoding.lengthcCs|jS)N)r)rrrrr!sz*FunctionBackedMaxentFeatureEncoding.labelscCsdS)Nzno description availabler)rrTrrrrLsz,FunctionBackedMaxentFeatureEncoding.describeN) rurvrwrxr r(rr!rLrrrrr|s r|c@sHeZdZdZdddZddZddZd d Zd d Ze dddZ dS)BinaryMaxentFeatureEncodinga A feature encoding that generates vectors containing a binary joint-features of the form: | joint_feat(fs, l) = { 1 if (fs[fname] == fval) and (l == label) | { | { 0 otherwise Where ``fname`` is the name of an input-feature, ``fval`` is a value for that input-feature, and ``label`` is a label. Typically, these features are constructed based on a training corpus, using the ``train()`` method. This method will create one feature for each combination of ``fname``, ``fval``, and ``label`` that occurs at least once in the training corpus. The ``unseen_features`` parameter can be used to add "unseen-value features", which are used whenever an input feature has a value that was not encountered in the training corpus. These features have the form: | joint_feat(fs, l) = { 1 if is_unseen(fname, fs[fname]) | { and l == label | { | { 0 otherwise Where ``is_unseen(fname, fval)`` is true if the encoding does not contain any joint features that are true when ``fs[fname]==fval``. The ``alwayson_features`` parameter can be used to add "always-on features", which have the form:: | joint_feat(fs, l) = { 1 if (l == label) | { | { 0 otherwise These always-on features allow the maxent model to directly model the prior probabilities of each label. Fcst|ttt|kr$tdt|_|_t|_d_ d_ |r~t fddt |D_ jtj 7_|rtdd|D}t fddt |D_ jt|7_dS)a :param labels: A list of the "known labels" for this encoding. :param mapping: A dictionary mapping from ``(fname,fval,label)`` tuples to corresponding joint-feature indexes. These indexes must be the set of integers from 0...len(mapping). If ``mapping[fname,fval,label]=id``, then ``self.encode(..., fname:fval, ..., label)[id]`` is 1; otherwise, it is 0. :param unseen_features: If true, then include unseen value features in the generated joint-feature vectors. :param alwayson_features: If true, then include always-on features in the generated joint-feature vectors. zHMapping values must be exactly the set of integers from 0...len(mapping)Nc3s |]\}}||jfVqdS)N)r})r5rPr))rrrr74sz7BinaryMaxentFeatureEncoding.__init__..css|]\}}}|VqdS)Nr)r5fnamefvalr)rrrr79sc3s |]\}}||jfVqdS)N)r})r5rPr)rrrr7;s) setvaluesrWrrorVr_mappingr} _alwayson_unseendictrJ)rr!mappingunseen_featuresalwayson_featuresfnamesr)rrr s"  z$BinaryMaxentFeatureEncoding.__init__cCsg}x|D]z\}}|||f|jkrB||j|||fdfq|jrx>|jD]}|||f|jkrPPqPW||jkr||j|dfqW|jr||jkr||j|df|S)Nr?)itemsrappendrrr)rr%r)rrrlabel2rrrr(?s  z"BinaryMaxentFeatureEncoding.encodecCst|tstdy |jWnHtk rddgt|j|_x |jD]\}}||j|<qJWYnX|t|jkr|j|\}}}d|||fS|jr||j krxr|jD]\}}||krd|SqWnJ|j r||j krx0|j D]\}}||krd|SqWnt ddS)Nzdescribe() expected an intz%s==%r and label is %rz label is %rz %s is unseenzBad feature id) isinstancerrh _inv_mappingAttributeErrorrrrrrrro)rr,inforPrrr)f_id2rrrrLZs(  z$BinaryMaxentFeatureEncoding.describecCs|jS)N)r)rrrrr!ssz"BinaryMaxentFeatureEncoding.labelscCs|jS)N)r})rrrrrwsz"BinaryMaxentFeatureEncoding.lengthrNc Ksi}t}tt}x|D]\}} |r8| |kr8td| || xX|D]L\} } || | fd7<|| | f|krL| | | f|krLt||| | | f<qLWqW|dkr|}|||f|S)a Construct and return new feature encoding, based on a given training corpus ``train_toks``. See the class description ``BinaryMaxentFeatureEncoding`` for a description of the joint-features that will be included in this encoding. :type train_toks: list(tuple(dict, str)) :param train_toks: Training data, represented as a list of pairs, the first member of which is a feature dictionary, and the second of which is a classification label. :type count_cutoff: int :param count_cutoff: A cutoff value that is used to discard rare joint-features. If a joint-feature's value is 1 fewer than ``count_cutoff`` times in the training corpus, then that joint-feature is not included in the generated encoding. :type labels: list :param labels: A list of labels that should be used by the classifier. If not specified, then the set of labels attested in ``train_toks`` will be used. :param options: Extra parameters for the constructor, such as ``unseen_features`` and ``alwayson_features``. zUnexpected label %sr?N)rrrIroaddrr) rprqrcr!optionsr seen_labelscounttokr)rrrrrrn{s   z!BinaryMaxentFeatureEncoding.train)FF)rN) rurvrwrxr r(rLr!rryrnrrrrrs' 3rc@s>eZdZdZdddZeddZdd Zd d Zd d Z dS) GISEncodinga A binary feature encoding which adds one new joint-feature to the joint-features defined by ``BinaryMaxentFeatureEncoding``: a correction feature, whose value is chosen to ensure that the sparse vector always sums to a constant non-negative number. This new feature is used to ensure two preconditions for the GIS training algorithm: - At least one feature vector index must be nonzero for every token. - The feature vector must sum to a constant non-negative number for every token. FNcCs>t||||||dkr4ttdd|Dd}||_dS)a  :param C: The correction constant. The value of the correction feature is based on this value. In particular, its value is ``C - sum([v for (f,v) in encoding])``. :seealso: ``BinaryMaxentFeatureEncoding.__init__`` Ncss|]\}}}|VqdS)Nr)r5rrr)rrrr7sz'GISEncoding.__init__..r?)rr rr_C)rr!rrrCrrrr s zGISEncoding.__init__cCs|jS)zOThe non-negative constant that all encoded feature vectors will sum to.)r)rrrrrsz GISEncoding.CcCsTt|||}t|}tdd|D}||jkr.z&Correction feature is not high enough!)rr(rsumrror)rr%r)rZ base_lengthr+rrrr(s  zGISEncoding.encodecCst|dS)Nr?)rr)rrrrrszGISEncoding.lengthcCs(|t|krd|jSt||SdS)NzCorrection feature (%s))rrrrL)rr,rrrrLs zGISEncoding.describe)FFN) rurvrwrxr propertyrr(rrLrrrrrs    rc@sDeZdZdddZddZddZdd Zd d ZedddZ d S)TadmEventMaxentFeatureEncodingFcCs*t||_t|_t|||j||dS)N)rr_label_mappingrr )rr!rrrrrrr s z'TadmEventMaxentFeatureEncoding.__init__cCsg}x|D]x\}}||f|jkr8t|j|j||f<||jkrht|ts^t|j|j|<n ||j|<||j||f|j|fqW|S)N)rrrrrrIr)rr%r)rfeaturevaluerrrr(s   z%TadmEventMaxentFeatureEncoding.encodecCs|jS)N)r)rrrrr!sz%TadmEventMaxentFeatureEncoding.labelscCs2x,|jD]"\}}|j||f|kr||fSqWdS)N)r)rrTrr)rrrrLsz'TadmEventMaxentFeatureEncoding.describecCs t|jS)N)rr)rrrrr sz%TadmEventMaxentFeatureEncoding.lengthrNc Kst}|sg}t|}x"|D]\}}||kr||qWxH|D]@\}}x6|D].}x(|D] }||f|krXt||||f<qXWqNWq@W|||f|S)N)rrVrr) rprqrcr!rrr%r)rrrrrns   z$TadmEventMaxentFeatureEncoding.train)FF)rN) rurvrwr r(r!rLrryrnrrrrrs rc@sHeZdZdZdddZddZddZd d Zd d Ze dddZ dS)TypedMaxentFeatureEncodingaZ A feature encoding that generates vectors containing integer, float and binary joint-features of the form: Binary (for string and boolean features): | joint_feat(fs, l) = { 1 if (fs[fname] == fval) and (l == label) | { | { 0 otherwise Value (for integer and float features): | joint_feat(fs, l) = { fval if (fs[fname] == type(fval)) | { and (l == label) | { | { not encoded otherwise Where ``fname`` is the name of an input-feature, ``fval`` is a value for that input-feature, and ``label`` is a label. Typically, these features are constructed based on a training corpus, using the ``train()`` method. For string and boolean features [type(fval) not in (int, float)] this method will create one feature for each combination of ``fname``, ``fval``, and ``label`` that occurs at least once in the training corpus. For integer and float features [type(fval) in (int, float)] this method will create one feature for each combination of ``fname`` and ``label`` that occurs at least once in the training corpus. For binary features the ``unseen_features`` parameter can be used to add "unseen-value features", which are used whenever an input feature has a value that was not encountered in the training corpus. These features have the form: | joint_feat(fs, l) = { 1 if is_unseen(fname, fs[fname]) | { and l == label | { | { 0 otherwise Where ``is_unseen(fname, fval)`` is true if the encoding does not contain any joint features that are true when ``fs[fname]==fval``. The ``alwayson_features`` parameter can be used to add "always-on features", which have the form: | joint_feat(fs, l) = { 1 if (l == label) | { | { 0 otherwise These always-on features allow the maxent model to directly model the prior probabilities of each label. Fcst|ttt|kr$tdt|_|_t|_d_ d_ |r~t fddt |D_ jtj 7_|rtdd|D}t fddt |D_ jt|7_dS)a :param labels: A list of the "known labels" for this encoding. :param mapping: A dictionary mapping from ``(fname,fval,label)`` tuples to corresponding joint-feature indexes. These indexes must be the set of integers from 0...len(mapping). If ``mapping[fname,fval,label]=id``, then ``self.encode({..., fname:fval, ...``, label)[id]} is 1; otherwise, it is 0. :param unseen_features: If true, then include unseen value features in the generated joint-feature vectors. :param alwayson_features: If true, then include always-on features in the generated joint-feature vectors. zHMapping values must be exactly the set of integers from 0...len(mapping)Nc3s |]\}}||jfVqdS)N)r})r5rPr))rrrr7sz6TypedMaxentFeatureEncoding.__init__..css|]\}}}|VqdS)Nr)r5rrr)rrrr7sc3s |]\}}||jfVqdS)N)r})r5rPr)rrrr7s) rrrWrrorVrrr}rrrrJ)rr!rrrrr)rrr ^s"  z#TypedMaxentFeatureEncoding.__init__cCsg}x|D]\}}t|ttfrX|t||f|jkr||j|t||f|fq|||f|jkr||j|||fdfq|jrx>|jD]}|||f|jkrPqW||jkr||j|dfqW|j r||j kr||j |df|S)Nr?) rrrfloattyperrrrr)rr%r)rrrrrrrr(s    z!TypedMaxentFeatureEncoding.encodecCst|tstdy |jWnHtk rddgt|j|_x |jD]\}}||j|<qJWYnX|t|jkr|j|\}}}d|||fS|jr||j krxr|jD]\}}||krd|SqWnJ|j r||j krx0|j D]\}}||krd|SqWnt ddS)Nzdescribe() expected an intrz%s==%r and label is %rz label is %rz %s is unseenzBad feature id) rrrhrrrrrrrrro)rr,rrPrrr)rrrrrLs(  z#TypedMaxentFeatureEncoding.describecCs|jS)N)r)rrrrr!sz!TypedMaxentFeatureEncoding.labelscCs|jS)N)r})rrrrrsz!TypedMaxentFeatureEncoding.lengthrNc Ksi}t}tt}x|D]\}} |r8| |kr8td| || xp|D]d\} } t| ttfkrlt| } || | fd7<|| | f|krL| | | f|krLt||| | | f<qLWqW|dkr|}|||f|S)a) Construct and return new feature encoding, based on a given training corpus ``train_toks``. See the class description ``TypedMaxentFeatureEncoding`` for a description of the joint-features that will be included in this encoding. Note: recognized feature values types are (int, float), over types are interpreted as regular binary features. :type train_toks: list(tuple(dict, str)) :param train_toks: Training data, represented as a list of pairs, the first member of which is a feature dictionary, and the second of which is a classification label. :type count_cutoff: int :param count_cutoff: A cutoff value that is used to discard rare joint-features. If a joint-feature's value is 1 fewer than ``count_cutoff`` times in the training corpus, then that joint-feature is not included in the generated encoding. :type labels: list :param labels: A list of labels that should be used by the classifier. If not specified, then the set of labels attested in ``train_toks`` will be used. :param options: Extra parameters for the constructor, such as ``unseen_features`` and ``alwayson_features``. zUnexpected label %sr?N) rrrIrorrrrr) rprqrcr!rrrrrr)rrrrrrns"   z TypedMaxentFeatureEncoding.train)FF)rN) rurvrwrxr r(rLr!rryrnrrrrr%s7 3 rr`cKs|ddt|}|dkr*tj||d}t|ds Training (%d iterations)r0z- Iteration Log Likelihood Accuracyz- ---------------------------------------z %9d %14.5f %9.3fr?z* Training stopped: keyboard interruptz! Final %14.5f %9.3f) setdefaultr rrnrUrhrcalculate_empirical_fcountrnumpynonzerozerosrNINF ConditionalExponentialClassifierlog2rFllr accr itercalculate_estimated_fcountrr"checkKeyboardInterrupt)rqrfrr!rs cutoffcheckerZCinvZempirical_fcount unattestedrrT classifierZlog_empirical_fcountrriternumZestimated_fcountZlog_estimated_fcountrrrrksb                  rkcCsPt|d}x:|D]2\}}x(|||D]\}}|||7<q,WqW|S)Nr)rrrr()rqrfcountrr)indexvalrrrrrs rc Cszt|d}xd|D]\\}}||}xH|D]<}||}x,|||D]\}} |||| 7<qNWq2WqW|S)Nr)rrrr#rErAr() rrqrrrr)rBrArTrrrrr|s   rc Ks|ddt|}|dkr*tj||d}t||t|}t||}tt ||j dd}t |t|df} t t |dkd} tt|d} x| D]} tj| | <qWt|| } |dkrtd |d|d krttd td yx|d kr8|jp t| |}|jpt| |}|j}td |||ft|| | |||| |}| } | |7} | | || |rPqWWn*tk rtdYnYnX|d krt| |}t| |}td||f| S)a Train a new ``ConditionalExponentialClassifier``, using the given training samples, using the Improved Iterative Scaling algorithm. This ``ConditionalExponentialClassifier`` will encode the model that maximizes entropy from all the models that are empirically consistent with ``train_toks``. :see: ``train_maxent_classifier()`` for parameter descriptions. rarN)r!)r1rr?rz ==> Training (%d iterations)r0z- Iteration Log Likelihood Accuracyz- ---------------------------------------z %9d %14.5f %9.3fz* Training stopped: keyboard interruptz! Final %14.5f %9.3f)rr rrnrrcalculate_nfmaprZarrayrD __getitem__ZreshaperrrrrrFrr rr rcalculate_deltasrr"rr)rqrfrr!rsrZempirical_ffreqnfmapnfarray nftransposerrrTrrrrdeltasrrrrjsb            rjc Cs`t}xB|D]:\}}x0|D]$}|tdd|||DqWq Wtddt|DS)a Construct a map that can be used to compress ``nf`` (which is typically sparse). *nf(feature_vector)* is the sum of the feature values for *feature_vector*. This represents the number of features that are active for a given labeled text. This method finds all values of *nf(t)* that are attested for at least one token in the given list of training tokens; and constructs a dictionary mapping these attested values to a continuous range *0...N*. For example, if the only values of *nf()* that were attested were 3, 5, and 7, then ``_nfmap`` might return the dictionary ``{3:0, 5:1, 7:2}``. :return: A map that can be used to compress ``nf`` to a dense vector. :rtype: dict(int -> int) css|]\}}|VqdS)Nr)r5idrrrrr7sz"calculate_nfmap..css|]\}}||fVqdS)Nr)r5rPnfrrrr7s)rr!rrr(rrJ)rqrZnfsetr_r)rrrrs (rc Cspd}d} t|d} tt||fd} x~|D]v\} } || }xb|D]V} || | }tdd|D}x2|D]*\}}| |||f| | |7<qzWqRWq6W| t|} xt | D]}t || }d|}||}tj|| dd}tj|| dd}x|D]}||d 7<qW| ||| 8} tt ||tt | }||kr| SqW| S) a Calculate the update values for the classifier weights for this iteration of IIS. These update weights are the value of ``delta`` that solves the equation:: ffreq_empirical[i] = SUM[fs,l] (classifier.prob_classify(fs).prob(l) * feature_vector(fs,l)[i] * exp(delta[i] * nf(feature_vector(fs,l)))) Where: - *(fs,l)* is a (featureset, label) tuple from ``train_toks`` - *feature_vector(fs,l)* = ``encoding.encode(fs,l)`` - *nf(vector)* = ``sum([val for (id,val) in vector])`` This method uses Newton's method to solve this equation for *delta[i]*. In particular, it starts with a guess of ``delta[i]`` = 1; and iteratively updates ``delta`` with: | delta[i] -= (ffreq_empirical[i] - sum1[i])/(-sum2[i]) until convergence, where *sum1* and *sum2* are defined as: | sum1[i](delta) = SUM[fs,l] f[i](fs,l,delta) | sum2[i](delta) = SUM[fs,l] (f[i](fs,l,delta).nf(feature_vector(fs,l))) | f[i](fs,l,delta) = (classifier.prob_classify(fs).prob(l) . | feature_vector(fs,l)[i] . | exp(delta[i] . nf(feature_vector(fs,l)))) Note that *sum1* and *sum2* depend on ``delta``; so they need to be re-computed each iteration. The variables ``nfmap``, ``nfarray``, and ``nftranspose`` are used to generate a dense encoding for *nf(ltext)*. This allows ``_deltas`` to calculate *sum1* and *sum2* using matrices, which yields a significant performance improvement. :param train_toks: The set of training tokens. :type train_toks: list(tuple(dict, str)) :param classifier: The current classifier. :type classifier: ClassifierI :param ffreq_empirical: An array containing the empirical frequency for each feature. The *i*\ th element of this array is the empirical frequency for feature *i*. :type ffreq_empirical: sequence of float :param unattested: An array that is 1 for features that are not attested in the training data; and 0 for features that are attested. In other words, ``unattested[i]==0`` iff ``ffreq_empirical[i]==0``. :type unattested: sequence of int :param nfmap: A map that can be used to compress ``nf`` to a dense vector. :type nfmap: dict(int -> int) :param nfarray: An array that can be used to uncompress ``nf`` from a dense vector. :type nfarray: array(float) :param nftranspose: The transpose of ``nfarray`` :type nftranspose: array(float) g-q=i,rcss|]\}}|VqdS)Nr)r5rrrrrr7`sz#calculate_deltas..r0r)Zaxisr?) rZonesrrrr#r!r(rrArWZouterr:)rqrrZffreq_empiricalrrrrZNEWTON_CONVERGEZ MAX_NEWTONrArr)Zdistr*rrrZrangenumZnf_deltaZ exp_nf_deltaZnf_exp_nf_deltaZsum1Zsum2rTZn_errorrrrrs2I  .    rc Ks&d}d}d|kr|d}d|kr(|d}|dkrP|dd}tj|||dd}n|dk r`tdyFtjd d \} } t| d } t||| ||d WdQRXt | Wn4t t tfk r} ztd | Wdd} ~ XYnXg} | dddg7} |r| dg7} |s | dg7} |r d|d}nd}| dd|dg7} |dkrJ| dg7} d|krh| dd|dg7} d|kr| ddt |dg7} t |dr| d g7} | d!| g7} t| }yt| Wn8t t fk r} ztd"| | fWdd} ~ XYnXt|||}|ttj9}t||S)#a Train a new ``ConditionalExponentialClassifier``, using the given training samples, using the external ``megam`` library. This ``ConditionalExponentialClassifier`` will encode the model that maximizes entropy from all the models that are empirically consistent with ``train_toks``. :see: ``train_maxent_classifier()`` for parameter descriptions. :see: ``nltk.classify.megam`` TrdreNrcr)r!rz$Specify encoding or labels, not bothznltk-)prefixw)rdrez,Error while creating megam training file: %sz-nobiasz-repeatZ10z -explicitz-fvalsg?r0z-lambdaz%.2fz-tuner`z-quietraz-maxiz%sll_deltaz-dppZcostz -multilabelZ multiclassz Warning: unable to delete %s: %s)getrrnrotempfilemkstempopenroscloseOSErrorIOErrorr:rUrremoverFrrrrer)rqrfrr!rgrtrdrercfdtrainfile_name trainfilerrZ inv_variancestdoutrrrrrls^           "rlc@seZdZeddZdS)rmc Ks|dd}|dd}|dd}|dd}|dd}|d d}|d } |d } |sptj|||d }tjd dd\} } tjdd\} }t| d}t||||g}|dg|d|g|r|dd|dg| r|dd| g| r|ddt | g|d| g|d|g|dkrN|dgn |dgt |t |d}t |}WdQRXt | t ||ttj9}|||S) NrrZtao_lmvmrfr`rr!rgrrcrarb)r!znltk-tadm-events-z.gz)rsuffixznltk-tadm-weights-)rrz-monitorz-methodz-l2z%.6fr0z-max_itz%dz-fatolz -events_inz -params_outz2>&1z-summaryr)rrrnrrrrrextendr:rrrrrrrr)rprqrtrrrfrr!ZsigmarcrarZ trainfile_fdrZ weightfile_fdZweightfile_namerrZ weightfilerrrrrnsL                 zTadmMaxentClassifier.trainN)rurvrwryrnrrrrrmsrmcCsddlm}|tj}dS)Nr) names_demo)nltk.classify.utilrrrn)rrrrrdemo%s r__main__)r`NN)r`NN)r`NNr)7rxZ __future__rrr ImportErrorrr collectionsrZsixrZnltkrZ nltk.datarZ nltk.utilrZnltk.probabilityr Znltk.classify.apir rr r r Znltk.classify.megamrrrZnltk.classify.tadmrrrZ __docformat__Zpython_2_unicode_compatiblerrobjectrzr|rrrrrkrrrjrrrlrmrrurrrr5sV        I/N=8m a  [ [=