p7]c@sdZddlZddlmZmZmZmZmZmZm Z m Z m Z m Z m Z ddlmZmZmZddlmZddlZddlZddlZddljjZddlmZddlmZddl m!Z!d d d d d dddgZ"dZ#dZ$de%fdYZ&de%fdYZ'dZ(dZ)dZ*dZ+dZ,dZ-dZ.iejd6e/d6Z0d e%fd!YZ1d"e2e3e4d#Z5e2d$d%d&Z6dS('s Input/Output tools for working with binary data. The Stata input tools were originally written by Joe Presbrey as part of PyDTA. You can find more information here http://presbrey.mit.edu/PyDTA See Also -------- numpy.lib.io iN( tziptlziptlmaptlranget string_typestlongtlfiltertasbytestasstrtrangetPY3(tunpacktcalcsizetpack(terror(tisnull(tStataMissingValue(t get_file_objs%tcs%tCs%tds%tws%tms%tqs%ths%tycCst|tjs!tdntjddd}|d"krm||}|jd|jd|jdS|d#krd d lm}|d t|S|d$kr||jS|d%krd|j |j |tj|j ddjdS|d&krd|j |j |j dS|d'krLd|j |j t |j ddS|d(kr}d|j |j t |j dkS|d)kr|j Std |d!S(*s# Convert from datetime to SIF. http://www.stata.com/help.cgi?datetime Parameters ---------- date : datetime.datetime The date to convert to the Stata Internal Format given by fmt fmt : str The format to convert to. Can be, tc, td, tw, tm, tq, th, ty s'date should be datetime.datetime formatiis%tcttci\&is%tCttCi(twarns'Stata Internal Format tC not supported.s%tdttds%twttwi4is%tmttmi s%tqttqiis%thtthiis%tyttysfmt %s not understoodN(s%tcR(s%tCR(s%tdR(s%twR(s%tmR(s%tqR(s%thR(s%tyR( t isinstancetdatetimet ValueErrortdaystsecondst microsecondstwarningsRt UserWarningtyeartmonthtint(tdatetfmtt stata_epochtdeltaR((s8lib/python2.7/site-packages/statsmodels/iolib/foreign.pyt_datetime_to_stata_elapseds2         "  ' % cCst|}tjddd}|d$krRddlm}||d|dS|d%krdd lm}|d t|S|d&kr|tjt|S|d'krtj|j|ddd}|dd}|tjt|S|d(kr1|j|d}|dd}tj||dS|d)krs|j|d}|ddd}tj||dS|d*kr|j|d}|ddd}tj||dS|d+kr|d krtj|ddSt d!nt d"|d#S(,s Convert from SIF to datetime. http://www.stata.com/help.cgi?datetime Parameters ---------- date : int The Stata Internal Format date to convert to datetime according to fmt fmt : str The format to convert to. Can be, tc, td, tw, tm, tq, th, ty Examples -------- >>> _stata_elapsed_date_to_datetime(52, "%tw") datetime.datetime(1961, 1, 1, 0, 0) Notes ----- datetime/c - tc milliseconds since 01jan1960 00:00:00.000, assuming 86,400 s/day datetime/C - tC - NOT IMPLEMENTED milliseconds since 01jan1960 00:00:00.000, adjusted for leap seconds date - td days since 01jan1960 (01jan1960 = 0) weekly date - tw weeks since 1960w1 This assumes 52 weeks in a year, then adds 7 * remainder of the weeks. The datetime value is the start of the week in terms of days in the year, not ISO calendar weeks. monthly date - tm months since 1960m1 quarterly date - tq quarters since 1960q1 half-yearly date - th half-years since 1960h1 yearly date - ty years since 0000 If you don't have pandas with datetime support, then you can't do milliseconds accurately. iis%tcRi(t relativedeltaR is%tCR(Rs9Encountered %tC format. Leaving in Stata Internal Format.s%tdRs%twRi4is%tmRi s%tqRiis%thRiis%tyRis!Year 0 and before not implementedsDate fmt %s not understoodN(s%tcR(s%tCR(s%tdR(s%twR(s%tmR(s%tqR(s%thR(s%tyR( R%Rtdateutil.relativedeltaR+R!RR"t timedeltaR#R(R&R'R(R+RR#t day_deltat month_delta((s8lib/python2.7/site-packages/statsmodels/iolib/foreign.pyt_stata_elapsed_date_to_datetimeCsB-           t_StataVariablecBseZdZdZdZdZedddZedddZed dd Z ed dd Z ed ddZ edddZ eje_e je_RS(s' A dataset variable. Not intended for public use. Parameters ---------- variable_data Attributes ---------- format : str Stata variable format. See notes for more information. index : int Zero-index column index of variable. label : str Data Label name : str Variable name type : str Stata data type. See notes for more information. value_format : str Value format. Notes ----- More information: http://www.stata.com/help.cgi?format cCs ||_dS(N(t_data(tselft variable_data((s8lib/python2.7/site-packages/statsmodels/iolib/foreign.pyt__init__scCs|jS(N(tindex(R3((s8lib/python2.7/site-packages/statsmodels/iolib/foreign.pyt__int__scCs|jS(N(tname(R3((s8lib/python2.7/site-packages/statsmodels/iolib/foreign.pyt__str__scCs |jdS(Ni(R2(R3((s8lib/python2.7/site-packages/statsmodels/iolib/foreign.pytttdocs*the variable's index within an observationcCs |jdS(Ni(R2(R3((s8lib/python2.7/site-packages/statsmodels/iolib/foreign.pyR:R;shthe data type of variable Possible types are: {1..244:string, b:byte, h:int, l:long, f:float, d:double)cCs |jdS(Ni(R2(R3((s8lib/python2.7/site-packages/statsmodels/iolib/foreign.pyR:R;sthe name of the variablecCs |jdS(Ni(R2(R3((s8lib/python2.7/site-packages/statsmodels/iolib/foreign.pyR:R;sthe variable's Stata formatcCs |jdS(Ni(R2(R3((s8lib/python2.7/site-packages/statsmodels/iolib/foreign.pyR:R;sthe variable's value formatcCs |jdS(Ni(R2(R3((s8lib/python2.7/site-packages/statsmodels/iolib/foreign.pyR:R;sthe variable's label( t__name__t __module__t__doc__R5R7R9tpropertyR6ttypeR8tformatt value_formattlabel(((s8lib/python2.7/site-packages/statsmodels/iolib/foreign.pyR1s             t StataReadercBseZdZiZdZd)ZeZeZe e e ddge ddD]Z dee ^qLdejfdejfdefdejfd ejfgZe ded Zid*d 6d+d6d,d6dd fd6dd fd6Zed-dZdZdZdZdZdZed Z d!Z!d"Z"d#Z#d$Z$d%Z%d-d&Z&d'Z'd(Z(RS(.sJ Stata .dta file reader. Provides methods to return the metadata of a Stata .dta file and a generator for the data itself. Parameters ---------- file : file-like A file-like object representing a Stata .dta file. missing_values : bool If missing_values is True, parse missing_values and return a Missing Values object instead of None. encoding : string, optional Used for Python 3 only. Encoding to use when reading the .dta file. Defaults to `locale.getpreferredencoding` See Also -------- statsmodels.iolib.foreign.genfromdta pandas.read_stata pandas.io.stata.StataReader Notes ----- This is known only to work on file formats 113 (Stata 8/9), 114 (Stata 10/11), and 115 (Stata 12). Needs to be tested on older versions. Known not to work on format 104, 108. If you have the documentation for older formats, please contact the developers. For more information about the .dta format see http://www.stata.com/help.cgi?dta http://www.stata.com/help.cgi?dta_113 iiitaiiiiitbhlfdiidtbiithiitlg_g_GtfggYtdcCs]tjdt|dkr:ddl}|j|_n ||_||_|j|dS(NsStataReader is deprecated as of 0.10.0 and will be removed in a future version. Use pandas.read_stata or pandas.io.stata.StataReader instead.i( R!Rt FutureWarningtNonetlocaletgetpreferredencodingt _encodingt_missing_valuest _parse_header(R3tfnametmissing_valuestencodingRO((s8lib/python2.7/site-packages/statsmodels/iolib/foreign.pyR5 s    cCs|jS(s Returns all .dta file headers. out: dict Has keys typlist, data_label, lbllist, varlist, nvar, filetype, ds_format, nobs, fmtlist, vlblist, time_stamp, srtlist, byteorder (t_header(R3((s8lib/python2.7/site-packages/statsmodels/iolib/foreign.pyt file_headersscCs |jdS(s Returns the file format. Returns ------- out : int Notes ----- Format 113: Stata 8/9 Format 114: Stata 10/11 Format 115: Stata 12 t ds_format(RW(R3((s8lib/python2.7/site-packages/statsmodels/iolib/foreign.pyt file_format$scCs |jdS(sc Returns the dataset's label. Returns ------- out: string t data_label(RW(R3((s8lib/python2.7/site-packages/statsmodels/iolib/foreign.pyt file_label4scCs |jdS(s Returns the date and time Stata recorded on last file save. Returns ------- out : str t time_stamp(RW(R3((s8lib/python2.7/site-packages/statsmodels/iolib/foreign.pytfile_timestamp>sc Cs\tttt|jd|jd|jd|jd|jd|jd|jdS(sI Returns a list of the dataset's StataVariables objects. tnvarttyplisttvarlisttsrtlisttfmtlisttlbllisttvlblist(RR1RRRW(R3((s8lib/python2.7/site-packages/statsmodels/iolib/foreign.pyt variablesHs  ccsy|jj|jWntk r*nX|rtt|j}x`tt|D] }t t ||j VqYWn)x&t|j dD]}|j VqWdS(s Returns a Python generator object for iterating over the dataset. Parameters ---------- as_dict : bool, optional If as_dict is True, yield each row of observations as a dict. If False, yields each row of observations as a list. Returns ------- Generator object for iterating over the dataset. Yields each row of observations as a list by default. Notes ----- If missing_values is True during instantiation of StataReader then observations with StataMissingValue(s) are not filtered and should be handled by your applcation. tnobsN( t_filetseekt_data_locationt ExceptionRtstrRfR tlentdictRt_nextRW(R3tas_dicttvarsti((s8lib/python2.7/site-packages/statsmodels/iolib/foreign.pytdatasetRs !cCs |jdS(s Return the number of observations in the dataset. This value is taken directly from the header and includes observations with missing values. Rg(RW(R3((s8lib/python2.7/site-packages/statsmodels/iolib/foreign.pyt__len__xscCst|ttf s8|dks8|t|dkrGt|n|jt|j|}|jj |kr|jj |n|j S(s Seek to an observation indexed k in the file and return it, ordered by Stata's output to the .dta file. k is zero-indexed. Prefer using R.data() for performance. ii( RR%RRmt IndexErrorRjtsumt _col_sizeRhttellRiRo(R3tktloc((s8lib/python2.7/site-packages/statsmodels/iolib/foreign.pyt __getitem__s 8cCstrItd}y |j||j| }WnnX|j|Std}y|j||j| SWn|SXdS(Ns(R RtlstripR6tdecode(R3tsRVt null_byte((s8lib/python2.7/site-packages/statsmodels/iolib/foreign.pyt_null_terminates    c Cs9||_|j}td|jjdd|jd<|jdd#kretd|jdntd|jjdddkrd pd }|jd gt|D]$}|j|jjd|^q|jd= 113 (Stata >= 9) are supported. Got format %s. Please report if you think this error is incorrect.t>t;;; ##  cCs*t|tr|p)t|jd|S(NR(RR%R RW(R3R'((s8lib/python2.7/site-packages/statsmodels/iolib/foreign.pyt _calcsizescs\tjdkr:tfdjd_n|dkrMjSj|SdS(s Calculate size of a data record.ics j|S(N(R(R(R3(s8lib/python2.7/site-packages/statsmodels/iolib/foreign.pyR:R;R`N(Rmt _col_sizesRRWRN(R3Ry((R3s8lib/python2.7/site-packages/statsmodels/iolib/foreign.pyRws  cCst|jd||d}|d|jkr|j|d\}}||ks`||kr|jrvt||SdSqn|S(NRii(R RWtMISSING_VALUESRRRRN(R3R'tbytRLtnmintnmax((s8lib/python2.7/site-packages/statsmodels/iolib/foreign.pyt_unpacks   csjdjrdgjd}xtt|D]t}t|trjjj |j ||R?RWRjRtFalseRRRRnRRR RrRltnptint16tint32R%tfloat32tfloat64RtlistRRRNR5RXRZR\R^RfRsRtR{RRSRRwRRo(((s8lib/python2.7/site-packages/statsmodels/iolib/foreign.pyREs8">!"  &   ?  cCs@|jdkrdS|jdkr,dStd|dS(NRtlittleRtbigsEndianness %s not understood(RR(RR(tlowerR(t endianness((s8lib/python2.7/site-packages/statsmodels/iolib/foreign.pyt_set_endianness s cCs|jtjkrt|jS|jtjkr;tdS|tjkrTtdS|tjkrmtdS|tjkrtdS|tj krtdS|tj ks|tj krtdSt d|dS( s Converts dtype types to stata types. Returns the byte of the given ordinal. See TYPE_MAP and comments for an explanation. This is also explained in the dta spec. 1 - 244 are strings of this length 251 - chr(251) - for int8 and int16, byte 252 - chr(252) - for int32, int 253 - chr(253) - for int64, long 254 - chr(254) - for float32, float 255 - chr(255) - double, double If there are dates to convert, then dtype will already have the correct type inserted. iiiiiisPData type %s not currently understood. Please report an error to the developers.N( RARtstring_tchrtitemsizetobject_RRtint64Rtint8RR(tdtype((s8lib/python2.7/site-packages/statsmodels/iolib/foreign.pyt_dtype_to_stata_types        cCs|jtjkr'dt|jdS|jtjkr=dS|tjkrPdS|tjkrcdS|tjkrvdS|tj krdS|tj ks|tj krdSt d|dS( sY Maps numpy dtype to stata's default format for this type. Not terribly important since users can change this in Stata. Semantics are string -> "%DDs" where DD is the length of the string float64 -> "%10.0g" float32 -> "%9.0g" int64 -> "%9.0g" int32 -> "%9.0g" int16 -> "%9.0g" int8 -> "%8.0g" t%R~s%244ss%10.0gs%9.0gs%8.0gsPData type %s not currently understood. Please report an error to the developers.N( RARRRlRRRRRRRRR(R((s8lib/python2.7/site-packages/statsmodels/iolib/foreign.pyt_dtype_to_default_stata_fmt5s cCs|d|t|S(sP Takes a char string and pads it wih null bytes until it's length chars s(Rm(R8tlength((s8lib/python2.7/site-packages/statsmodels/iolib/foreign.pyt _pad_bytesUscCs(gtd|dD]}d|^qS(s7 Returns default Stata names v1, v2, ... vnvar isv%d(R (R_Rr((s8lib/python2.7/site-packages/statsmodels/iolib/foreign.pyt_default_names[scCs'|dkrtjStd|dS(sK Converts from one of the stata date formats to a type in TYPE_MAP Rs%tcRs%tdRs%twRs%tmRs%tqRs%thRs%tysfmt %s not understoodN(Rs%tcRs%tdRs%twRs%tmRs%tqRs%thRs%ty(RRR(R'((s8lib/python2.7/site-packages/statsmodels/iolib/foreign.pyt_convert_datetime_to_stata_typeas cCsi}x|D]}||jds;d||||", "<", "little", or "big". The default is None which uses `sys.byteorder` Returns ------- writer : StataWriter instance The StataWriter instance has a write_file method, which will write the file to the given `fname`. Examples -------- >>> writer = StataWriter('./data_file.dta', data) >>> writer.write_file() Or with dates >>> writer = StataWriter('./date_data_file.dta', date, {2 : 'tw'}) >>> writer.write_file() iiRFiiiiiRGieRHiRIiRJgGRKgRLslatin-1cCstjdt||_tj|dr;|j|ntj|drt j |}tj |r{|j |q|dk rt dn|j|nt dt||dkrtj}nt||_||_t|d||_dS(NsStataWriter is deprecated as of 0.10.0 and will be removed in a future version. Use pandas.DataFrame.to_stata or pandas.io.stata.StatWriter instead.s-Not able to convert dates in a plain ndarray.sType %s for data not understoodtwb(R!RRMt_convert_datest data_utilt_is_using_pandasRNt_prepare_pandast_is_array_likeRtasarrayt_is_structured_ndarrayt_prepare_structured_arrayRt_prepare_ndarrayRAtsysRRt _byteorderRQRRh(R3RTRRRVR((s8lib/python2.7/site-packages/statsmodels/iolib/foreign.pyR5s&     cCs|jjt|dS(sT Helper to call asbytes before writing to file for Python 3 compat. N(RhtwriteR(R3tto_write((s8lib/python2.7/site-packages/statsmodels/iolib/foreign.pyt_writescCst||_t|j|_||_t||_|j}|j}|jdkrlt |j}n |j}|j }|dk rt ||}||_ x0|D](}||dt ||f||R5R((R(s8lib/python2.7/site-packages/statsmodels/iolib/foreign.pyRs (tobjectt reset_indexRRRgR_RtcolumnsttolistRatdtypesRRNRRRRRR`RRc(R3RRRRRtnew_typetdt((Rs8lib/python2.7/site-packages/statsmodels/iolib/foreign.pyRs(        ""  cCs^|j|j|j|jtdd|jdkrP|jn |jdS(NR;i( t _write_headert_write_descriptorst_write_variable_labelsRRRRNt_write_data_nodatest_write_data_dates(R3((s8lib/python2.7/site-packages/statsmodels/iolib/foreign.pyt write_file0s    cCs\|j}|jtdd|j|dkr7dp:d|jd|jd|jt|d|jd |jt|d |jd |dkr|j|jtd d |jn)|j|jt|d d |j|dkrt j j }nt |t s3t d n|j|j|j d|jdS(NRHirRsssRIiRriR;iPs"time_stamp should be datetime types%d %b %Y %H:%M(RRR R_RgRNRRRQRtnowRRtstrftime(R3R[R]R((s8lib/python2.7/site-packages/statsmodels/iolib/foreign.pyR<s&   !!    c Cs|j}x|jD]}|j|qWxL|jD]A}|j||j}tt|d d}|j|q4Wtdd|d}|j|x'|jD]} |jt| dqWx*t |D]} |jtddqWdS(Ni i!R;iii1( R_R`RRaRRQRRRcR ( R3R`RaRbRcRdR_RR8R'Rr((s8lib/python2.7/site-packages/statsmodels/iolib/foreign.pyR[s  cCsI|j}|dkrEx-t|D]}|jtddq"WndS(NR;iQ(R_RNR RR(R3tlabelsR_Rr((s8lib/python2.7/site-packages/statsmodels/iolib/foreign.pyRus  c Cs3|j}|j}|j}|j}x|D]}xt|D]\}}t||}|dkrt||krtt|t|d}n|j |q>yA|t krt ||}n|j t ||||Wq>t k r&|j t |||t ||q>Xq>Wq+WdS(Nii( RRRR`t enumerateRRmRRRt_type_convertersR t struct_error( R3RRRR`RRrtvarR((s8lib/python2.7/site-packages/statsmodels/iolib/foreign.pyR{s$      " " c Cs3|j}|j}|j}|j}|j}|j}x|D]}xt|D]\}} t||} ||krt| |j |} n| dkrt | rd} nt | | krt | t | d} n|j | qPt | r || } n|j t||| | qPWq=WdS(NiR;i(RRRRRR`RRR*RcRRmRRR ( R3RRRRRR`RRrRR((s8lib/python2.7/site-packages/statsmodels/iolib/foreign.pyRs(             cCs5d}tr#||7}|j|S||7}|SdS(Ns(R tencode(R3R~RVR((s8lib/python2.7/site-packages/statsmodels/iolib/foreign.pyRs    N( R=R>R?RnRRR RrRlRRRR%RRRRRRRNR5RRRRRRRRRRR(((s8lib/python2.7/site-packages/statsmodels/iolib/foreign.pyR{s0$>!"    $     g8cstjdtt|trCtt|ddtd|}n@t|dskt dt |nt|dtd|}|j }|d}|d}|d } |d } |d |d } |d } t j || f} |j}t jt| |}t j |d|} xot|D]a\}}d|krx6t|D]%\}}|dkrU|||The input should be a string or a filehandle. (got %s instead)RRgR_RaRcR[ReRi(t DataFramecSs |tkS(N(t _date_formats(R((s8lib/python2.7/site-packages/statsmodels/iolib/foreign.pyR:R;itargscSs |tkS(N(R(R((s8lib/python2.7/site-packages/statsmodels/iolib/foreign.pyR:R;cst|S(N(R0(R(tcolRc(s8lib/python2.7/site-packages/statsmodels/iolib/foreign.pytconvert sN( R!RRMRRREtopenRthasattrt TypeErrorRARXRtzerosRsRRRRNttupletpandasRt from_recordstwhereRRtapplyR0RRtastypeR(RTt missing_fltRVRRtfhdtheaderttypesRgtnumvarstvarnamestdatanameRRt stata_dtaRtrownumtlineRrtvalRtcolsRt sub_dtypeR((RRcs8lib/python2.7/site-packages/statsmodels/iolib/foreign.pyt genfromdtasb               > ,s%.18et c Cst|d}tj|}|jdkrr|jjdkr]tj|j}d}qt |jj }n |j d}t |t tfrt ||krtdt|n|j|}ntt |trE|jddkr|g|}|j|}qE|jd|kr<td|qE|}n|dkrl|jjrl|jj}n|dk r|j|j|dnx)|D]!}|j|t|dqWWdQXdS(s Save an array to a text file. This is just a copy of numpy.savetxt patched to support structured arrays or a header of names. Does not include py3 support now in savetxt. Parameters ---------- fname : filename or file handle If the filename ends in ``.gz``, the file is automatically saved in compressed gzip format. `loadtxt` understands gzipped files transparently. X : array_like Data to be saved to a text file. names : list, optional If given names will be the column header in the text file. If None and X is a structured or recarray then the names are taken from X.dtype.names. fmt : str or sequence of strs A single format (%10.5f), a sequence of formats, or a multi-format string, e.g. 'Iteration %d -- %10.5f', in which case `delimiter` is ignored. delimiter : str Character separating columns. See Also -------- save : Save an array to a binary file in NumPy ``.npy`` format savez : Save several arrays into a ``.npz`` compressed archive Notes ----- Further explanation of the `fmt` parameter (``%[flag]width[.precision]specifier``): flags: ``-`` : left justify ``+`` : Forces to preceed result with + or -. ``0`` : Left pad the number with zeros instead of space (see width). width: Minimum number of characters to be printed. The value is not truncated if it has more characters. precision: - For integer specifiers (eg. ``d,i,o,x``), the minimum number of digits. - For ``e, E`` and ``f`` specifiers, the number of digits to print after the decimal point. - For ``g`` and ``G``, the maximum number of significant digits. - For ``s``, the maximum number of characters. specifiers: ``c`` : character ``d`` or ``i`` : signed decimal integer ``e`` or ``E`` : scientific notation with ``e`` or ``E``. ``f`` : decimal floating point ``g,G`` : use the shorter of ``e,E`` or ``f`` ``o`` : signed octal ``s`` : string of characters ``u`` : unsigned decimal integer ``x,X`` : unsigned hexadecimal integer This explanation of ``fmt`` is not complete, for an exhaustive specification see [1]_. References ---------- .. [1] `Format Specification Mini-Language `_, Python Documentation. Examples -------- >>> savetxt('test.out', x, delimiter=',') # x is an array >>> savetxt('test.out', (x,y,z)) # x,y,z equal sized 1D arrays >>> savetxt('test.out', x, fmt='%1.4e') # use exponential notation twisfmt has wrong shape. %sRs'fmt has wrong number of %% formats. %ss N(RRRRRRRNt atleast_2dtTRmRRRRRtAttributeErrorRltjoinRtcountR( RTtXRR't delimitertfhtncolRBR((s8lib/python2.7/site-packages/statsmodels/iolib/foreign.pytsavetxts4[       (7R?R!tstatsmodels.compat.pythonRRRRRRRRRR R tstructR R R RRRRtnumpyRtstatsmodels.tools.datattoolsRRRRtpandas.io.stataRtstatsmodels.iolib.openfileRRR*R0RR1RERRRRRRRR%RRRNRRRR(((s8lib/python2.7/site-packages/statsmodels/iolib/foreign.pyt s< L    & T1B  $   >  Z