ó šxŠ\c@sddZddlmZddlZddlZddlZddlmZddlZ ddlm Z ddl m Z m Z dZdZd Zd Zd d d dddddddddddddgZdZdZdZdZdied 6ed!6ed"6ed#6Zd$ied 6ed"6Zd%Zd&„Zd'„Zd(„Zd)„Zd*e fd+„ƒYZdS(,sË Read a SAS XPort format file into a Pandas DataFrame. Based on code from Jack Cushman (github.com/jcushman/xport). The file format is defined here: https://support.sas.com/techsup/technote/ts140.pdf iÿÿÿÿ(tdatetimeN(tAppender(tcompat(t BaseIteratortget_filepath_or_buffersPHEADER RECORD*******LIBRARY HEADER RECORD!!!!!!!000000000000000000000000000000 sKHEADER RECORD*******MEMBER HEADER RECORD!!!!!!!000000000000000001600000000sPHEADER RECORD*******DSCRPTR HEADER RECORD!!!!!!!000000000000000000000000000000 sPHEADER RECORD*******OBS HEADER RECORD!!!!!!!000000000000000000000000000000 tntypetnhfunt field_lengthtnvar0tnametlabeltnformtnflt num_decimalstnfjtnfilltniformtnifltnifdtnpost_s…Parameters ---------- filepath_or_buffer : string or file-like object Path to SAS file or object implementing binary read method.såindex : identifier of index column Identifier of column that should be used as index of the DataFrame. encoding : string Encoding for text data. chunksize : int Read file `chunksize` lines at a time, returns iterator.sEformat : string File format, only `xport` is currently supported.s_iterator : boolean, default False Return XportReader object for reading file incrementally.sRead a SAS file into a DataFrame. %(_base_params_doc)s %(_format_params_doc)s %(_params2_doc)s %(_iterator_doc)s Returns ------- DataFrame or XportReader Examples -------- Read a SAS Xport file: >>> df = pd.read_sas('filename.XPT') Read a Xport file in 10,000 line chunks: >>> itr = pd.read_sas('filename.XPT', chunksize=10000) >>> for chunk in itr: >>> do_something(chunk) t_base_params_doct_format_params_doct _params2_doct _iterator_docsãClass for reading SAS Xport files. %(_base_params_doc)s %(_params2_doc)s Attributes ---------- member_info : list Contains information about the file fields : list Contains information about the variables in the file sÊRead observations from SAS Xport file, returning as data frame. Parameters ---------- nrows : int Number of rows to read from data file; if None, read whole file. Returns ------- A DataFrame. cCs0ytj|dƒSWntk r+tjSXdS(s3 Given a date in xport format, return Python date. s%d%b%y:%H:%M:%SN(Rtstrptimet ValueErrortpdtNaT(tdatestr((s6lib/python2.7/site-packages/pandas/io/sas/sas_xport.pyt _parse_datexs cCsSi}d}x9|D]1\}}||||!jƒ||<||7}qW|d=|S(s Parameters ---------- s: string Fixed-length string to split parts: list of (name, length) pairs Used to break up string, name '_' will be filtered from output. Returns ------- Dict of name:contents of string at given location. iR(tstrip(tstpartstouttstartR tlength((s6lib/python2.7/site-packages/pandas/io/sas/sas_xport.pyt _split_lines cCsn|dkrjtjt|ƒtjdƒƒ}tjd|d|fƒ}|jd|ƒ}||d<|S|S(NitS8sS%d,S%dtdtypetf0(tnptzerostlenR'tview(tvectnbytestvec1R'tvec2((s6lib/python2.7/site-packages/pandas/io/sas/sas_xport.pyt_handle_truncated_float_vec—s ! c CsYtjdƒ}|jd|ƒ}|d}|d}|d@}tjt|ƒdtjƒ}d|tj|d@ƒB}|dM}||d?d@dd>|dd>|d@BO}tjt|ƒfddƒ}||d<||d<|jddƒ}|jdƒ}|S(sf Parse a vector of float values representing IBM 8 byte floats into native 8 byte floats. s>u4,>u4R'R(tf1iÿÿÿii ii@ii€iiIÿÿïÿiiiAiÿiI€s>f8tf8( R)R'R,R*R+tuint8twheretemptytastype( R-R'R/txport1txport2tieee1tshifttieee2tieee((s6lib/python2.7/site-packages/pandas/io/sas/sas_xport.pyt_parse_float_vecªs(     +  t XportReadercBszeZeZd dd d„Zd„Zd„Zd„Zd„Z d„Z d d„Z d„Z e eƒd d „ƒZRS( s ISO-8859-1cCsÚ||_d|_||_||_t|tƒrTt|d|ƒ\}}}}nt|ttjt fƒr„t |dƒ|_ nH|j ƒ}y|j |jƒ}Wntk r¹nXtj|ƒ|_ |jƒdS(Nitencodingtrb(t _encodingt _lines_readt_indext _chunksizet isinstancetstrRRt text_typetbytestopentfilepath_or_buffertreadtencodetUnicodeEncodeErrortBytesIOt _read_header(tselfRKtindexR@t chunksizet compressiont should_closetcontents((s6lib/python2.7/site-packages/pandas/io/sas/sas_xport.pyt__init__æs       cCs|jjƒdS(N(RKtclose(RQ((s6lib/python2.7/site-packages/pandas/io/sas/sas_xport.pyRXscCs|jjdƒjƒS(NiP(RKRLtdecode(RQ((s6lib/python2.7/site-packages/pandas/io/sas/sas_xport.pyt_get_rowscCsË|jjdƒ|jƒ}|tkrA|jƒtdƒ‚n|jƒ}ddgddgddgddgd d gg}t||ƒ}|dd kr¸|jƒtd ƒ‚nt|d ƒ|d <||_|jƒ}t|d ƒ|d <|jƒ}|jƒ}|j t ƒ}|t k} |o1| sM|jƒtdƒ‚nt |dd!ƒ} ddgddgddgddgddgddgd d gg} t|jƒ| ƒ} d d gdd gddgddgg} | j t|jƒ| ƒƒt| d ƒ| d hhhh8s40s8shhh2s8shhl52sRRs0Floating field width {0} is not between 2 and 8.sObservation header not found.R R tS(*RKtseekRZt_correct_line1RXRR%Rt file_infot startswitht_correct_header1t_correct_header2tinttupdatet member_infoRLR+tljusttstructtunpacktdicttzipt _fieldkeyst TypeErrortformattitemsRtAttributeErrort_correct_obs_headertfieldst record_lengthttellt record_startt _record_counttnobsRYtcolumnst enumerateRGR)R't_dtype(RQtline1tline2tfifRhtline3theader1theader2t headflag1t headflag2tfieldnamelengthtmemRnttypest fieldcountt datalengtht fielddataRzt obs_lengthtfieldt fieldstructtfltmsgtktvtheadertxtitdtypelR'((s6lib/python2.7/site-packages/pandas/io/sas/sas_xport.pyRPs’            **    (       )CcCs|jd|jpdƒS(Ntnrowsi(RLRE(RQ((s6lib/python2.7/site-packages/pandas/io/sas/sas_xport.pyt__next__iscCs|jjddƒ|jjƒ|j}|ddkrItjdƒn|jdkrv|jj|jƒ||jS|jjddƒ|jjdƒ}tj |dtj ƒ}tj |dkƒ}t |ƒdkrãd}ndt |ƒ}|jj|jƒ|||jS( sÇ Get number of records in file. This is maybe suboptimal because we have to seek to the end of the file. Side effect: returns file position to record_start. iiiPsxport file may be corruptedi°ÿÿÿR'I i( RKRfR|R}twarningstwarnR{RLR)t frombuffertuint64t flatnonzeroR+(RQttotal_records_lengtht last_cardtixttail_pad((s6lib/python2.7/site-packages/pandas/io/sas/sas_xport.pyR~ls"     cCs(|dkr|j}n|jd|ƒS(s Reads lines from Xport file and returns as dataframe Parameters ---------- size : int, defaults to None Number of lines to read. If None, reads whole file. Returns ------- DataFrame RœN(tNoneRERL(RQtsize((s6lib/python2.7/site-packages/pandas/io/sas/sas_xport.pyt get_chunk‘s  cCs†|jddƒ}|ddk|ddk@|ddk@}|ddk|dd k@|dd kB|dd kB}||M}|S( NR's u1,u1,u2,u4R2itf2tf3R(iAiZi_i.(R,(RQR-R—tmisstmiss1((s6lib/python2.7/site-packages/pandas/io/sas/sas_xport.pyt_missing_double¢s ,: cCs|dkr|j}nt||j|jƒ}||j}|dkr]|jƒt‚n|jj|ƒ}t j |d|j d|ƒ}t j dt|ƒƒ}xt|jƒD]\}}|d|} |j|d} | dkr1t| |j|dƒ} |j| ƒ} t| ƒ} t j| | tnantrstripRtPY3RBRYRDRRt set_index(RQRœt read_linestread_lentrawtdatatdftjR™R-RR¬R—ty((s6lib/python2.7/site-packages/pandas/io/sas/sas_xport.pyRLªs<         .N(t__name__t __module__t_xport_reader_doct__doc__R§RWRXRZRPRR~R©R®Rt_read_method_docRL(((s6lib/python2.7/site-packages/pandas/io/sas/sas_xport.pyR?ãs    c  %   (RÂRRpRžtnumpyR)tpandas.util._decoratorsRtpandasRRtpandas.io.commonRRRgRjRkRyRtRRRRt _read_sas_docRÁRÃRR%R1R>R?(((s6lib/python2.7/site-packages/pandas/io/sas/sas_xport.pyt s>           9