B T\@sfddlmZddlZddlZddlmZmZddlZddd Z d d Z dddZ dddZ ddZ dS))absolute_importN) open_files read_bytesrecordsTutf-8strictc  s|dkr|dk}|dkr$|r$td|d<|o6|dkd<t|df||dd|j|d|pbi} fd d t| |D} |rt| d d | DS| SdS) aWrite dataframe into JSON text files This utilises ``pandas.DataFrame.to_json()``, and most parameters are passed through - see its docstring. Differences: orient is 'records' by default, with lines=True; this produces the kind of JSON output that is most common in big-data applications, and which can be chunked when reading (see ``read_json()``). Parameters ---------- df: dask.DataFrame Data to save url_path: str, list of str Location to write to. If a string, and there are more than one partitions in df, should include a glob character to expand into a set of file names, or provide a ``name_function=`` parameter. Supports protocol specifications such as ``"s3://"``. encoding, errors: The text encoding to implement, e.g., "utf-8" and how to respond to errors in the conversion (see ``str.encode()``). orient, lines, kwargs passed to pandas; if not specified, lines=True when orient='records', False otherwise. storage_options: dict Passed to backend file-system implementation compute: bool If true, immediately executes. If False, returns a set of delayed objects, which can be computed at a later time. encoding, errors: Text conversion, ``see str.encode()`` compression : string or None String like 'gzip' or 'xz'. Nrz;Line-delimited JSON is only available withorient="records".orientlinesZwt name_function)encodingerrorsr Znum compressioncs"g|]\}}tt||qS)daskdelayedwrite_json_partition).0Zoutfiled)kwargsr5lib/python3.7/site-packages/dask/dataframe/io/json.py =szto_json..cSsg|] }|jqSr)path)rfrrrrAs) ValueErrorrpopZ npartitionszipZ to_delayedrcompute) dfurl_pathrr storage_optionsrr r r rZoutfilespartsr)rrto_json s&%     r!c Cs"|}|j|f|WdQRXdS)N)r!)rZopenfilerrrrrrFsrinferc  sddlm} dkrdkdkr0r0td|rHdks@sHtd|pNi}|rt|df|||d|\} ttj| } tfdd | D} n0t |d f|d |} fd d | D} | | S) aVCreate a dataframe from a set of JSON files This utilises ``pandas.read_json()``, and most parameters are passed through - see its docstring. Differences: orient is 'records' by default, with lines=True; this is appropriate for line-delimited "JSON-lines" data, the kind of JSON output that is most common in big-data scenarios, and which can be chunked when reading (see ``read_json()``). All other options require blocksize=None, i.e., one partition per input file. Parameters ---------- url_path: str, list of str Location to read from. If a string, can include a glob character to find a set of file names. Supports protocol specifications such as ``"s3://"``. encoding, errors: The text encoding to implement, e.g., "utf-8" and how to respond to errors in the conversion (see ``str.encode()``). orient, lines, kwargs passed to pandas; if not specified, lines=True when orient='records', False otherwise. storage_options: dict Passed to backend file-system implementation blocksize: None or int If None, files are not blocked, and you get one partition per input file. If int, which can only be used for line-delimited JSON files, each partition will be approximately this size in bytes, to the nearest newline character. sample: int Number of bytes to pre-load, to provide an empty dataframe structure to any blocks wihout data. Only relevant is using blocksize. encoding, errors: Text conversion, ``see bytes.decode()`` compression : string or None String like 'gzip' or 'xz'. Returns ------- dask.DataFrame Examples -------- Load single file >>> dd.read_json('myfile.1.json') # doctest: +SKIP Load multiple files >>> dd.read_json('myfile.*.json') # doctest: +SKIP >>> dd.read_json(['myfile.1.json', 'myfile.2.json']) # doctest: +SKIP Load large line-delimited JSON files using partitions of approx 256MB size >> dd.read_json('data/file*.csv', blocksize=2**28) rNrz;Line-delimited JSON is only available withorient="records".zSJSON file chunking only allowed for JSON-linesinput (orient='records', lines=True). ) blocksizesampler c s,g|]$}tt|dddqS)Nr)meta)rrread_json_chunk)rchunk)r r firstrrrrszread_json..Zrt)r r r cs g|]}tt|qSr)rrread_json_file)rr)rr rrrrs) Zdask.dataframeZ dataframerrlistrZcoreZflattenr(rZ from_delayed)rrr rr%r&r r r rZddZchunksr filesr)r r r*rr rr read_jsonKs*?      r.cCsNt|||}|dtj|fddd|}|dk rF|jrF|S|SdS)NrrT)rr )ioStringIOdecodeseekpdr.empty)r)r r rr'srrrrr(s  r(c Cs(|}tj|f||d|SQRXdS)N)rr )r3r.)rrr rrrrr+sr+)rNNTrrN)rNNNr"rrr#)N)Z __future__rr/Zpandasr3Z dask.bytesrrrr!rr.r(r+rrrrs  ; Y