B T\F@sddlmZmZmZddlZddlZddlmZddlm Z ddl m Z m Z ddl m Z ddlmZmZd d lmZe d d Z dd e dejd ddfddZddZddZdS))print_functiondivisionabsolute_importN)concat)unicode)system_encoding parse_bytes)delayed) open_files read_bytes) from_delayedT)ZpureZinferstrictc s |dk r|dk rtdt|ttfr.t|}t|fd|d|pFi} |dkr|dkrldd| D} qg} xtdt| |D]4} | | | |} tt tt t | } | | qWn>t |f||d|d |pi\}} fd dt | D} | s td ||s| St| SdS) a Read lines from text files Parameters ---------- urlpath : string or list Absolute or relative filepath(s). Prefix with a protocol like ``s3://`` to read from alternative filesystems. To read from multiple files you can pass a globstring or a list of paths, with the caveat that they must all have the same protocol. blocksize: None, int, or str Size (in bytes) to cut up larger files. Streams by default. Can be ``None`` for streaming, an integer number of bytes, or a string like "128MiB" compression: string Compression format like 'gzip' or 'xz'. Defaults to 'infer' encoding: string errors: string linedelimiter: string collection: bool, optional Return dask.bag if True, or list of delayed values if false storage_options: dict Extra options that make sense to a particular storage connection, e.g. host, port, username, password, etc. files_per_partition: None or int If set, group input files into partitions of the requested size, instead of one partition per file. Mutually exclusive with blocksize. Examples -------- >>> b = read_text('myfiles.1.txt') # doctest: +SKIP >>> b = read_text('myfiles.*.txt') # doctest: +SKIP >>> b = read_text('myfiles.*.txt.gz') # doctest: +SKIP >>> b = read_text('s3://bucket/myfiles.*.txt') # doctest: +SKIP >>> b = read_text('s3://key:secret@bucket/myfiles.*.txt') # doctest: +SKIP >>> b = read_text('hdfs://namenode.example.com/myfiles.*.txt') # doctest: +SKIP Parallelize a large file by providing the number of uncompressed bytes to load into each partition. >>> b = read_text('largefile.txt', blocksize='10MB') # doctest: +SKIP Returns ------- dask.bag.Bag if collection is True or list of Delayed lists otherwise See Also -------- from_sequence: Build bag from Python sequence Nz7Only one of blocksize or files_per_partition can be setZrt)modeencodingerrors compressioncSs g|]}tttt|qS)r listfile_to_blocks).0Zfilrr,lib/python3.7/site-packages/dask/bag/text.py Pszread_text..rF)Z delimiter blocksizeZsamplercsg|]}tt|qSr)r decode)rb)rrrrr\szNo files found) ValueError isinstancestrrr r rangelenr rmaprappendr encoder)ZurlpathrrrrZ linedelimiterZ collectionZstorage_optionsZfiles_per_partitionfilesZblocksstartZ block_filesZ block_lines_r)rrr read_texts25    r(c cs(|}x|D] }|Vq WWdQRXdS)Nr)Z lazy_fileflinerrrrgs rcCs|||}t|}t|S)N)rioStringIOr)blockrrtextlinesrrrrms  r)Z __future__rrrr+osZtoolzrZ compatibilityrZutilsrr r bytesr r Zcorerlinesepr(rrrrrrs     S