ó è?F[c@sKddlZddlZddlZddlZddlZddlZddlmZddlZddl m Z m Z m Z m Z mZddlmZmZmZeƒZejdƒZdefd„ƒYZdefd „ƒYZd ejfd „ƒYZd efd „ƒYZdefd„ƒYZdefd„ƒYZdS(iÿÿÿÿN(tQueue(tDEFAULT_PART_SIZEtminimum_part_sizet chunk_hashest tree_hasht bytes_to_hex(tUploadArchiveErrortDownloadArchiveErrortTreeHashDoesNotMatchErrorsboto.glacier.concurrenttConcurrentTransferercBs2eZedd„Zd„Zd„Zd„ZRS(i cCs||_||_g|_dS(N(t _part_sizet _num_threadst_threads(tselft part_sizet num_threads((s6lib/python2.7/site-packages/boto/glacier/concurrent.pyt__init__+s  cCslt|ƒ}|j|kr'|j}n|}tjd|j|ƒttj|t|ƒƒƒ}||fS(NsfThe part size specified (%s) is smaller than the minimum required part size. Using a part size of: %s(RR tlogtdebugtinttmathtceiltfloat(R t total_sizetmin_part_size_requiredRt total_parts((s6lib/python2.7/site-packages/boto/glacier/concurrent.pyt_calculate_required_part_size0s    cCsYtjdƒx|jD]}t|_qWx|jD]}|jƒq4WtjdƒdS(NsShutting down threads.sThreads have exited.(RRR tFalsetshould_continuetjoin(R tthread((s6lib/python2.7/site-packages/boto/glacier/concurrent.pyt_shutdown_threads<s   cCsbtjdƒx't|ƒD]}|j||fƒqWx$t|jƒD]}|jtƒqGWdS(NsAdding work items to queue.(RRtrangetputR t _END_SENTINEL(R Rt worker_queueRti((s6lib/python2.7/site-packages/boto/glacier/concurrent.pyt_add_work_items_to_queueDs  (t__name__t __module__RRRRR%(((s6lib/python2.7/site-packages/boto/glacier/concurrent.pyR *s tConcurrentUploadercBs;eZdZedd„Zdd„Zd„Zd„ZRS(sConcurrently upload an archive to glacier. This class uses a thread pool to concurrently upload an archive to glacier using the multipart upload API. The threadpool is completely managed by this class and is transparent to the users of this class. i cCs/tt|ƒj||ƒ||_||_dS(su :type api: :class:`boto.glacier.layer1.Layer1` :param api: A layer1 glacier object. :type vault_name: str :param vault_name: The name of the vault. :type part_size: int :param part_size: The size, in bytes, of the chunks to use when uploading the archive parts. The part size must be a megabyte multiplied by a power of two. :type num_threads: int :param num_threads: The number of threads to spawn for the thread pool. The number of threads will control how much parts are being concurrently uploaded. N(tsuperR(Rt_apit _vault_name(R tapit vault_nameRR((s6lib/python2.7/site-packages/boto/glacier/concurrent.pyRVs c Cs6tj|ƒj}|j|ƒ\}}dg|}tƒ}tƒ}|jj|j||ƒ} | d} |j |||ƒ|j || ||ƒy|j |||ƒWn<t k ré} t jdƒ|jj|j| ƒ| ‚nXt jdƒ|jj|j| tt|ƒƒ|ƒ} t jdƒ| dS(s^Concurrently create an archive. The part_size value specified when the class was constructed will be used *unless* it is smaller than the minimum required part size needed for the size of the given file. In that case, the part size used will be the minimum part size required to properly upload the given file. :type file: str :param file: The filename to upload :type description: str :param description: The description of the archive. :rtype: str :return: The archive id of the newly created archive. tUploadIdsHAn error occurred while uploading an archive, aborting multipart upload.sCompleting upload.sUpload finished.t ArchiveIdN(toststattst_sizeRtNoneRR*tinitiate_multipart_uploadR+R%t_start_upload_threadst_wait_for_upload_threadsRRRtabort_multipart_uploadtcomplete_multipart_uploadRR( R tfilenamet descriptionRRRt hash_chunksR#t result_queuetresponset upload_idte((s6lib/python2.7/site-packages/boto/glacier/concurrent.pytuploadns2               cCsƒxrt|ƒD]d}|jƒ}t|tƒr[tjd|ƒ|jƒtd|ƒ‚n|\}}|||R#R9RDR((s6lib/python2.7/site-packages/boto/glacier/concurrent.pyR5°s   N( R&R't__doc__RRR3R@R6R5(((s6lib/python2.7/site-packages/boto/glacier/concurrent.pyR(Ls    2 tTransferThreadcBs,eZd„Zd„Zd„Zd„ZRS(cCs2tt|ƒjƒ||_||_t|_dS(N(R)RNRt _worker_queuet _result_queuetTrueR(R R#R<((s6lib/python2.7/site-packages/boto/glacier/concurrent.pyR¼s  cCs‡xv|jrxy|jjddƒ}Wntk r;qnX|tkrV|jƒdS|j|ƒ}|jj|ƒqW|jƒdS(Nttimeouti( RRORAtEmptyR"t_cleanupt_process_chunkRPR!(R tworkRE((s6lib/python2.7/site-packages/boto/glacier/concurrent.pytrunÄs    cCsdS(N((R RV((s6lib/python2.7/site-packages/boto/glacier/concurrent.pyRUÑscCsdS(N((R ((s6lib/python2.7/site-packages/boto/glacier/concurrent.pyRTÔs(R&R'RRWRURT(((s6lib/python2.7/site-packages/boto/glacier/concurrent.pyRN»s  RHcBs5eZdded„Zd„Zd„Zd„ZRS(ic Csntt|ƒj||ƒ||_||_||_t|dƒ|_||_||_ ||_ | |_ dS(Ntrb( R)RHRR*R+t _filenametopent_fileobjt _upload_idt _num_retriest_time_between_retriest_retry_exceptions( R R,R-R9R>R#R<t num_retriesttime_between_retriestretry_exceptions((s6lib/python2.7/site-packages/boto/glacier/concurrent.pyRÙs      c Cs¥d}x˜t|jdƒD]ƒ}y|j|ƒ}PWq|jk rœ}tjd|d|j|d|jd|j|j |ƒt j |j ƒ|}qXqW|S(NispException caught uploading part number %s for vault %s, attempt: (%s / %s), filename: %s, exception: %s, msg: %si( R3R R]t _upload_chunkR_RterrorR+RYt __class__RIRJR^(R RVRER$R?((s6lib/python2.7/site-packages/boto/glacier/concurrent.pyRUçs c CsÍ|\}}||}|jj|ƒ|jj|ƒ}tj|ƒjƒ}tt|ƒƒ}||t|ƒdf}t j d||ƒ|j j |j |j|t|ƒ||ƒ} | jƒ||fS(NisUploading chunk %s of size %s(R[tseektreadthashlibtsha256t hexdigestRRtlenRRR*t upload_partR+R\R( R RVRFRt start_bytetcontentst linear_hashttree_hash_bytest byte_rangeR=((s6lib/python2.7/site-packages/boto/glacier/concurrent.pyRc÷s     cCs|jjƒdS(N(R[tclose(R ((s6lib/python2.7/site-packages/boto/glacier/concurrent.pyRTs(R&R'RCRRURcRT(((s6lib/python2.7/site-packages/boto/glacier/concurrent.pyRHØs   tConcurrentDownloadercBs8eZdZedd„Zd„Zd„Zd„ZRS(sý Concurrently download an archive from glacier. This class uses a thread pool to concurrently download an archive from glacier. The threadpool is completely managed by this class and is transparent to the users of this class. i cCs&tt|ƒj||ƒ||_dS(s :param job: A layer2 job object for archive retrieval object. :param part_size: The size, in bytes, of the chunks to use when uploading the archive parts. The part size must be a megabyte multiplied by a power of two. N(R)RsRt_job(R tjobRR((s6lib/python2.7/site-packages/boto/glacier/concurrent.pyRs cCsª|jj}|j|ƒ\}}tƒ}tƒ}|j|||ƒ|j||ƒy|j|||ƒWn)tk r˜}tj d|ƒ|‚nXtj dƒdS(s’ Concurrently download an archive. :param filename: The filename to download the archive to :type filename: str s2An error occurred while downloading an archive: %ssDownload completed.N( Rtt archive_sizeRRR%t_start_download_threadst_wait_for_download_threadsRRR(R R9RRRR#R<R?((s6lib/python2.7/site-packages/boto/glacier/concurrent.pytdownload$s    c CsBdg|}t|dƒ°}x¦t|ƒD]˜}|jƒ}t|tƒrztjd|ƒ|jƒt d|ƒ‚n|\}} } } | ||<|| } |j | ƒ|j | ƒ|j ƒq,WWdQXt t|ƒƒ} tjd|jj| ƒ|jj| kr4|jƒtd|jj| fƒ‚n|jƒdS(s Waits until the result_queue is filled with all the downloaded parts This indicates that all part downloads have completed Saves downloaded parts into filename :param filename: :param result_queue: :param total_parts: twbs?An error was found in the result queue, terminating threads: %ss0An error occurred while uploading an archive: %sNs?Verifying final tree hash of archive, expecting: %s, actual: %ssBTree hash for entire archive does not match, expected: %s, got: %s(R3RZR RARBRCRRRRRftwritetflushRRRttsha256_treehashR(R R9R<RR;tfRDRERFRt actual_hashtdataRmt final_hash((s6lib/python2.7/site-packages/boto/glacier/concurrent.pyRx9s6            cCsgtjdƒxSt|jƒD]B}t|j||ƒ}tjdƒ|jƒ|j j |ƒqWdS(NsStarting threads.gš™™™™™É?( RRR R tDownloadWorkerThreadRtRIRJRKR RL(R R<R#RDR((s6lib/python2.7/site-packages/boto/glacier/concurrent.pyRw`s    (R&R'RMRRRyRxRw(((s6lib/python2.7/site-packages/boto/glacier/concurrent.pyRs s    'R‚cBs,eZdded„Zd„Zd„ZRS(icCsAtt|ƒj||ƒ||_||_||_||_dS(s  Individual download thread that will download parts of the file from Glacier. Parts to download stored in work queue. Parts download to a temp dir with each part a separate file :param job: Glacier job object :param work_queue: A queue of tuples which include the part_number and part_size :param result_queue: A priority queue of tuples which include the part_number and the path to the temp file that holds that part's data. N(R)R‚RRtR]R^R_(R RuR#R<R`RaRb((s6lib/python2.7/site-packages/boto/glacier/concurrent.pyRjs    cCsd}xtt|jƒD]c}y|j|ƒ}PWq|jk rx}tjd|d|jƒtj |j ƒ|}qXqW|S(s‹ Attempt to download a part of the archive from Glacier Store the result in the result_queue :param work: s6Exception caught downloading part number %s for job %siN( R3R R]t_download_chunkR_RRdRtRIRJR^(R RVRERDR?((s6lib/python2.7/site-packages/boto/glacier/concurrent.pyRUƒs c Cs¼|\}}||}|||df}tjd||ƒ|jj|ƒ}|jƒ}ttt|ƒƒƒ}|d|kr£td||d|fƒ‚n||t j |ƒ|fS(s§ Downloads a chunk of archive from Glacier. Saves the data to a temp file Returns the part number and temp file location :param work: isDownloading chunk %s of size %stTreeHashsBTree hash for part number %s does not match, expected: %s, got: %s( RRRtt get_outputRgRRRRtbinasciit unhexlify( R RVRFRRmRqR=R€R((s6lib/python2.7/site-packages/boto/glacier/concurrent.pyRƒ–s    (R&R'RCRRURƒ(((s6lib/python2.7/site-packages/boto/glacier/concurrent.pyR‚is   (R0Rt threadingRhRItloggingt boto.compatRR†tboto.glacier.utilsRRRRRtboto.glacier.exceptionsRRRtobjectR"t getLoggerRR R(tThreadRNRHRsR‚(((s6lib/python2.7/site-packages/boto/glacier/concurrent.pyts"       ( "o4]