Composite wells file format

2.3.1

The CWF file is a container format which stores multiple “streams” of information. The container itself is a “ZIP” file (http://www.pkware.com/documents/casestudies/APPNOTE.TXT) with a single level hierarchy. Each stream is named and compressed separately, allowing for rapid access to any information in the file. The CWF file format is inspired from the “OpenDocument format” described in ISO/IEC 26300:2006 (http://www.iso.org/iso/iso_catalogue/catalogue_tc/catalogue_detail.htm?csnumber=43485). The OpenDocument format benefits from the segregation of concerns by separating the content, styles, metadata and application settings into four separate XML streams; the CWF format maintains a similar separation: flowgrams, called bases, meta data, processing history and run metrics are stored individually. The user should not normally unpack the CWF file, as each file is read as needed. Nonetheless, a C library (libcwf) is available that can read the CWF file format, for convenience.

The data for each region of the PicoTiterPlate device is stored in a separate CWF file (GS FLX+ System only). Because this file is fully self-contained, it is the only file that needs to be moved between the instrument and the data processing system for continued analysis (after the image processing phase of the GS Run Processor: this file format contains all the necessary information to generate any pipeline output artifact on-demand except for Standard Flowgram Format (SFF) files. Therefore, the user only needs to store one CWF file and one SFF file per region to fully archive the experiment’s processed data.

Textual data in the CWF container will generally be stored as XML. Specifically, there are four main required XML streams: meta.xml, metrics.xml, history.xml, sequences.xml and one optional one, filters.xml. Each of these files references a single XML shema (which is available on request).

Table 3 shows an example listing of a CWF file’s streams that might exist at the end of signal processing. This file represents the data from one region of a high-quality 2-region sequencing Run (GS FLX+ System), and the Table shows the size savings provided by the CWF compressed format. Each stream is described separately below.

Stream Name	Uncompressed Size	Compressed Size	Compression Ratio
mimetype	24	24	0%
rawWellDensity.pgm	8376341	1519187	82%
cfValues.double.dat	3749552	3328881	11%
ieValues.double.dat	3749552	3021477	19%
keyPassWellDensity.pgm	8376341	1103849	87%
histogram.unfiltered.ATGC.pgm	204695	24439	88%
histogram.unfiltered.TCAG.pgm	684743	40138	94%
filterResults.uint8.dat	468694	130450	72%
trimInfo.uint16.dat	937388	444081	53%
histogram.filteredCounts.ATGC.pgm	204695	22734	89%
histogram.nmers.ATGC.pgm	263879	20221	92%
histogram.filteredCounts.TCAG.pgm	684743	48430	93%
histogram.nmers.TCAG.pgm	1064508	60786	94%
0-149421.char.dna	33554213	10135865	70%
0-149421.uint_8.flow	33554213	9872137	71%
0-149421.uint_8.score	33554213	21064266	37%
149422-288120.char.dna	33554192	10070875	70%
149422-288120.uint_8.flow	33554192	9787599	71%
149422-288120.uint_8.score	33554192	20866664	38%
288121-424287.char.dna	33554389	10066101	70%
288121-424287.uint_8.flow	33554389	9787253	71%
288121-424287.uint_8.score	33554389	20863713	38%
424288-468693.char.dna	10323379	3139442	70%
424288-468693.uint_8.flow	10323379	3056298	70%
424288-468693.uint_8.score	10323379	6530831	37%
h0-41220.wel	33553894	29209536	13%
h41221-82441.wel	33553894	29206671	13%
h82442-123662.wel	33553894	29126347	13%
h123663-164883.wel	33553894	29069883	13%
h164884-206104.wel	33553894	29068170	13%
h206105-247325.wel	33553894	29082037	13%
h247326-288546.wel	33553894	29120018	13%
h288547-329767.wel	33553894	29184495	13%
h329768-370988.wel	33553894	29262051	13%
h370989-412209.wel	33553894	29361393	13%
h412210-453430.wel	33553894	29461534	12%
signalPerBase.float.dat	1874776	1539471	18%
h453431-468693.wel	12424082	10939973	12%
baseCalledSeq.dat	2812164	1126775	60%
sequences.xml	2334	612	74%
location.idx	4686940	2962553	37%
meta.xml	2162	701	68%
filters.xml	839	286	66%
metrics.xml	130111	18528	86%
history.xml	3286	1284	61%
TOTAL	752 MB	482MB	36%

Table 3: List of the streams in a CWF file following image and signal processing of a sequencing Run

2.3.1.1

mimetype

This is a single line file containing the words:

application/vnd.454.cwf

It must be the first stream in the file and must be stored uncompressed.

2.3.1.2

meta.xml

Like the OpenDocument format, meta.xml stores information about the Run itself. As a convenience, the schema defining the XML data stored in the CWF file references the Dublin Core (“DC”) metadata elements. The DC elements used and their interpreted meaning are summarized in Table 4. An example meta.xml file is shown in Figure 8.

DC Element	454 Sequencing System Usage
Title	Run name
Description	User-defined
Type	“flowgrams”
Source	Serial number of instrument performing the Run
Relation	Original Run name. e.g. “R_2007_06_27_15_44_21_rig3_ccelone_1007075seqkit93555420PELTxxEX2xxVERIIF2”
Creator	Instrument operator’s name
Date	“dcterms:created”: Date analysis was performed
Identifier	UUID version of job

Table 4: Dublin Core metadata elements used in CWF files

<?xml version="1.0" encoding="UTF-8"?>
<Metadata xmlns:tns="http://purl.org/dc/terms/" 
xmlns:tnsa="http://purl.org/dc/elements/1.1/" 
xmlns:tnsb="http://purl.org/dc/dcmitype/" 
xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" 
xsi:noNamespaceSchemaLocation="GSDataProcessing-1.0.xsd">
  <tnsa:title>R_2011_04_04_10_00_36_FLX08100646_Administrator_apr_2_build</tnsa:title>
  <tnsa:type>flowgrams</tnsa:type>
  <tnsa:creator>Administrator</tnsa:creator>
  <tns:created>2011-04-04T10:00:36Z</tns:created>
  <InstrumentSerialNumber>FLX08100646</InstrumentSerialNumber>
  <InstrumentVersion>2.6 (20110324_1135)</InstrumentVersion>
  <InstrumentModel>GSFLX</InstrumentModel>
  <InstrumentConfiguration>FLX+</InstrumentConfiguration>
  <Run>
    <Name>apr_2_build</Name>
    <RunId></RunId>
    <Project>Applications II</Project>
    <Kit>XL+KIT</Kit>
    <Script>400x_TACG_70x75_XLPLUSKIT.icl</Script>
    <RegionCount>2</RegionCount>
    <RegionLayoutName>2_region</RegionLayoutName>
    <PTP>
      <ID>749573</ID>
      <WellSize unit="um">35</WellSize>
      <Size unit="mm">
        <Width>70</Width>
        <Height>75</Height>
      </Size>
    </PTP>
    <Barcodes>
      <Barcode>123456</Barcode>
    </Barcodes>
    <Flow>
      <FlowCount>1603</FlowCount>
      <CycleCount>400</CycleCount>
      <ActualOrder>SSSSOOOOSOOOOSOOOOSOOOOSPSSSSTACG…TACGPS</ActualOrder>
      <FlowOrder>PTACG…TACGP</FlowOrder>
    </Flow>
    <Images>
      <ImageWidth>4096</ImageWidth>
      <ImageHeight>4096</ImageHeight>
      <DcOffset>495</DcOffset>
      <MaxValue>16383</MaxValue>
    </Images>
  </Run>
  <Region>
    <Name>region 1</Name>
    <Number>1</Number>
    <TemplateBounds unit="pixel">
      <Center>
        <X>1024</X>
        <Y>2048</Y>
      </Center>
      <Dimension>
        <Width>2047</Width>
        <Height>4095</Height>
      </Dimension>
    </TemplateBounds>
    <RevisedBounds unit="pixel">
      <Center>
        <X>1024</X>
        <Y>2048</Y>
      </Center>
      <Dimension>
        <Width>2047</Width>
        <Height>4095</Height>
      </Dimension>
    </RevisedBounds>
  </Region>
  <WellStatus>uncorrected</WellStatus>
  <WellCount>1104008</WellCount>
</Metadata>

Figure 8: Example meta.xml stream

2.3.1.3

history.xml

This is a single XML file showing what processing has been done to these wells. It contains reference copies of the pipeline parameters that were used to create the final result, as well as processing times, dates, software revision numbers and a Universally Unique Identifier (“UUID”) for each processing step. Each analysis performed on the data set adds a new “Job” element to the stream. An example history.xml file is shown in Figure 9.

<?xml version="1.0" encoding="UTF-8"?>
<History xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance" 
xsi:noNamespaceSchemaLocation="GSDataProcessing-1.0.xsd">
  <Job>
    <ID>cc8dfc88-70ff-11e0-99ce-00215e70718c</ID>
    <Name>cwf_with_instrumentConfiguration_take3</Name>
    <ProcessingDirectoryName>D_2011_04_27_14_54_37_FLX08100646_imageProcessing
Only_cwf_with_instrumentConfiguration_take3</ProcessingDirectoryName>
    <OS>Linux</OS>
    <StartTime>2011-04-27T18:54:38Z</StartTime>
    <TotalJobSeconds>12435</TotalJobSeconds>
    <PartialJobSeconds>11927</PartialJobSeconds>
    <GsRunProcessorVersion>2.6</GsRunProcessorVersion>
    <GsRunProcessorBuild>20110426_1019</GsRunProcessorBuild>
    <Host>FLX08100646</Host>
    <NumDataSetsInJob>2</NumDataSetsInJob>
    <NumProcessors>4</NumProcessors>
    <PartialNumProcessors>2</PartialNumProcessors>
    <Type>imageProcessingOnly</Type>
    <Pipeline>imageProcessingOnly</Pipeline>
    <DisplayText>Image processing only</DisplayText>
    <Description>
            This is the default pipeline for taking the raw images and
            making intermediate files that can be analyzed by
            other pipelines. 
        </Description>
    <CommandLine>
--run=/data/2011_04_04/R_2011_04_04_10_00_36_FLX08100646_Administrator
_apr_2_build/D_2011_04_27_14_54_37_FLX08100646_imageProcessingOnly_cwf_with_instrument
Configuration_take3/dataRunParams.xml 
--imageLog=/data/2011_04_04/R_2011_04_04_10_00_36_FLX08100646_Administrator
_apr_2_build/imageLog.parse 
--images=/data/2011_04_04/R_2011_04_04_10_00_36_FLX08100646_Administrator
_apr_2_build/rawImages/ 
--out=/data/2011_04_04/R_2011_04_04_10_00_36_FLX08100646_Administrator
_apr_2_build/D_2011_04_27_14_54_37_FLX08100646_imageProcessingOnly_cwf_with_instrument
Configuration_take3/regions 
--log=/data/2011_04_04/R_2011_04_04_10_00_36_FLX08100646_Administrator
_apr_2_build/D_2011_04_27_14_54_37_FLX08100646_imageProcessingOnly_cwf_with_instrument
Configuration_take3/gsRunProcessor.log 
--error=/data/2011_04_04/R_2011_04_04_10_00_36_FLX08100646_Administrator
_apr_2_build/D_2011_04_27_14_54_37_FLX08100646_imageProcessingOnly_cwf_with_instrument
Configuration_take3/gsRunProcessor_err.log 
--job=cc8dfc88-70ff-11e0-99ce-00215e70718c 
--pipe=/usr/local/rig/apps/gsRunProcessor/etc/gsRunProcessor/imageProcessingOnly.xml 
--name=cwf_with_instrumentConfiguration_take3 
--progress 
--remoteProgress=localhost:4540 
    </CommandLine>
    <ParamsUsed>
      <CameraClassifier computeTime="6" name="CameraClassifier">
        <enable>true</enable>
        <removeHotPixels>true</removeHotPixels>
        <imageScaleFactor>1</imageScaleFactor>
        <hotPixelThreshold>500</hotPixelThreshold>
        <computeSharpnessMask>true</computeSharpnessMask>
        <minPPI>30</minPPI>
        <standardDeviationLimit>2.2</standardDeviationLimit>
        <sampleBlockSize>128</sampleBlockSize>
        <fftRadius>45</fftRadius>
      </CameraClassifier>
      <WellFinder computeTime="398" name="WellFinder">
        <enable>true</enable>
        <imageScaleFactor>1</imageScaleFactor>
        <kernelSize>51</kernelSize>
        <upsampleHighDensityPtps>true</upsampleHighDensityPtps>
        <minPPISignal>30</minPPISignal>
        <minConsensusSignal>20</minConsensusSignal>
        <minWellSpacing>3</minWellSpacing>
        <secondSearchPass>false</secondSearchPass>
        <blockSize>20000</blockSize>
        <upsampleFactor>2</upsampleFactor>
        <maskBeta>0.6</maskBeta>
        <maskAlpha>0.1</maskAlpha>
        <numPixelsPerWell>1</numPixelsPerWell>
        <morphologyThresholdMultiplier>1</morphologyThresholdMultiplier>
        <morphologyNumInARow>5</morphologyNumInARow>
      </WellFinder>
      <WellBuilder computeTime="11520" name="WellBuilder">
        <enable>true</enable>
        <kernelSize>51</kernelSize>
        <minPPISignal>30</minPPISignal>
        <scaleFactor>1</scaleFactor>
        <useBicubic>true</useBicubic>
        <usePpiInterpolation>false</usePpiInterpolation>
        <firstFlowToInterpolate>20</firstFlowToInterpolate>
        <imageScaleFactor>1</imageScaleFactor>
        <skipBackgroundSubtraction>false</skipBackgroundSubtraction>
      </WellBuilder>
      <MetricsGenerator computeTime="1" name="MetricsGenerator">
        <enable>true</enable>
      </MetricsGenerator>
    </ParamsUsed>
    <FinishTime>2011-04-27T22:21:58Z</FinishTime>
    <TotalUserSeconds>37614</TotalUserSeconds>
    <TotalSystemSeconds>3974</TotalSystemSeconds>
  </Job>
</History>

Figure 9: Example history.xml stream

2.3.1.4

location.idx

This is a binary index to the wells files. It contains common data about each well that can be used to support a well browser-style application. Items in this stream are stored in Intel Little-Endian format (i.e. the rank is stored as Byte3 Byte2 Byte1 Byte0). The wells file contains one field per well, and each field is made up of the packed structure shown in Figure 10:

	15			0
1	Rank (unsigned integer)
2
3
4
5	X Coordinate (integer part)
6	X Coordinate (integer part)		fract.
7	Y Coordinate (integer part)
8	Y Coordinate (integer part)		fract.
9	P	Sequence ID
10	Reserved

Figure 10: Packed structure of the location.idx file

•	P is a bit showing if the well has passed filtering (1) or has been discarded (0).

•	Sequence ID is an index to the table contained in sequences.xml (see below) showing which sequence (if any) is matched.

•	The reserved field is for future use and should be set to all 0's.

•

The X and Y coordinates are stored with two bits of fractional information. This allows the storage of well coordinates with sub-pixel resolution in the CWF file format. Applications accessing the coordinates may simply choose to map bytes 5/6 and 7/8 to 16-bit integers and divide by 4, preserving or discarding the subpixel data as needed. Also note that all coordinates are relative to a common “0,0” location representing the upper left hand corner of the PicoTiterPlate device (the corner opposite the DataMatrix code), no matter what the region. In other words, the coordinate reflects the actual location of the well on the original PicoTiterPlate device and not the offset into the region itself.

2.3.1.5

metrics.xml

This file contains all the derived statistics created during data processing. This file can be used to create all the ancillary output files, including all the reports that were produced by the 454 Sequencing System software versions anterior to 2.0.00. The data is divided into various sections in four main types.

•	The first is the header section, which contains metrics that are valid across all keys and sequences.

•	The next sections are the “MetricsPerKey,” containing metrics that cover one key, either library or control. There will be one MetricsPerKey block per key used in the experiment.

•	The next sections are the “MetricsPerSequence” blocks. The metrics for each Control DNA sequence are contained in separate blocks.

•

The “Other” block is a free-form container for metrics that may be used by Roche and 454 Life Sciences’ troubleshooters to evaluate problematic sequencing Runs. These metrics can and will change between releases of software, and therefore, users should not depend upon them for the assessments of Runs.

An additional block, the “Streams” block, acts as a manifest for data stored in auxiliary streams, inside the CWF file. Each data stream is individually tagged with a “type” identifier. The stream block contains the exact stream name, and the type of the binary data contained in the data. While the exact stream name and data type may change with future releases of the software, the “type” name will remain constant. Users implementing CWF file readers should use the streams information contained in metrics.xml file to find their data and not depend on a particular file naming convention. See the Sections on “Other Streams” (2.3.1.10 and 2.3.1.11) for information on the data types and stream names.

An example metrics.xml file is shown in Figure 11.

<?xml version="1.0" encoding="utf-8"?>
<Metrics xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:noNamespaceSchemaLocation="GSDataProcessing-1.0.xsd">
  <RunMetrics>
    <MaxWellCount>529820</MaxWellCount>
    <RawWellCount>468698</RawWellCount>
    <SampleKeyPassWellCount>451747</SampleKeyPassWellCount>
    <ControlKeyPassWellCount>7818</ControlKeyPassWellCount>
    <ControlKeys>
      <Key>ATGC</Key>
    </ControlKeys>
    <SampleKeys>
      <Key>TCAG</Key>
    </SampleKeys>
    <Streams>
      <Stream type="rawWellDensity">
        <StreamName>rawWellDensity.pgm</StreamName>
        <DataType>image</DataType>
      </Stream>
      <Stream type="carryForwardCorrections">
        <StreamName>cfValues.double.dat</StreamName>
        <DataType>double</DataType>
      </Stream>
      <Stream type="incompleteExtensionCorrections">
        <StreamName>ieValues.double.dat</StreamName>
        <DataType>double</DataType>
      </Stream>
        <Stream type="filterResults">
        <StreamName>filterResults.uint8.dat</StreamName>
        <DataType>byte</DataType>
      </Stream>
      <Stream type="signalPerBase">
        <StreamName>signalPerBase.float.dat</StreamName>
        <DataType>float</DataType>
      </Stream>
    </Streams>
    <Other>
      <NukeSignalStrengthBalancer>
        <medianOneMerA>1.09085</medianOneMerA>
        <medianOneMerT>0.909846</medianOneMerT>
        <medianOneMerG>0.898174</medianOneMerG>
        <medianOneMerC>0.894138</medianOneMerC>
      </NukeSignalStrengthBalancer>
      <BlowByCorrector>
        <droopLambda>-0.00171434</droopLambda>
        <MedianSignal>1375.71</MedianSignal>
        <MaximumSignal>5086.11</MaximumSignal>
        <MedianDensity>12</MedianDensity>
        <MinimumDensity>1</MinimumDensity>
        <MaximumDensity>19</MaximumDensity>
        <num_low_density_low_signal_wells>
        14742</num_low_density_low_signal_wells>
        <num_high_density_low_signal_wells>
        10481</num_high_density_low_signal_wells>
        <num_low_density_high_signal_wells>
        13645</num_low_density_high_signal_wells>
        <num_high_density_high_signal_wells>
        14899</num_high_density_high_signal_wells>
        <mask_averaging_used>true</mask_averaging_used>
        <FinalMask>
          <class density="high" signal="high" class="0">
            <epsilon>0.174537</epsilon>
            <beta>0.964658</beta>
          </class>
          <class density="low" signal="high" class="1">
            <epsilon>0.188713</epsilon>
            <beta>0.988837</beta>
          </class>
          <class density="high" signal="low" class="2">
            <epsilon>0.171184</epsilon>
            <beta>0.950913</beta>
          </class>
          <class density="low" signal="low" class="3">
            <epsilon>0.184087</epsilon>
            <beta>0.900547</beta>
          </class>
        </FinalMask>
      </BlowByCorrector>
      <CafieCorrector>
        <droopLambda>-0.00158241</droopLambda>
      </CafieCorrector>
      <NukeSignalStrengthBalancer>
        <medianOneMerA>1.01745</medianOneMerA>
        <medianOneMerT>0.986296</medianOneMerT>
        <medianOneMerG>0.987408</medianOneMerG>
        <medianOneMerC>0.980024</medianOneMerC>
      </NukeSignalStrengthBalancer>
    </Other>
  </RunMetrics>
</Metrics>

Figure 11: Example metrics.xml stream

2.3.1.6

sequences.xml

A list of sequences referred to by the Sequence ID field in the locations.idx node. This stream can also be used to identify which sequences denote Control DNA reads and which are library reads. By convention, library keys are named “ATCG-library” where “ATGC” is the four letter library key. An example sequences.xml file is shown in Figure 12. (Note that the sequences were truncated (ellipses) on the Figure, for brevity).

<?xml version="1.0" encoding="iso-8859-1"?>
<Sequences>
  <Sequence Type="None">
    <ID>0</ID>
    <Name>unknown</Name>
    <Key></Key>
    <Seq></Seq>
  </Sequence>
  <Sequence Type="Control">
    <ID>1</ID>
    <Name>ATGC-control</Name>
    <Key>ATGC</Key>
    <Seq>ATGC</Seq>
  </Sequence>
  <Sequence Type="Library">
    <ID>2</ID>
    <Name>TCAG-key</Name>
    <Key>TCAG</Key>
    <Seq>TCAG</Seq>
  </Sequence>
  <Sequence Type="Control">
    <ID>3</ID>
    <Name>TF2LonG</Name>
    <Key>ATGC</Key>
    <Seq>ATGCCA...TGTGTG</Seq>
  </Sequence>
  <Sequence Type="Control">
    <ID>4</ID>
    <Name>TF7LonG</Name>
    <Key>ATGC</Key>
    <Seq>ATGC...TTCCTGTGTG</Seq>
  </Sequence>
  <Sequence Type="Control">
    <ID>5</ID>
    <Name>TF90LonG</Name>
    <Key>ATGC</Key>
    <Seq>ATGCCGCA...GTGTG</Seq>
  </Sequence>
  <Sequence Type="Control">
    <ID>6</ID>
    <Name>TF100LonG</Name>
    <Key>ATGC</Key>
    <Seq>ATGCAT...GTGTG</Seq>
  </Sequence>
  <Sequence Type="Control">
    <ID>7</ID>
    <Name>TF120LonG</Name>
    <Key>ATGC</Key>
    <Seq>ATGCA...CCTGTGTG</Seq>
  </Sequence>
  <Sequence Type="Control">
    <ID>8</ID>
    <Name>TF150MMP7A</Name>
    <Key>ATGC</Key>
    <Seq>ATGCGC...ATGG</Seq>
  </Sequence>
</Sequences>

Figure 12: Example sequences.xml stream

2.3.1.7

filters.xml

A list of filters referred to by the values in the “filterResults.uint8.dat” stream (see section 2.3.1.8, below). Note that the order of filters in this file is not guaranteed. It is also likely that the filters will be reorganized in a future release of the software to provide more detail. An example filters.xml file is shown in Figure 13.

<?xml version="1.0" encoding="utf-8"?>
<Filters xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
xsi:noNamespaceSchemaLocation="GSDataProcessing-1.0.xsd">
  <Filter basic="true">
    <ID>0</ID>
    <Name>Pass</Name>
  </Filter>
  <Filter basic="true">
    <ID>1</ID>
    <Name>No Key</Name>
  </Filter>
  <Filter basic="true">
    <ID>2</ID>
    <Name>Bad Band</Name>
  </Filter>
  <Filter basic="true">
    <ID>3</ID>
    <Name>Trimmed Too Short Quality</Name>
  </Filter>
  <Filter basic="true">
    <ID>4</ID>
    <Name>Low Pass Filter</Name>
  </Filter>
  <Filter basic="true">
    <ID>5</ID>
    <Name>Classifier Filter</Name>
  </Filter>
  <Filter basic="true">
    <ID>6</ID>
    <Name>Dot Filter</Name>
  </Filter>
  <Filter basic="true">
    <ID>7</ID>
    <Name>Mixed Filter</Name>
  </Filter>
  <Filter basic="true">
    <ID>8</ID>
    <Name>Trimmed Too Short Primer</Name>
  </Filter>
  <Filter basic="true">
    <ID>9</ID>
    <Name>Low Quality</Name>
  </Filter>
</Filters>

Figure 13: Example filters.xml stream

2.3.1.8

filterResults.uint8.dat

This file contains a binary list of wells that failed filtering and the specific filter that the well did not pass. The filters are defined in file filters.xml. This stream consists of one byte per well, sorted by rank. See the discussion of “Other Streams” (sections 2.3.1.10 and 2.3.1.11) for a description on the layout of this stream.

The majority of the CWF payload consists of the well flow values themselves. In order to support rapid, random flow extraction from a CWF file, wells are stored in “blocks”. The data format for the blocks can vary, depending on the application. The size of each block is equal to the largest multiple of the data size times the number of flows, that is smaller than 32 Megabytes. In other words, no well’s flow values will be broken between blocks and no block will be larger than 32 MB.

The naming convention for each block of flowgrams is:

TY-Z.wel

… where

•	T is a letter indicating the type of the block (see below),

•	Y is the starting well index, and

•	Z is the final well index.

Y and Z are integer values and should not be padded with zeros. The number of wells in each block is arbitrary and should not be hard coded. There is no assumption that all blocks in the file are stored in the same format nor that all blocks have the same number of wells. For example, Control DNA reads and failed wells may be archived in a lower fidelity format, while the library fragments are stored as full floating point numbers. However, the format cannot currently store discontinuous ranges of elements in a block, which constitutes a limitation of this feature. Users of the CWF format are encouraged to use libcwf to insulate them from the complexity of extracting the well information from the CWF file.

The block types are as follows:

•	r: This is a raw well block type, identical to the *.wells format of software versions 1.1.03 and earlier. Values are stored “Little Endian” byte order as generated by Intel brand x86 processors.

•

Header:

◦	32 bits: numWells (as unsigned integer)

◦	16 bits: numFlows (as unsigned integer)

◦	numFlows bytes: flowLabels (one of “A”,”T”,”G”,”C”,”P”)

•

Body:

◦	32 bits: rank (as unsigned integer)

◦	16 bits: xCoord (as unsigned integer)

◦	16 bits: yCoord (as unsigned integer)

◦	32 bits * numFlows: flowValues (as IEEE Single Precision Float)

•

h: Half-precision floating point: Each block is made up of four arrays stored back-to-back without padding. The size of the first three arrays is equal to the data type size times the number of wells in the block. The last array consumes the rest of the block, and is equal to 2 bytes times the number of wells times the number of flows. The number of wells is derived from the name of the stream, and the number of flows can be retrieved from the meta.xml stream. Values are stored “Little Endian” byte order as generated by Intel brand x86 processors.

◦	X-coordinates as unsigned 16-bit numbers

◦	Y-coordinates as unsigned 16-bit numbers

◦	flow ranks as unsigned 32-bit numbers

◦	flow values as “half-precision binary floating point numbers”

The half-precision floating point is a relatively new binary floating point format that uses 2 bytes and which is not covered by the IEEE 754 standard for encoding floating point numbers (but is included in the IEEE 754r proposed revision; http://www.validlab.com/754R/). The format uses 1 sign bit, a 5-bit excess-15 exponent, 10 mantissa bits (with an implied 1 bit) and all the standard IEEE rules. The minimum and maximum representable values are 2.98×10^-8 and 65504, respectively. Libcwf includes a half to full precision floating point conversion routine.

•

f: Full-precision floating point: Each block is made up of four arrays stored back-to-back without padding. The size of the first three arrays is equal to the data type size times the number of wells in the block. The last array consumes the rest of the block, and is equal to 4 bytes times the number of wells times the number of flows. The number of wells is derived from the name of the stream, and the number of flows can be retrieved from the meta.xml stream. Values are stored “Little Endian” byte order as generated by Intel brand x86 processors.

◦	X-coordinates as unsigned 16-bit numbers

◦	Y-coordinates as unsigned 16-bit numbers

◦	flow ranks as unsigned 32-bit numbers

◦	flow values as IEE 754 binary full precision floating point number (http://grouper.ieee.org/groups/754/)

•

i: Integer: Each block is made up of four arrays stored back-to-back without padding. The size of the first three arrays is equal to the data type size times the number of wells in the block. The last array consumes the rest of the block, and is equal to 2 bytes times the number of wells times the number of flows. The number of wells is derived from the name of the stream, and the number of flows can be retrieved from the meta.xml stream. Values are stored “Little Endian” byte order as generated by Intel brand x86 processors. (Note that this block is rarely used because it cannot store values less than zero, which can occur during signal processing routines, and lacks precision for values near one.)

◦	X-coordinates as unsigned 16-bit numbers

◦	Y-coordinates as unsigned 16-bit numbers

◦	flow ranks as unsigned 32-bit numbers

◦	flow values as IEE 754 binary full-precision floating point number

2.3.1.9

Base Called Data

If a data set has processed through the baseCaller section of the GS Run Processor, the actual bases are written into the CWF file. This allows for the generation of FASTA FNA and QUAL files on demand. Like the flowgrams, the reads are stored in blocks. Unlike the flowgrams, however, each block consists of a variable number of reads. A special stream called “baseCalledSeq.dat” is used to index the basecalled blocks. This stream contains 6 bytes per well:

•	Byte 0/1: Total stored read length

•	Byte 2/3: Number of reads to trim from the distal end (3’)

•	Byte 4/5: Number of reads to trim from the local end (5’)

The complete data for any read is stored in three separate streams, identified by the “dna”, “qual” and “flow” extensions, containing the reads, the PHRED based quality values and the offset flow indices, respectively. Each base of each well uses one byte in each file.

The total number of bytes consumed by a read is reflected in the first field of the baseCalledSeq.dat. Therefore these two bytes can be used as an index of sorts. For example, the byte offset in the “dna” file for read 100 can be found by summing the first two bytes of the first 99 entries in baseCalledSeq.dat. The 100^th entry can then be used to tell how many bytes are available for read 100. It is important to note that since the basecalls are stored in blocks, one must first find the appropriate block, then compute the offset from there. Again, users of the CWF format are encouraged to use the libcwf to insulate them from errors in extracting the base information.

2.3.1.10

Other Public Streams

The cwf file can also contain other binary streams. These are usually identified by the .dat suffix (except in the case of the “image” type, see note below). Many are used for storing intermediate results of various processing stages, but there are three notable streams that contain metrics data that may be of interest to end users.

•	The first, “filterResults,” can be used to find which wells have passed which filter.

•	The second, “rawWellDensity” is a grayscale image containing the bead loading density plot.

•	The last is the “keyPassDensity” stream, a grayscale image showing the keypass density.

With the exception of the “image” format stream, each stream contains one entry per read, sorted in rank order. The location.idx can be used to find which offset corresponds to which read.

The possible data types are listed in Table 5, and the current stream types, in Table 6.

Name	Size in Bytes/Well	Notes
byte	1
short	2	Signed short, Intel byte order
unsignedShort	2	Unsigned short, Intel byte order
int	4	Standard integer, intel byte order
unsignedInt	4	Unsigned integer, Intel byte order
float	4	IEEE 754 Single Precision floating point number
double	4	IEEE 754 Double Precision floating point number
image	N/A	PGM format graphics. See note below on resolution/ registration.

Table 5: Possible data types

Stream Type	Contents
filterResults	The information about which well failed which filter in the qualityFiltering section of code.
trimInfo	The trim points from the end of the reads. This number is in flowgram-space instead of base-space like the information in the baseCallerSeq.dat stream.
cfValues	The carry-forward corrections for each well.
ieValues	The incomplete extension correction factors for each well.
signalPerBase	The average value of the keypass flows for this well. Can be used to do a simple basecalling. Also can be used to judge the strength of the well.
keyPassDensity	An image referenced to the region showing the density wells that pass key. This is normalized for each PTP pitch, so 0=0% loading and 255=100% loading.
rawWellDensity	An image referenced to the region showing the density of the bead loading. This is normalized for each PTP pitch, so 0=0% loading and 255=100% loading.

Table 6: Current stream types

Note on Image Formats: The only image format stored in CWF is the lossless Portable Anymap format, specifically the “P5” PGM (Portable Graymap) variant (http://netpbm.sourceforge.net/doc/pgm.html). To save space, only the area encompassed by the region is included in the image. To properly register the image against the original PTP device, you must offset the image slice by the region boundary. This region boundary can be read in the “RevisedBounds” element of the “Region” block in meta.xml stream. 454 Life Sciences Corporation may introduce other image formats in future variants of the CWF file, so it is important to read the “magic number” and/or file extension of the image to determine the correct image decoder to use.

2.3.1.11

Other Private Streams

There are other streams of data that can be stored in the CWF file, that contain intermediate results from various pipeline and post-processing functions. These are left unspecified to avoid restricting the development of new algorithms in the data processing applications. CWF file users should neither depend on their existence nor attempt to parse them as their contents may change between revisions of the software or invocations of different processing streams.

2.3.1.12

Other Files

There may be other files in the CWF container, the contents of which may include log files and other binary data. A proper CWF writer (like the GS Run Processor) will copy any other unknown stream verbatim to the destination. CWF file readers should not balk at these extra streams, but should not depend on their existence either. This allows advanced users to add additional payloads to the CWF container before moving them to the data analysis system.