Standard Flowgram Files (.sff)

2. Data Files and Formats : 2.3 Standard File Formats : 2.3.8 Standard Flowgram Files (.sff)

2.3.8

Standard Flowgram Files (.sff)

The Standard Flowgram File is used to store the information on one or many 454 Sequencing reads and their trace data. Sequencing reads obtained using the 454 Sequencing System differ from reads obtained using more traditional methods (“Sanger sequencing”) in that the 454 Sequencing data does not provide individual base measurements from which basecalls can be derived. Instead, it provides measurements that estimate the length of each homopolymer stretch in the sequence (e.g. in "AAATGG", "AAA" is a 3-mer stretch of A, "T" is a 1-mer stretch of T and "GG" is a 2-mer stretch of G). A basecalled sequence is then derived by converting each estimate into a homopolymer stretch of that length and concatenating the homopolymers.

The .sff file format consists of three sections: a common header section occurring once in the file; then for each read stored in the file, a read header section and a read data section. The data in each section consists of a combination of numeric and character data; the specific fields for each section are defined below. The sections adhere to the following rules:

•	The standard Unix types uint8_t, uint16_t, uint32_t and uint64_t are used to define 1, 2, 4 and 8 byte numeric values.

•	All multi-byte numeric values are stored using big endian byteorder (same as the SCF file format).

•	All character fields use single-byte ASCII characters.

•	Each section definition ends with an "eight_byte_padding" field, which consists of 0 to 7 bytes of padding, so that the byte length of each section is divisible by 8 (and hence the next section is aligned on an 8-byte boundary).

2.3.8.1

Common Header Section

The common header section consists of the following fields:

Field name	Format	Properties
magic_number	uint32_t	The magic_number field value is 0x2E736666, the uint32_t encoding of the string ".sff".
version	char[4]	The version number is 0001, or the byte array "\0\0\0\1".
index_offset	uint64_t	The index_offset and index_length fields are the offset and length of an optional index of the reads in the SFF file. If no index is included in the file, both fields must be 0.
index_length	uint32_t
number_of_reads	uint32_t	The number_of_reads field should be set to the number of reads stored in the file.
header_length	uint16_t	The header_length field should be the total number of bytes required by this set of header fields, and should be equal to "31 + number_of_flows_per_read + key_length", rounded up to the next value divisible by 8.
key_length	uint16_t	The key_length field should be set to the length of the key sequence used for these reads.
number_of_flows_per_read	uint16_t	The number_of_flows_per_read should be set to the number of flows for each of the reads in the file.
flowgram_format_ code	uint8_t	The flowgram_format_code should be set to the format used to encode each of the flowgram values for each read. Currently, only one flowgram format has been adopted, so this value should be set to 1. The flowgram format code 1 stores each value as a uint16_t, where the floating point flowgram value is encoded as "(int) round(value * 100.0)", and decoded as "(storedvalue / 100.0)". In other words, the values are stored as an integer encoding of a limited precision floating point value, keeping 2 places to the right of the decimal point, and capping the values at 655.35.
flow_chars	char[number_of_ flows_per_read]	The flow_chars should be set to the array of nucleotide bases ('A', 'C', 'G' or 'T') that correspond to the nucleotides used for each flow of each read. The length of the array should equal number_of_flows_per_read. Note that the flow_chars field is not null-terminated.
key_sequence	char[key_length]	The key_sequence field should be set to the nucleotide bases of the key sequence used for these reads. Note that the key_sequence field is not null-terminated.
eight_byte_ padding	uint8_t[*]	If any eight_byte_padding bytes exist in the section, they should have a byte value of 0.

If an index is included in the file, the index_offset and index_length values in the common header should point to the section of the file containing the index. To support different indexing methods, the index section should begin with the following two fields:

	index_magic_number	uint32_t
	index_version	char[4]

… and should end with an eight_byte_padding field, so that the length of the index section is divisible by 8. The format of the rest of the index section is specific to the indexing method used. The index_length given in the common header should include the bytes of these fields and the padding.

2.3.8.2

Read Header Section

The rest of the file contains the information about the reads, namely number_of_reads entries consisting of read header and read data sections. Each read header section consists of the following fields:

Field name	Format	Properties
read_header_length	uint16_t	The read_header_length should be set to the length of the read header for this read, and should be equal to "16 + name_length" rounded up to the next value divisible by 8.
name_length	uint16_t	The name_length field should be set to the length of the read's accession or name.
number_of_bases	uint32_t	The number_of_bases should be set to the number of bases called for this read.
clip_qual_left	uint16_t	The clip_qual_left and clip_adapter_left fields should be set to the position of the first base after the clipping point, for quality and/or an adapter sequence, at the beginning of the read. If only a combined clipping position is computed, it should be stored in clip_qual_left. The clip_qual_right and clip_adapter_right fields should be set to the position of the last base before the clipping point, for quality and/or an adapter sequence, at the end of the read. If only a combined clipping position is computed, it should be stored in clip_qual_right. Note that the position values use 1-based indexing, so the first base is at position 1. If a clipping value is not computed, the field should be set to 0. Thus, the first base of the insert is: max(1, max(clip_qual_left, clip_adapter_left)) …and the last base of the insert is: min( (clip_qual_right == 0 ? number_of_bases : clip_qual_right), (clip_adapter_right == 0 ? number_of_bases : clip_adapter_right) )
clip_qual_right	uint16_t
clip_adapter_left	uint16_t
clip_adapter_right	uint16_t
Name	char[name_length]	The name field should be set to the string of the read's accession or name. Note that the name field is not null-terminated.
eight_byte_padding	uint8_t[*]	If any eight_byte_padding bytes exist in the section, they should have a byte value of 0.

2.3.8.3

Read Data Section

The read data section consists of the following fields:

Field name	Format	Properties
flowgram_values	uint*_t[number_of_flows]	The flowgram_values field contains the homopolymer stretch estimates for each flow of the read. The number of bytes used for each value depends on the common header flowgram_format_code value (where the current value uses a uint16_t for each value).
flow_index_per_base	uint8_t[number_of_bases]	The flow_index_per_base field contains the flow positions for each base in the called sequence (i.e., for each base, the position in the flowgram whose estimate resulted in that base being called). Note that these values are "incremental" values, i.e. the stored position is the offset from the previous flow index in the field. All position values (prior to their incremental encoding) use 1-based indexing, so the first flow is flow 1.
Bases	char[number_of_ bases]	The bases field contains the basecalled nucleotide sequence.
quality_scores	uint8_t[number_of_bases]	The quality_scores field contains the quality scores for each of the bases in the sequence, where the values use the standard -log10 probability scale.
eight_byte_padding	uint8_t[*]	If any eight_byte_padding bytes exist in the section, they should have a byte value of 0.

2.3.8.4

Computing Lengths and Scanning the File

The length of the various read's section will vary because of different length accession numbers and different length nucleotide sequences. However, the various flow, name and bases lengths given in the common and read headers can be used to scan the file, accessing each read's information or skipping read sections in the file. The following pseudocode gives an example method to scan the file and access each read's data:

1.	Open the file and/or reset the file pointer position to the first byte of the file.

2.	Read the first 31 bytes of the file, confirm the magic_number value and version, then extract the number_of_reads, number_of_flows_per_read, flowgram_format_code, header_length, key_length, index_offset and index_length values.

a.	Convert the flowgram_format_code into a flowgram_bytes_per_flow value (currently with format_code 1, this value is 2 bytes).

3.	If the flow_chars and key_sequence information is required, read the next "header_length - 31" bytes, then extract that information. Otherwise, set the file pointer position to byte header_length.

4.	While the file contains more bytes, do the following:

a.	If the file pointer position equals index_offset, either read or skip index_length bytes in the file, processing the index if read.

b.	Otherwise,

i.	Read 16 bytes and extract the read_header_length, name_length and number_of_bases values.

ii.	Read the next "read_header_length - 16" bytes to read the name.

iii.	At this point, a test of the name field can be performed, to determine whether to read or skip this entry.

(a)	Compute the read_data_length as

"number_of_flows * flowgram_bytes_per_flow + 3 * number_of_bases"

…rounded up to the next value divisible by 8.

(b)	Either read or skip read_data_length bytes in the file, processing the read data if the section is read.