Data Processing Pipelines

1.1	Data Processing Pipelines

The data processing steps can be configured as a part of the sequencing Run using the Instrument Procedure Wizard (see GS Sequencer application, in Part A of this manual), or invoked post-Run using the GS Run Processor Manager tool in the GS Run Browser application (see Section 3.4.3), or by using the command line interface (CLI) on an Attendant PC or DataRig (described below). A data processing pipeline specifies the options of how to carry out the data processing with respect to;

•	When to process (during or post sequencing Run)

•	Where to process (on-instrument, Attendant PC, DataRig or cluster)

•	Which processing steps (no processing, image processing, signal processing, full processing)

•	What to process (standard shotgun/Paired End or Amplicon libraries)

The 454 Sequencing System Software supports five data processing pipelines.

•

None

•	Image processing only

•	Full processing for Shotgun or Paired End

•	Full processing for Amplicons

•	Signal processing – filter only

The data processing pipeline options are shown in Figure 1.

Figure 1: Data processing options in the GS Junior and GS FLX+ Systems. The blocks identify the various data acquisition, data processing or data analysis applications and their outputs. Raw images are captured as part of the sequencing Run (data acquisition). Depending on the processing type selected during Run set up (top of each column), the image processing and signal processing steps can either be performed as part of the sequencing Run (above the dotted line), or they can be carried out separately following the Run (below the dotted line) using the off-instrument version of the software (on a DataRig or a GS Junior Attendant PC). Note the two options for signal processing: Shotgun or Paired End vs. Amplicons. Image Processing Only (shaded) is not an option on the GS Junior System. Applications launched by direct user input are shown in Bold-Italics; others are run automatically after the previous stage is completed. The GS Run Browser application (not shown) can be invoked on the GS FLX+ Instrument, a DataRig, or a GS Junior Attendant PC to view raw images and other Run data, e.g. for troubleshooting purposes or to reanalyze the data set using different settings.

The “None”, “Image processing only”, “Full processing for Shotgun or Paired End” and “Full processing for Amplicons” pipelines are available through the Instrument Procedure Wizard on the GS FLX+ Instrument. All pipelines except “Image processing only” are available through the Instrument Procedure Wizard on the GS Junior Attendant PC.

The data processing pipelines are also available via the GS Run Processor Manager Tool accessed via the GS Run Browser interface.

Data processing pipelines can be accessed through the CLI using launch script commands. There is an additional launch script command, runAnalysisFilter, for running the signal processing filtering step only. When any processing job is launched, a pipeline processing script is called to specify the function calls to the gsRunProcessor executable. The processing scripts are XML-based text files located in /etc/gsRunProcessor on a DataRig, or in $RIGDIR/config on the GS FLX+ Instrument CPU or GS Junior Attendant PC. The output report generation (gsReporter) sections of the processing scripts can be configured by the user to alter the default output reports generated. This is discussed in Section 2.2. The XML script file specifying the filter parameters in the signal processing step can also be customized by the user by first generating a template XML file using the --template option of the gsRunProcessor command (see Section 1.4.4 for a full description of the filtering process and adjustment of filter parameters.) The pipelines and associated scripts are listed in Table 3 below:

Pipeline	Launch Script Command	Default Pipeline Processing Script
None	(none)	(none)
Image processing only	runImagePipe	imageProcessingOnly.xml
Full processing for Shotgun or Paired End	runAnalysisPipe	signalProcessing.xml (if passed a D_ directory) fullProcesssing.xml (if passed an R_ directory)
Full processing for Amplicons	runAnalysisPipeAmplicons	signalProcessingAmplicons.xml (if passed a D_ directory) fullProcesssingAmplicons.xml (if passed an R_ directory)
Signal processing – filter only	runAnalysisFilter	No default (created locally using --template option of gsRunProcessor)

Table 3: Data Processing Pipeline Launch Commands and Processing Scripts

The "Paired End" pipeline with its associated launch script "runAnalysisPipePairedEnd" have been merged into the standard pipeline. Use "runAnalysisPipe" to process Paired End Runs via the command line or select "Processing for Shotgun or Paired End” from the GS Run Browser or instrument GUIs.

An additional command, startGsProcessor, is used to prepare the directory structure for GS Run Processor jobs. It is a wrapper script that is called initially by all the ‘run’ launch script commands. Its use will be transparent to most users. It is described in the appendix Section 6.2 for reference.

The data flow for the image and signal processing is illustrated in Figure 2.

Figure 2: Data Flow for Image and Signal Processing

The directory structure for a GS FLX+ sequencing Run on a DataRig, after the data processing and report generation steps are carried out, is described below for a two-region sequencing Run: (Note that for a GS Junior System sequencing Run, the directory structure would be similar, except for the multiple regions).

Figure 3: Sequencing Run Data file structure after full Data Processing and default report generation