Loading Read Data Sets

3.5.8

Read Data Sets are imported into the Project using the “load” command (see section 3.4.8 for the usage statement). The exact way to load the Read Data Sets into the Project depends on how the data is organized on the disk. In the case where individual SFF files are stored together in a given repository (e.g. /data/sffFiles/EGFR_sff_files) and you know the specific file names you need, you might load the files as shown below (shown in the “here” file format):

load –sffDir /data/sffFiles/EGFR_sff_files -file - << HERE_TERMINATOR

"SffName"           "ReadGroup"     "SymLink"     "Name"

"DGVS90J01.sff"     "ReadGrp_1"     "false"     "DGVS90J01"

"DGVS90J02.sff"     "ReadGrp_1"     "false"     "DGVS90J02"

"DGVS90J03.sff"     "ReadGrp_1"     "false"     "DGVS90J03"

"DGVS90J04.sff"     "ReadGrp_1"     "false"     "DGVS90J04"

HERE_TERMINATOR

Note that a Read Group can be specified as above even if it was not explicitly added to the Project beforehand: the load command will automatically create Read Groups of given names, as needed. To explicitly add a Read Group to the Project, use the “create readGroup” command (see section 3.4.4.6 for the usage statement). In the example above, if you had wanted to create the read group in advance you could have typed “create readGroup ReadGrp_1”.

In our example, the 4 SFF files we want to load into the Project happen to be from the same sequencing Run. This allows us to use a more compact form of the command, with the “‑filePrefix” option:

load –sffDir /data/sffFiles/EGFR_sff_files \

–readGroup ReadGrp_1 -filePrefix DGVS90J \

–regions 1,2,3,4 –symLink “false”

Specifying the file prefix and choosing which regions to load gives the command enough context so each file does not need to be specified individually. Using the file prefix also provides sufficient specificity to prevent sff files from another Run from being erroneously picked up during the load. If there were a firstRun01.sff and a secondRun01.sff in the directory and you specified region 1 without a file prefix, both files would be imported.

If the SFF files you want to import are not gathered together in a repository but are instead still located in their respective Run analysis directories, the situation may be a bit different: while you might know what Runs and regions you want to import, you may not know the specific file prefix to use. This is because a Run’s SFF file names are assigned during analysis by the pipeline software. To get around this, one may set up an “alias” for the loaded files which you can use to refer to the files by region without knowing their actual names. In this next example, we use the alias mechanism, and further use the “analysisDir” option to specify a Run analysis directory:

   load –analysisDir /data/sequencingRuns/EGFR_Run_Dir/EGFR_Analysis_Dir/ \

  –readGroup ReadGrp_1 –regions 1,2,3,4 \

  –symLink false –alias EGFR_reads

In the command above, the alias has been set to “EGFR_reads”. With this alias established, you would refer to the region 1 file as “EGFR_reads01” and to the region 4 file as “EGFR_reads04”. The files are still loaded into the Project with their actual names, but the alias enables you to refer to them without knowing those names. With this facility, you could actually create the script for processing an Amplicon Project in advance of the completion of Pipeline analysis.

In most cases, when Read Data Sets are loaded into a Project, actual copies of the read data files are stored within the directory structure of the Project. This makes the Project directory portable so it can be moved to another location and maintain its integrity as a functional Project. However, you can save disk space and transfer time by setting the “symlink” parameter to “true” (it defaults to “false”). Enabling symlinking causes the Read Data Sets loaded to be stored as symbolic links pointing to the original read data files instead of creating physical copies of the data in the Project folder. If you use this option, your Projects may become nonfunctional if you either move the Project to a location where the symlinks can no longer reach the original data (such as to a different computational host) or if you move or delete the original data. Without access to the Read Data, you will be unable to rerun computations for the Project, and you will also be unable to view Flowgrams.