# BPNet
[![CircleCI](https://circleci.com/gh/kundajelab/bpnet.svg?style=svg&circle-token=f55c1cf580b05df76e260993f7645e35d5302e76)](https://circleci.com/gh/kundajelab/bpnet)

BPNet is a python package with a CLI to train and interpret base-resolution deep neural networks trained on functional genomics data such as ChIP-nexus or ChIP-seq. It addresses the problem of pinpointing the regulatory elements in the genome:

<img src="./docs/theme_dir/bpnet/dna-words.png" alt="BPNet" style="width: 600px;"/>

Specifically, it aims to answer the following questions:
- What are the sequence motifs?
- Where are they located in the genome?
- How do they interact?

For more information, see the BPNet manuscript:

*Deep learning at base-resolution reveals motif syntax of the cis-regulatory code* (http://dx.doi.org/10.1101/737981.)

## Overview

<img src="./docs/theme_dir/bpnet/overview.png" alt="BPNet" style="width: 400px;"/>

## Getting started

Main documentation of the bpnet package and an end-to-end example higlighting the main features are contained in the following colab notebook **<https://colab.research.google.com/drive/1VNsNBfugPJfJ02LBgvPwj-gPK0L_djsD>**. You can run this notebook yourself by clicking on '**Open in playground**'. Individual cells of this notebook can be executed by pressing the Shift+Enter keyboard shortcut.

<img src="./docs/theme_dir/bpnet/colab-header.png" alt="BPNet" style="width: 300px;"/>

To learn more about colab, visit <https://colab.research.google.com> and follow the 'Welcome To Colaboratory' notebook.

## Main commands

Compute data statistics to inform hyper-parameter selection such as choosing to trade off profile vs total count loss (`lambda` hyper-parameter):

```bash
bpnet dataspec-stats dataspec.yml
```

Train a model on BigWig tracks specified in [dataspec.yml](examples/chip-nexus/dataspec.yml) using an existing architecture [bpnet9](bpnet/premade/bpnet9-pyspec.gin) on 200 bp sequences with 6 dilated convolutional layers:

```bash
bpnet train --premade=bpnet9 dataspec.yml --override='seq_width=200;n_dil_layers=6' .
```

Compute contribution scores for regions specified in the `dataspec.yml` file and store them into `contrib.scores.h5`

```bash
bpnet contrib . --method=deeplift contrib.scores.h5
```

Export BigWig tracks containing model predictions and contribution scores

```bash
bpnet export-bw . --regions=intervals.bed --scale-contribution bigwigs/
```

Discover motifs with TF-MoDISco using contribution scores stored in `contrib.scores.h5`, premade configuration [modisco-50k](bpnet/premade/modisco-50k.gin) and restricting the number of seqlets per metacluster to 20k:

```bash
bpnet modisco-run contrib.scores.h5 --premade=modisco-50k --override='TfModiscoWorkflow.max_seqlets_per_metacluster=20000' modisco/
```

Determine motif instances with CWM scanning and store them to `motif-instances.tsv.gz`

```bash
bpnet cwm-scan modisco/ --contrib-file=contrib.scores.h5 modisco/motif-instances.tsv.gz
```

Generate additional reports suitable for ChIP-nexus or ChIP-seq data:

```bash
bpnet chip-nexus-analysis modisco/
```

Note: these commands are also accessible as python functions:
- `bpnet.cli.train.bpnet_train`
- `bpnet.cli.train.dataspec_stats`
- `bpnet.cli.contrib.bpnet_contrib`
- `bpnet.cli.export_bw.bpnet_export_bw`
- `bpnet.cli.modisco.bpnet_modisco_run`
- `bpnet.cli.modisco.cwm_scan`
- `bpnet.cli.modisco.chip_nexus_analysis`

## Main python classes

- `bpnet.seqmodel.SeqModel` - Keras model container specified by implementing output 'heads' and a common 'body'. It contains methods to compute the contribution scores of the input sequence w.r.t. differnet output heads.
- `bpnet.BPNet.BPNetSeqModel` - Wrapper around `SeqModel` consolidating profile and total count predictions into a single output per task. It provides methods to export predictions and contribution scores to BigWig files as well as methods to simulate the spacing between two motifs.
- `bpnet.cli.contrib.ContribFile` - File handle to the HDF5 containing the contribution scores
- `bpnet.modisco.files.ModiscoFile` - File handle to the HDF5 file produced by TF-MoDISco.
  - `bpnet.modisco.core.Pattern` - Object containing the PFM, CWM and optionally the signal footprint
  - `bpnet.modisco.core.Seqlet` - Object containing the seqlet coordinates.
  - `bpnet.modisco.core.StackedSeqletContrib` - Object containing the sequence, contribution scores and raw data at seqlet locations.
- `bpnet.dataspecs.DataSpec` - File handle to the `dataspec.yml` file
- `dfi` - Frequently used alias for a pandas `DataFrame` containing motif instance coordinates produced by `bpnet cwm-scan`. See the [colab notebook](https://colab.research.google.com/drive/1VNsNBfugPJfJ02LBgvPwj-gPK0L_djsD) for the column description.

## Installation

Supported python version is 3.6. After installing anaconda ([download page](https://www.anaconda.com/download/)) or miniconda ([download page](https://conda.io/miniconda.html)), create a new bpnet environment by executing the following code:

```bash
# Clone this repository
git clone git@github.com:kundajelab/bpnet.git
cd bpnet

# create 'bpnet' conda environment
conda env create -f conda-env.yml

# Disable HDF5 file locking to prevent issues with Keras (https://github.com/h5py/h5py/issues/1082)
echo 'export HDF5_USE_FILE_LOCKING=FALSE' >> ~/.bashrc

# Activate the conda environment
source activate bpnet
```

Alternatively, you could also start a fresh conda environment by running the following

```bash
conda env create -n bpnet python=3.6
source activate bpnet
conda install -c bioconda pybedtools bedtools pybigwig pysam genomelake
pip install git+https://github.com/kundajelab/DeepExplain.git
pip install tensorflow # or tensorflow-gpu if you are using a GPU
pip install bpnet
echo 'export HDF5_USE_FILE_LOCKING=FALSE' >> ~/.bashrc
```

When using bpnet from the command line, don't forget to activate the `bpnet` conda environment before:

```bash
# activate the bpnet conda environment
source activate bpnet

# run bpnet
bpnet <command> ...
```

### (Optional) Install `vmtouch` to use `bpnet train --vmtouch`

To use the `--vmtouch` in `bpnet train` command and thereby speed-up data-loading, install [vmtouch](https://hoytech.com/vmtouch/). vmtouch is used to load the bigWig files into system memory cache which allows multiple processes to access
the bigWigs loaded into memory. 

Here's how to build and install vmtouch:

```bash
# ~/bin = directory for localy compiled binaries
mkdir -p ~/bin
cd ~/bin
# Clone and build
git clone https://github.com/hoytech/vmtouch.git vmtouch_src
cd vmtouch_src
make
# Move the binary to ~/bin
cp vmtouch ../
# Add ~/bin to $PATH
echo 'export PATH=$PATH:~/bin' >> ~/.bashrc
```