%% A quick bootstrap tutoral to microarray analysis using
%% pycompClust
%%
%% Written By:  Christopher E. Hart

\documentclass[10pt]{article}
\usepackage{html, graphicx}
\usepackage[T1]{fontenc}
\usepackage{ae}
\usepackage{times}
\usepackage{html}
\usepackage{makeidx}
\usepackage{verbatim}

\let\VERBATIM\verbatim
\def\verbatim{%
\def\verbatim@font{\tiny\ttfamily}%
\VERBATIM}


\title{A Quick Start Guide to Microarray Analysis using compClust}
\author{Christopher E. Hart\\
        California Institute of Technology \\
        Pasadena, CA 91125 }

\begin{document}

\maketitle
\tableofcontents

\newpage

This tutorial is designed to bootstrap a user into using compClust and python
for microarray data analysis.  I will try and provide an explanation of what
each piece of code is doing and why it works, but further reading my be
required in you're favorite python reference.  Code segments shown as below:

\begin{verbatim}
# this is a code segment
\end{verbatim}

Should be ``copy-and-pastable'' directly into the python interpretor assuming
all the modules are preloaded as specified below or using IPython the provided
configuration files are being used. 

\section{Setting up the Environment - Python, IPython, compClust}

I'll assume that Python, IPython and compClust have been installed by you or your
system administrator.  Follow the installation guide's instructions.

One of the greatest thing about python is the exposed interpretor.  On a unix
system, simply type python or on a windows system double click on the
python.exe file and you'll be faced with a command prompt ``>>>'' where you can
enter any valid python statement and it will be evaluated when you hit enter.
You can exit this environment by pressing ctrl-D.  Through the rest of this
tutorial I'll expect that the reader has at least read through some of the
documentation or tutorials at
\htmladdnormallink{http://python.org/doc/}{http://python.org/doc/} and has a
working understanding of python's basic syntax (which is quite straight forward
and easy to learn).

\begin{verbatim}
\$ python
Python 2.2.3+ (#1, Jul  5 2003, 11:04:18) 
[GCC 3.3.1 20030626 (Debian prerelease)] on linux2
Type "help", "copyright", "credits" or "license" for more information.
>>> 1+1
2
\end{verbatim}

\subsubsection{IPython}
Although the standard python interpretor is quite powerful, IPyhon written
Fernando Perez and available at
\htmladdnormallink{http://ipython.scipy.org/}{http://ipython.scipy.org/} adds
many useful features to the standard python interface and I highly recommand
that you use IPython.  It creates a very rich and powerful command line
interface (CLI) complete with tab-completion, object introspection, source code
exploration, debuging tools, online help, variable management and many other
very cool features.

Using IPython (or if you resist a custom \$PYTHONSTARTUP file) you can set
things up to autoload all the compClust modules that you find useful along with any
other modules such as Numeric, MLab and stats perhaps.  I've posted my
ipython-compClust startup files at
\htmladdnormallink{http://woldlab.caltech.edu/compClust/}{http://woldlab.caltech.edu/compClust/}
and you can either use my setup or customize them, but be forewarned the rest
of this tutorial will assume you've left the import semantics alone.

My IPython configuration preloads most compClust modules and many other modules which
I find useful on a regular basis.  The import semantics loads the modules into
the name space of the top most module and avoids loading anything into the
global name space to avoid conflicts.  If you were to do these manually the
import semantics would be as follows. 

\begin{verbatim} 
## compClust modules
from compClust.mlx import datasets 
from compClust.mlx import labelings 
from compClust.mlx import views 
from compClust.mlx import models
from compClust.iplot import IPlotTk as IPlot
from compClust.score import roc
from compClust.score import ConfusionMatrix
from compClust.util import listOps
from compClust.util import Histogram
from compClust.util import DistanceMetrics

## non-standard but nice modules
import MLab
import Numeric

## standard modules
import sys
import os
import string
import re

# now classes and methods are available as follows:

ds = datasets.Dataset(MLab.rand(10,10))
l  = labelings.Labeling(ds, 'labeling 1')
v  = views.RowSubsetView(ds, [1,2,3,9])
\end{verbatim}

As a a usage note, I recommend using a text editor for drafting out code
statements and then simply copy-and-pasting the segments into the IPython
shell.  This accomplishes several things: 1) Saves you're analysis path for
further investigation.  2) provides a place to debug and edit your code.  If
you are using a X11 based system and can install
\htmladdnormallink{xsel}{http://www.niksula.cs.hut.fi/~vherva/xsel/} then the
IPython magic function @pasteNrun may be quite useful to you.  I find in
practice as I dig deeper into unraveling a dataset I end up constructing a very
specialized python module for interacting for a specific dataset.  This dataset
specific analysis module provides you with an instant notebook of your work
which is critical to reevaluate your analysis and can be given to
collaborators.  Also remember that anything right of a \# sign on a line is a
comment and is not evaluated by python. 

%% you might want to mention that @pasteNrun is from your environemnt

\section{A First Step}

I think the best way to understand the utility of compClust in terms of microarray
analysis comes by example.  The first step will be working with an
uninteresting small synthetic dataset.

\subsection{data creation}

Lets create the dataset, throughout this tutorial I'll only write the python
code and I'll leave out the command prompt, this allows you to easily copy and
paste examples here into the IPython shell and explore the results.

This code will generate a 5-dimensional dataset from 3 distinct Guassian
clusters.  Imagine this begin a time-course of microarray results.   

\begin{verbatim}
# this first cluster is mostly flat-liners (non-responders)

import RandomArray  # this comes with Numeric 

# we'll be using the multivarate_normal function, learn more about it
# by typing multivarate_normal? in IPython.  Do the same for MLab.diag
# also know that you can get a list of all attributes (functions/classes) 
# by typing RandomArray. and hitting the tab-key

nonResponders = RandomArray.multivariate_normal([0,0,0,0,0], 
                                                MLab.diag(MLab.rand(5)*.5), 
                                                [30])
posResponders = RandomArray.multivariate_normal([0,1,2,3,4], 
                                                MLab.diag(MLab.rand(5)*.5), 
                                                [20])
negResponders = RandomArray.multivariate_normal([0,-1,-2,-3,-4], 
                                                MLab.diag(MLab.rand(5)*.5), 
                                                [20])

# now we'll assemble a dataset, first by concatenating these 
# three arrays together

a = Numeric.concatenate((nonResponders, posResponders, negResponders))

# now we can pass this array to the dataset constructor which is "smart" and
# can create a dataset from a list or lists , Numeric array, or a filename 
# containing tab-delimanated data. NOTE: that module names are lowercase and 
# class names are uppercase.  

ds = datasets.Dataset(a)

# now we'll set the name of the dataset
ds.setName('Simple Example')

# lets also keep track of which group each of these vectors came from using 
#labelings.  I'll describe in more detail the difference between 
#GlobalLabelings GlobalWrappers and Labelings.  In microarray analysis  its
#almost always desirable to have GlobalLabelings of GlobalWrappers.

## I can't stress enough that you should us IPython's help system to
## investigate the parameters each of these function calls take.

originLab = labelings.GlobalLabeling(ds, 'origins')
originLab.addLabelToRows(ds, 'nonResponders', range(00,30))
originLab.addLabelToRows(ds, 'posResponders', range(30,50))
originLab.addLabelToRows(ds, 'negResponders', range(50,70))

# lets also keep track of the original index of each of the ``genes'' in this
# dataset

indexLab = labelings.GlobalLabeling(ds, 'index')
indexLab.labelRows(ds, range(ds.getNumRows()))

\end{verbatim}

I highly recommend at this point you spend some time exploring the objects that
you've created.  IPython has pervasive tab-completion which is both great to
save typing times, but also exploration.  To use the tab-completion, simple
start to type a variable, function, module or class name and hit the tab key.
If when you hit tab you have unambiguously identified a variable it will
automatically be completed to that variable.  However, if it is ambiguous
regarding which variable might be completed a list of all possible completions
will be displayed.  Continue type the name that you intend and hit tab again to
repeat the process on a narrowed set of possibilities.  This also works for
member attributes for both classes and modules - for instance type
``indexLab.'' then hit <tab>.  You'll see a list of all the class functions
available.  Now you can learn more about any of these functions by completing
the name and following it with a ``?''.  At this point you should be able to
learn about all the basic functionality of both labelings and datasets.

\subsection{simple visualization}

Visualization is essential to the analysis of most data and microarray data is
no exception.  Most of the visualization tools I'll be discussing reside in the
IPlot module.  Effectively IPlot strives to provide interactive plotting tools
such that users can identify data by ``clicking'' on points on the plot.

The DatasetPlot and DatasetPlotView classes form the core of the IPlot system.
Currently the DatasetPlot is built on top of the very nice Pmw.BLT.Graph widget
- therefore any operations that can be done to a BLT.Graph can also be done to
a DatasetPlot.  The DatasetPlot is considered the rendering engine and the
DatasetPlotView provides mappings of arbitrary data attributes to plot
characteristics such as marker and line coordinates, size and color.

As usual this is best demonstrated.  This is a continuation from the previous
snippet of code, so make sure you are working from within the same python
session.  

\begin{verbatim}

# first we'll create a DatasetPlot and a DatasetRowPlotView

dp = IPlot.DatasetPlot()
v  = IPlot.DatasetRowPlotView(ds)

# now we render the plotview by tell the DatasetPlot to plot it
dp.plot(v)

\end{verbatim}

You should see a plot something like figure~\ref{fig:simple}

\begin{figure}
\begin{center}
  \includegraphics[scale=.5]{graphics/simplePlot1}
  \caption{Generated Data Visualized in using a DatasetRowPlotView}
  \label{fig:simple}
\end{center}
\end{figure}

You should also find that clicking on a line results in a small information box
being displaying in the upper left hand corner.  Holding down Control and Left
clicking on this box will open another dialog box with more information about
that gene vector as shown in figure~\ref{fig:infobox}.  From this box you can
explore which labels are attached to this vector.  By clicking on a labeling in
the left box the right box shows what label(s) are attached to it from that
labeling.  Further you can highlight all gene-vectors which share that label
using the pull-down action list on the right-hand box.  You can zoom the plot
by holding shift and the left mouse button, zoom-out by holding control and
pressing the left mouse button.  

\begin{figure}
\begin{center}
  \includegraphics[scale=.5]{graphics/infobox}
  \caption{Sample Information Box From \ref{fig:simple}}
  \label{fig:infobox}
\end{center}
\end{figure}

Often you might want to have the plot display more information when you click on
a gene-vector.  This can be accomadated for using the primaryLabeling and
secondaryLabeling parameters which work on almost all IPlot functions.    The
primaryLabeling always defaults to the index or key of the gene-vector in the
particular view or dataset but can be over ridden, but you \emph{must} be sure
that there is a unique label for each gene vector.  It is usually best to leave
the primaryLabeling alone.  The seondaryLabeling however can be anything -
we'll set it to be the origin in the example below.  

\begin{verbatim}

v  = IPlot.DatasetRowPlotView(ds, secondaryLabeling=ds.getLabeling('origins'))
dp.clear()
dp.plot(v)

\end{verbatim}

Now when you click on a line (or gene vector) you'll see the origin label in
the information box.

Now it is important to understand that the DatasetRowPlotView is composed of 5
Mappers which take attributes of the data and map them onto attributes of the
plot.  The five mappers are listed below:

\begin{description}
  \item[DataMapper] Positions sets of x,y points for each row in the dataset. 
  \item[ColorMapper] Sets the color for each row in the dataset.
  \item[MarkersMapper] Determines the marker size and color for each row in the
    dataset.  
  \item[AnnotationMapper] Determines the annotation that should be
    rendered for each row in the dataset.  
  \item[BindingsMapper] Determines
    what events should be triggered when data points are clicked on.
\end{description}

Next I'll demonstrate how to use the built in general purpose mappers to do
some interesting analysis of the data.  As you become more comfortable using
compClust and python you may find it useful to subclass out these mappers for special
purposes. 

\subsubsection{Color Mapping}

First we'll investigate color.  Again we'll build on the code we've been
workign with so don't close any windows or exit IPython.  If you did this
should all in a blink of an eye, so worry not.

%% FIXME: should all what???
\begin{verbatim}

# get the colorMapper that is currenlty being used by the DatasetRowPlotView 
cm = v.getColorMapper()

# lets set the color based on the value a vector has at the t=0
cm.setColorByColValue(0)
# now tell the plot to update itself.
dp.plot()

# it will probably be more interesting to color based on the last column of
# the dataset
cm.setColorByColValue(4)
dp.plot()

# if you didn't want to scale between red and blue you can adjust the 
# colorRange parameter.  Here we'll go between red and yellow.  If you
# think of a color wheel that starts at 0 and ends at 1 that is the 
# how color is assigned.

cm.setColorByColValue(4, colorRange=(0,0.3))
dp.plot()

# You can also threshold the color scale.  Here we'll cap at +/- 0.5
cm.setColorByColValue(4, minValue=-0.5, maxValue=0.5)
dp.plot()

# all colors are represented as HSV colors and so far we've only 
# adjusted the H - or hue parameter.  I'll leave it as an excersize to
# the reader to adjust the saturation and value parameters.

# perhaps we would like to color each line by there index in the file.  That is
# gene-vector 1 gets to be red and gene-vector 70 gets to be blue and 
# everything else is in between

cm.setColorByIndex()
dp.plot()

# we can also color based on a labeling (say the group of origin)
cm.setColorByLabeling(ds.getLabeling('origins'))
dp.plot()

# lastly we might want to color based on a function - say the average 
# value of each vector. 

def rowAvg(ds, row):
  return(MLab.mean(ds.getRowData(row)))

cm.setColorByFunction(rowAvg)
dp.plot()

# notice I had to define a function above which expects a dataset object 
# and a row and returns a value.  The ColorMapper takes care of scalling
# this value and turning it into a color.  However, sometimes though you might
# want to do this in one line using a lambda function (look it up in the python
# docs, its quite powerful).  The below does exactly the same thing
# as above.

cm.setColorByFunction(lambda ds,row: MLab.mean(ds.getRowData(row)))
dp.plot()

\end{verbatim}

The default color map used throughout IPlot is shown in
\ref{fig:colorScale} and the renderings of the above example code is shown in
figure \ref{fig:coloredBy} we can see that we can quickly choose different
coloring schemes.

\begin{figure}
  \begin{center}
    \includegraphics[scale=.75]{graphics/colorScale}
    \caption{the standard color scale used in IPlot}
    \label{fig:colorScale}
  \end{center}
\end{figure}

\begin{figure}
  \begin{center}
    \includegraphics[scale=.25]{graphics/coloredByCol0} 
    \includegraphics[scale=.25]{graphics/coloredByCol4}
    \includegraphics[scale=.25]{graphics/coloredByCol4RG}
    \includegraphics[scale=.25]{graphics/coloredByCol4Trunc}
    \includegraphics[scale=.25]{graphics/coloredByIndex}
    \includegraphics[scale=.25]{graphics/coloredByOrigins}
    \includegraphics[scale=.25]{graphics/coloredByMean}
    \caption{Various different ways to adjust the ColorMapper.  Colored by: A)
    value at column 0 B) value at column 4 C) value at column 4 (only using
    Red-to-Green) D) value at column 4, truncating color scale from +/- .5 E)
    index in the dataset F) the origins labeling G) the mean of each vector}
    \label{fig:coloredBy} 
  \end{center} 
\end{figure}

\subsubsection {Data Mapping}

Next we'll explore different DataMappers.  DataMappers project a
high-dimensional vector into some representation inside a 2d graph.  In all of
the proceding figures we've been using the default DataMapper - the
RowDataMapper.  This data mapper for each vector in the dataset plots its
points vs the index value.  However, sometimes you would like the X-axis to be
scaled according to time or some other non-evenly spaced interval
\ref{fig:xaxisByTime}

\begin{verbatim}

# first we'll create a labeling on ds that describes a time axis
# imagine we've taken RNA samples at the below time interval.

time = labelings.GlobalLabeling(ds, 'time')
time.labelCols(ds, [0,30,60,120,240])

# now we'll grab the DataMapper
rdm = v.getDataMapper()
rdm.setXAxisLabeling(ds.getLabeling('time'))
dp.plot()

%% FIXME: how do you get this plot to rescale?

\end{verbatim}

\begin{figure}
  \begin{center}
    \includegraphics[scale=.25]{graphics/xaxisByTime}
  \end{center}
  \caption{example modifing the RowDataMapper to scale the X-axis}
  \label{fig:xaxisByTime}
\end{figure}


Sometimes you may want to generate a scatter matrix of your data where the X-axis represents the values of expression during 1 measurement and the Y-axis another.  We can use the ColumnScatterDataMapper for this \ref{fig:csDM}

\begin{verbatim}

# we construct a pcaDM
csDM = IPlot.ColumnScatterDataMapper(v)

# now we can set v's dataMapper to be this new dataMapper
v.setDataMapper(csDM)
dp.plot()

# Now we can modify which two columns are plotted against each other
csDM.setXColumn(0)
csDM.setYColumn(4)
dp.plot()


\end{verbatim}

\begin{figure}
  \begin{center}
    \includegraphics[scale=.25]{graphics/csDM0vs1}
    \includegraphics[scale=.25]{graphics/csDM0vs4}
  \end{center}
  \caption{example using the ColumnScatterDataMapper.  A) column 0 vs 1 B)
  column   0 vs 4} 
  \label{fig:csDM} 
\end{figure}



Principle component analysis (PCA) has become quite a useful tool in mapping
out overall data structure of a dataset.  We can simply swap out the
RowDataMapper with the PCADataMapper \ref{fig:simplePCA}.  Note that even
though the plot has made quite a pronounced change, all the other Mappers are
still the same, so color, annotation and bindings are all maintained.  To
better understand the PCA projection I've written IPlot.PCAExplorer which will
be discussed below.

\begin{verbatim}

# we construct a pcaDM
pcaDM = IPlot.PCADataMapper(v)

# now we can set v's dataMapper to be this new dataMapper
v.setDataMapper(pcaDM)
dp.plot()

\end{verbatim}

\begin{figure}
  \begin{center}
    \includegraphics[scale=.25]{graphics/simplePCA}
  \end{center}
  \caption{PCA view of the same data that was shown in \ref{fig:xaxisByTime}}
  \label{fig:simplePCA}
\end{figure}

\subsubsection{Marker Mapper}

The final Mapper we'll discuss is the Marker Mapper.  Using the MarkerMapper
you can adjust the size of each marker much like you can adjust the color using
the color mapper \ref{fig:mm}. 

\begin{verbatim}
# first we get a handle to the MarkersMapper
mm = v.getMarkersMapper()

# now we'll adjust the size proportionally to the value in col 4
mm.setMarkerSizeByColValue(4)

dp.plot()

\end{verbatim}

\begin{figure}
  \begin{center}
    \includegraphics[scale=.25]{graphics/mm}
  \end{center}
  \caption{PCA view of the dataset, but now the marker size is proportional to value in the 4th column of each vector.}
  \label{fig:mm}
\end{figure}

\subsubsection{Binding Mapping and Annotation Mapping}
These mapper are an implementation detail that can be ignored by most users.

\subsection{Complex Visualization}

Many of the concepts behind these visualization and analysis techniques are
discussed in more detail in \cite{hart_etal03} and will be minimally discussed
in this tutorial.    

\subsubsection{Trajectory Summaries}

Often given a partitioning or classification of your data you'll want to
inspect each of these clusters.  The TracjectorySummary is a quick
visualization to draw a mean/std dev. plot for each cluster.  Inside of
compClust you'll usually have your classification described in terms of a
labeling.  If this is a globalLabeling than all is good, if it is not, then
you'll have to cast the local labeling to a globalLabeling using the
labelings.castToGlobalLabeling function.  Here we'll generate a trajectory
summary based each gene vector's cluster of origin, the origin's labeling
contains this information.  Figure \ref{fig:ts} illistrates the results of the
below function call.  Note you can get a detailed view of any of the clusters
by clicking on ``plot all'' button.  

\begin{verbatim} 
ts =IPlot.TrajectorySummary(ds, ds.getLabeling('origins')) 
\end{verbatim}

\begin{figure}
  \begin{center}
    \includegraphics[scale=.25]{graphics/simpleTS}
  \end{center}
  \caption{A trajectory summary of the synthetic dataset divided by the origins labeling.}
  \label{fig:ts}
\end{figure}

\subsubsection{Confusion Matrices}

Comparitive analysis of clustering results can be quite a powerful tool
\cite{hart_etal03}.  Briefly a confusion matrix describes the cardinalities of
all pairwise intersections between two clustering results or partitions.  We've
defined a confusionArray to be an array of sets containing all pairwise
intersections between two clustering results or partitions.  The software blurs
this distinction and refers to a confusion array as a confusion matrix.  There
are two basic ways of working with confusion matrices: 1) graphically, 2)
computationally.

Before we can create a confusion matrix we need two labelings to compare, and
insofar we only have one.  We'll generate a quick clustering of the data.  The
details on clustering is described in more detail below and in the pyMLX user
tutorial.  On purpose I'm using a k=4 to forse a slight description between the
two partitions for demonstrative purposes.


\begin{verbatim}
# first we create a parameters dictionary.  
# In this case, we're using DiagEM, so here are the parameters
p = {'distance_metric': 'euclidean',
     'init_method': 'random_sample',
     'samples': 1,
     'k': 4,
     'k_strict': 'false',
     'num_iterations': 50,
     'seed': 664361265
    }
# first we create an algorithm wrapper
aw = wrapper.DiagEM(ds, p)
aw.run()

# now I like to make the resulting labeling global
l = labelings.castToGlobalLabeling(aw.getLabeling(), 'Diagem')
\end{verbatim}


First I'll describe the visualization of the confusion matrix (fig
\ref{fig:simpleCM}). followed below by some of the backend API calls that can
be useful when setting up analysis chains or scripts.  Middle clicking on any
of these graphs will bring up a detailed version of the plot and all the
previous interactions descriped above in the datasetPlot section apply to these
graphs as well.

\begin{verbatim}
cm = IPlot.ConfusionMatrixSummary(ds, 
                                  ds.getLabeling('origins'), 
                                  ds.getLabeling('Diagem'))
\end{verbatim}

\begin{figure}
  \begin{center}
    \includegraphics[scale=.25]{graphics/simpleCM}
  \end{center}
  \caption{Confusion Matrix of origns vs EM/MoDG clustering results. White boxes
  across the columns show a mean (blue) and standard deviation (red) summary
  plots for each cluster in the EM/MoDG.  White boxes aligned down the rows are
  the mean and standard deviation summary plots for each class in the origins 
  Classification.  The upper left corner of each plot displays the feature
  (gene vector) count.  Each cell ($C{i,j}$) within the confusion array shows a
  mean and standard deviation summary plot for the intersection set of genes
  that are in common between origins class $i$ and the EM/MoDG result cluster
  $j$.  The background of each plot is colored according to a heat-map that
  registers the proportionate number of genes in the cell compared with the
  corresponding class in the EM/MoDG result (coloring can also be specified
  relative to origins input or other functions).  Intersection cells with dark
  outlines indicate are optimal pairings between the two data partitions, as
  determined during the LA calculation}
  \label{fig:simpleCM} 
\end{figure}

Now lets compare the 'diagem' labeling with the 'origins' labeling using the
API calls and no visualization.  

\begin{verbatim}
# first we create a confusion matrix, then we populate it.
cm = score.ConfusionMatrix()
cm.createConfusionMatrixFromLabeling(ds.getLabeling('origins'), 
                                     ds.getLabeling('Diagem'))
\end{verbatim}

Now we can start to integate the matrix using the confusion matrix's methods.  

\begin{verbatim}
# this returns the confusion matrix (cardinalities)
cm.getCounts()

## Now we can quantify the simularity
# by linear assignment
cm.linearAssignment()
# by normalized mutual information
cm.NMI()
# or the by transposed NMI
cm.transposedNMI()

# We can also determine the adjancy mapping
cm.getAdjacencyList()

\end{verbatim}

\subsubsection{ROC Analysis}

Receiver Operator Characteristic  (ROC) analysis provides a way to assess the
degree a cluster overlaps with other clusters.  Again IPlot impliments a nice
visualization and the score module impliments the underlying algorithm and
functionality. I'll demonstrate the IPlot visualization \ref{fig:simpleROC}
Note that in this example dataset things are a little too perfect.  

\begin{verbatim}
IPlot.rocPlot(ds, ds.getLabeling('0origins'), 'nonResponders')
\end{verbatim}

\begin{figure}
  \begin{center}
    \includegraphics[scale=.25]{graphics/simpleROC}
  \end{center}
  \caption{A curve and distance histograms} 
  \label{fig:simpleROC}
\end{figure}

\subsubsection{Histograms}

Often it array analysis understanding the distribution of data is critical -
further being able to explore either outliers or the center of the distribution
is an often desired task.  The compClust.Histogram module coupled with the
IPlot.HistogramPlotter provides a very flexiable system for looking at data
distributions.

Effectively the IPlot.HistogramPlotter simple plots a bar graph of counts based
on a labeling.  For example to get a feel for the cluster sizes of the origins
labeling we can simple do this \ref{fig:sizes}:

\begin{verbatim}
g = IPlot.HistogramPlotter(ds.getLabeling('origins'))
# we can reorder the elements by size 
g.plot(sortBy='size')
# or we sort by label name
g.plot(sortBy='label')
\end{verbatim}

\begin{figure}
  \begin{center}
    \includegraphics[scale=.25]{graphics/sizes}
  \end{center}
  \caption{Bar graph of cluster sizes} 
  \label{fig:sizes}
\end{figure}

As you can see any labeling can be passed to the IPlot.HistogramPlotter and it
will simple create a bar graph of sizes.  You can also middle click on any bar
to open a detailed trajectory summmary of genes contained within that group.
If you need to change the attributes of that opened plot, there is a handle for
it as a member variable selectedPlot and selectedView.  

To actually create a histogram then we need to create a labeling which maps bin
names to data elements.  By creating labeling to represent the histogram all
the functionality of labelings are available for interogation of the histogram
including confusion matrixes.  In the compClust.util.Histogram module several
methods for creating histogram labelings are provided.  The labelings can be
attached across rows or columns.  Below I'll demonstrate building histograms
across the rows.

\begin{verbatim}
# to create a histogram bining of RowData
h =  Histogram.binOnColData(ds, 4, 20) 
# now we can plot this:
IPlot.HistogramPlotter(h)
\end{verbatim}

Figure \ref{fig:binOnColData} shows the result above.  We can see that there is
effectively 3 peaks which correspond with the origins of the cluster.  Using
the IPlot.ConfsionMatrixSummary we can inspect the relationships of these bins
with the origin labelings.

\begin{verbatim}
h = labelings.castToGlobalLabeling(h)
cm = IPlot.ConfusionMatrixSummary(ds, 
                                  ds.getLabeling('origins'),
                                  h, 
                                  l2Order=listOps.sort(h.getLabels()))
\end{verbatim}

Note above we use the l1Order optional argument to insist that the histogram
labels are arranged according to their value as opposed to the default ordering
which is based on cluster size.

\begin{figure}
  \begin{center}
    \includegraphics[scale=.25]{graphics/histoVsOrigins}
  \end{center}
  \caption{Confusion Matrix Summary of a histogram of values column 4 of the
  dataset vs origins. Notice here each bin in the histogram corresponds to one
  and only one origin.} 
  \label{fig:histoVsOrigin} \end{figure}


Next we'll build a histogram based on a row function.  This becomes a little
tricky and will involve the use of a lambda funtion, this is just like we did
in the colorMapper.setColorByRowFunction. The function will require two
parameters a dataset and a row and it will return a single numeric value.

\begin{verbatim}
h = Histogram.binOnRowFunction(ds, lambda ds,row: MLab.mean(ds.getRowData(row)), 20)
IPlot.HistogramPlotter(h)
\end{verbatim}

Again this looks identical to plot of cluster sizes for the origins labeling.  

\subsubsection{PCA Explorer}

This PCAExplorer tool provides tools to better understand the PCA space of the
data.  Upon initialization of the pcaExplorer you'll see 3 windows.  A PCA
scatter plot, a list of eigen vectors (or sometimes they're called eigen genes)
and a native space projection.   The PCA space represents a projection of the
data into a space where each dimension or basis represents a linear combination
of real dimensions in the dataset.  The order of these PCA dimensions are
ranked according to the amount of variance they explain or capture.  The eigen
vector window shows what each PCA dimension is representing.  The red ``X'' in
the PCA plot indicates the position in that particular PCA projection that the
native-space projection vector is illustrating.  You can either middle click on
the PCA plot or use the sliders next to each eigen vector to move the
indicator ``X'' around in the PCA space.  Note, if you are moving the ``X''
along a PCA dimension that is not be plotted (by default the first two) the
native-space projection will change but the ``X'' will not move.  That is
because you move orthoganolly to the two displayed dimensions.  Use the pull
down menus on the bottom of the PCA display to select the PCA dimensions to
plot.  Often the first PCA dimension describes uninteresting aspects of the
data, such as magnitude of expression.

\section{Running with your data}

Hopefully in reading through the first section you've become comfortable with
the pyMLX schema and IPlot functionality.   

\subsection{Loading Data}

The easiest way to load data is to create a tab-deliminated text file with each
line containing just the expression measurements.

\begin{verbatim}
1.2 .8  0.2  0.4  -1.2  -0.4
-4  0.2  1.2 -0.02  .05 -0.1
\end{verbatim}

Now create labeling files with one label per line corresponding to the data file

\begin{verbatim}
geneA
geneB
\end{verbatim}

You can do the same thing for column labels, one label per line correpsonding to the columns

\begin{verbatim}
time0
time1
time2
time3
\end{verbatim}

If you had stored your data in a file called data.dat and the gene names in a
file called genes.rlab and the time labels in a file called times.clab (the
extension names are only for convention and aren't used by any of the software,
but helps keep track of things.  dat for dataset files, rlab for row labels and
clab for column labels).  We can now instantiate a pyMLX dataset with labelings
with thing following code:

\begin{verbatim}
ds = datasets.Dataset('data.dat')
 
genes = labelings.GlobalWrapper(ds, 'gene names')
genes.labelRows('genes.rlab')

times = labelings.GlobalWrapper(ds, 'time')
times.labelCols('times.clab')
\end{verbatim}

It is important to note that the Dataset constructor has is ``smart'' and can
load a dataset from a file or from a numeric array or list of lists as we saw
above.  The labeling.labelRows or labeling.labelCols method is also smart and
will work with a filename or a list of labels for each row.  If it is working
with a filename, then each label is assumed to be a string and sometimes you
may want them to be either int or float labels.  This can be easily
accomplished using a map.  The map function in python apples a function to a
list, so its sytex is map(function, list).  In the below example imagine we
have a file named seconds which contains the number of seconds past some
treatment for each column in a dataset.   Below the list is created by opening
the file and then immediately invoking the readlines command which returns a
list of strings where each item in the list is the string representing a line
in the file.  The map function simply casts each line into a float creating a
list of floats which is then passed to the labelCols constructor.

\begin{verbatim} 
secs =labelings.GlobalWrapper(ds, 'seconds')
secs.labelCols(map(float,open('seconds.clab').readlines())) 
\end{verbatim}



\end{document}
