// doc/dnn3_code_data_types.dox


// Copyright 2015   Johns Hopkins University (author: Daniel Povey)

// See ../../COPYING for clarification regarding multiple authors
//
// Licensed under the Apache License, Version 2.0 (the "License");
// you may not use this file except in compliance with the License.
// You may obtain a copy of the License at

//  http://www.apache.org/licenses/LICENSE-2.0

// THIS CODE IS PROVIDED *AS IS* BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
// KIND, EITHER EXPRESS OR IMPLIED, INCLUDING WITHOUT LIMITATION ANY IMPLIED
// WARRANTIES OR CONDITIONS OF TITLE, FITNESS FOR A PARTICULAR PURPOSE,
// MERCHANTABLITY OR NON-INFRINGEMENT.
// See the Apache 2 License for the specific language governing permissions and
// limitations under the License.

namespace kaldi {
namespace nnet3 {

/**
  \page dnn3_code_data_types Data types in the "nnet3" setup.

  - Up: \ref dnn3
  - Next: \ref dnn3_code_compilation

  \section dnn3_dt_problem  Objectives and background

  The previous \ref dnn1 "nnet1" and \ref dnn2 "nnet2" setups are based on a Component
  object, where a neural net is a stack of Components.  Each Component
  corresponds to a layer of the neural net, with the wrinkle that we represent a
  single layer as an affine transform followed by a nonlinearity, so there are
  two Components per layer.  These old Components had a Propagate function and a Backprop
  function, both geared towards operating on minibatches, as well as other functions.

  Both setups supported more than just a linear sequence of nonlinearities, but
  in different ways.  In the \ref dnn1 "nnet1" code, networks with more complex topologies
  were represented by components-within-components: for instance there was a
  ParallelComponent, which could contain multiple sequences of Components inside itself.
  Also, LSTMs were implemented at the C++ level by defining a Component to implement
  the LSTM.  In the \ref dnn2 "nnet2" code, the network had a notion of a time index to support
  splicing of features across time as part of the framework directly.  This enabled
  us to support TDNNs by including splicing at intermediate layers of the network.

  The objective in the nnet3 code is to support the kinds of topologies that
  both the \ref dnn1 "nnet1" and \ref dnn2 "nnet2" codebases support, and more;
  and to do so in a natural, config-file-driven way that should not require
  coding to support most interesting new ideas.

  \section dnn3_dt_outline  Outline of approach

  In nnet3, instead of just a sequence of Components we have a general graph structure.
  An "nnet3" neural net (class Nnet) consists of:
     - A list of named Components, in no particular order
     - A graph structure containing "glue" that specifies how the Components fit together.
  .
  The graph refers to the Components by name (this enables certain types of
  parameter sharing).  Part of what this "glue" does is to enable things like
  recurrent neural nets (RNN), where something on time t can depend on time t-1.
  It also enables us to handle edge effects in a natural way (e.g. the kind of
  edge effect that happens in an RNN when we reach the beginning of the file).

  An example config-file representation of the Components and the graph
  is given here (we will discuss it in more detail later):
\verbatim
# First the components
component name=affine1 type=NaturalGradientAffineComponent input-dim=48 output-dim=65
component name=relu1 type=RectifiedLinearComponent dim=65
component name=affine2 type=NaturalGradientAffineComponent input-dim=65 output-dim=115
component name=logsoftmax type=LogSoftmaxComponent dim=115
# Next the nodes
input-node name=input dim=12
component-node name=affine1_node component=affine1 input=Append(Offset(input, -1), Offset(input, 0), Offset(input, 1), Offset(input, 2))
component-node name=nonlin1 component=relu1 input=affine1_node
component-node name=affine2 component=affine2 input=nonlin1
component-node name=output_nonlin component=logsoftmax input=affine2
output-node name=output input=output_nonlin
\endverbatim

  The graph and the Components, together with the inputs provided and the
  outputs requested, will be used to construct a "computation graph" (class
  ComputationGraph), which is an important stage in the compilation process.
  The computation graph will be an acyclic graph where the nodes corresponded to
  vector-valued quantities.  Each node in that acyclic graph will be identified
  by the node in the neural network graph (i.e. the layer of the network) together
  with a number of additional indexes: time (t), an index (n) that
  indicates the example within the minibatch (e.g. 0 through 511 for
  a 512-example minibatch), plus an "extra" index (x) that may eventually be
  useful in convolutional approaches but is usually zero for now.

  To formalize the above, we define an Index is a tuple (n, t, x).  We will also
  define a \ref Cindex as a tuple (node-index, Index), where the node-index is
  the index corresponding to a node in the neural network (i.e., the layer).
  The actual computation that we create is expressed during compilation as a
  directed acyclic graph on Cindexes.

  The process of using a neural net (whether training or decoding) is as follows:
   - The user supplies a ComputationRequest saying what indexes (e.g. time-indexes)
     of what inputs are available and what outputs are requested
   - The ComputationRequest together with a neural network
      is compiled into a sequence of commands as an NnetComputation
   - The NnetComputation is further optimized for speed (think of this is a compiler
     optimization, like gcc's -O flag).
   - Class NnetComputer is responsible
     for receiving matrix-valued input, evaluating the NnetComputation, and supplying
     matrix-valued output.  Think of this as the run-time of a very limited interpreted
     language.

  \section dnn3_dt_data_structures  Basic data structures in nnet3

   \subsection dnn3_dt_datastruct_index Indexes

  As mentioned above, an Index is a tuple (n, t, x), where n is the index within
  the minibatch, t is the time index, and x is a placeholder
  index for future use that will usually be zero for now.  In the neural net computation
  there are vector-valued quantities that we deal with (say, a 1024-dimensional quantity
  corresponding to a hidden layer activation).  In the actual neural net computation,
  1024 would become the number of columns of a matrix, and there is a one-to-one
  correspondence between Indexes on the one hand, and rows of the matrix, on the other.
  The nnet3 framework thus differs from packages like Theano, where tensor operations are
  used throughout: instead, we operate efficiently on tensors by packing them into matrices.
  This enables us to use optimized BLAS operations.

  As an example of Indexes: if we are training the very simplest kind of feedforward
  network, the Indexes would probably only vary in the "n" dimension and we could set the
  "t" values arbitrarily to zero, so the Indexes would look like
  \verbatim
   [ (0, 0, 0)  (1, 0, 0)  (2, 0, 0) ... ]
  \endverbatim
  On the other hand, if we were to decode a single utterance using the same type of network,
  the Indexes would only vary in the "t" dimension, so we'd have
  \verbatim
   [ (0, 0, 0)  (0, 1, 0)  (0, 2, 0) ... ]
  \endverbatim
  corresponding to the rows of the matrix.  In a network that uses temporal context, for
  the early layers we would need different "t" values even during training, so we might
  encounter lists of Indexes that vary in both "n" and "t", e.g:
  \verbatim
   [ (0, -1, 0)  (0, 0, 0)  (0, 1, 0) (1, -1, 0) (1, 0, 0) (1, 1, 0) ... ]
  \endverbatim
  Struct Index has a default sorting operator that sorts first by n, then t, then x,
  so we'd normally order them as above.  When you see vectors of Indexes printed out
  in code you'll often see them in compressed form, where the "x" index (if zero) is
  omitted, and ranges of "t" values are expressed compactly, so the above vector
  might be written as
  \verbatim
   [ (0, -1:1) (1, -1:1) ... ]
  \endverbatim


   \subsection dnn3_dt_datastruct_cindex  Cindexes

  A \ref Cindex is a pair (int32, Index), where the int32 corresponds to the index of a node
  in a neural network.  As mentioned above, a \ref Nnet "neural network" consists of a collection of
  named Components and a kind of graph on "nodes", and the nodes have indexes.
  Cindexes are used during the compilation process, and
  they correspond to the nodes of a "computation graph" corresponding to a specific
  neural net computation.  There is a correspondence
  between a \ref Cindex and the output of a particular node, and there will generally (at least,
  before optimization) be a one-to-one correspondence between Cindexes and rows
  of matrices in the compiled computation.  We previously mentioned that there is a correspondence
  between Indexes and rows of matrices; the difference is that a Cindex will tell us which
  matrix, in addition to which row of that matrix.   For example, assuming there is a node called "affine1"
  in the graph, with output dimension 1000 and numbered 2 in the list of nodes, the
  \ref Cindex (2, (0, 0, 0)) would correspond to some row of a matrix of column dimension 1000,
  that is allocated as the output of the "affine1" component.

   \subsection dnn3_dt_datastruct_computation_graph  ComputationGraph

  A ComputationGraph represents a directed graph on Cindexes, where each Cindex
  has a list of other Cindexes it depends on.  In a simple feedforward
  structure, the graph will have a simple topology with multiple linear
  structures where, [using the names not the integers for clarity], we might
  have (nonlin1, (0, 0, 0)) depending on (affine1, (0, 0, 0)), and (nonlin1, (1,
  0, 0)) depending on (affine1, (1, 0, 0)), and so on.  In the ComputationGraph
  and elsewhere you will see integers called cindex_ids.  Each cindex_id is an
  index into an array of Cindexes stored in the graph, and it identifies a
  particular Cindex; cindex_ids are used for efficiency, as a single integer is
  easier to work with than a Cindex.

   \subsection dnn3_dt_datastruct_computation_request ComputationRequest

  A ComputationRequest identifies a set of named inputs and output nodes, each
  with an associated list of \ref Index "Indexes".  For input nodes, the list
  identifies which Indexes are to be provided to the computation; for
  output nodes, it identifies which Indexes are requested to be
  computed.  In addition the ComputationRequest contains various flags, such as
  information about which output/input nodes have backprop derivatives
  supplied/requested respectively, and whether model update is to be performed.

  As an example, a ComputationRequest might specify that there is one input-node,
  named "input", and with Indexes <code>[ (0, -1, 0), (0, 0, 0), (0, 1, 0) ]</code>
  provided; and one output-node, named "output", with Indexes
  <code>[ (0, 0, 0) ]</code> requested.  This would make sense if the network
  required one frame of left and right context.  Actually we would typically
  only request individual output frames like this during training; and during
  training we would normally have multiple examples in the minibatch so the "n"
  dimension of the Indexes would vary too.

  The computation of the objective function and its derivatives at the output is
  not part of the core neural network framework; we leave that to the user.  A
  neural network may in general have multiple input and output nodes; this might be
  useful in multi-task learning or in frameworks which process multiple different types
  of input data (e.g. multi-view learning).

  \subsection dnn3_dt_data_struct_computation NnetComputation (brief)

  A NnetComputation represents a specific computation that has been compiled
  from an Nnet and a ComputationRequest.  It contains a sequence of \ref
  NnetComputation::Command "Commands", each of which could be a Propagate
  operation, a matrix copy or add operation, various other simple matrix
  commands such as copying particular rows from one matrix to another; a
  Backprop operation, matrix sizing commands, and so on.  The variables that the
  Computation acts on are a list of matrices, and also submatrices that may
  occupy row or column ranges of a matrix.  A Computation also contains various
  sets of indexes (arrays of integers and so on) that are sometimes required as
  arguments to particular matrix operations.

  We will describe this in more detail below in \ref dnn3_dt_nnet_computation.

  \subsection dnn3_dt_data_struct_computer NnetComputer

 The NnetComputer object is responsible for actually executing the NnetComputation.
 This code for this is actually quite simple (chiefly a loop with a switch statement)
 since most of the complexity happens during compilation and optimization of
 the NnetComputation.


  \section dnn3_dt_nnet  Neural networks in nnet3

 The previous section should have given you a high-level overview of how the
 framework fits together.  In this section we will go into a little more detail
 on the structure of the neural network itself, and how we glue Components together
 and express things like dependencies on an input from time t-1.

 \subsection dnn3_dt_nnet_component_basics Components (the basics)

 A Component, in nnet3, is an object with Propagate and Backprop functions.  It
 may contain parameters (say, for an affine layer) or it may just implement a
 fixed nonlinearity, such as a Sigmoid component.  The most important part of
 the interface of a Component is as follows:
\verbatim
class Component {
 public:
  virtual void Propagate(const ComponentPrecomputedIndexes *indexes,
                         const CuMatrixBase<BaseFloat> &in,
                         CuMatrixBase<BaseFloat> *out) const = 0;
  virtual void Backprop(const std::string &debug_info,
                        const ComponentPrecomputedIndexes *indexes,
                        const CuMatrixBase<BaseFloat> &in_value,
                        const CuMatrixBase<BaseFloat> &out_value,
                        const CuMatrixBase<BaseFloat> &out_deriv,
                        Component *to_update, // may be NULL; may be identical
                                              // to "this" or different.
                        CuMatrixBase<BaseFloat> *in_deriv) const = 0;
   ...
};
\endverbatim
 For now, please ignore the <code>const ComponentPrecomputedIndexes *indexes</code> argument.
 A particular Component will have an input dimension and an output dimension, and
 it will generally transform the data "row-by-row".  That is, the in and out matrices in
 Propagate() have the same number of rows, and each row of the input is processed
 to create the corresponding row of the output.  In terms of Indexes, this means
 that the Indexes corresponding to each element of input and output are the same.
 Similar logic holds in the Backprop function.


 \subsection dnn3_dt_nnet_component_properties Components (properties)

A Component has a virtual function "Properties()" that will return a bitmask
value containing various binary flags defined as enum \ref ComponentProperties.
\verbatim
class Component {
  ...
  virtual int32 Properties() const = 0;
  ...
};
\endverbatim
 These properties identify various characteristics of the component, such as
 whether it contains updatable parameters (kUpdatableComponent), whether its
 propagate function supports in-place operation (kPropagateInPlace), and various
 other things.  Many of these are needed by the optimization code so it can know
 which optimizations are applicable.  You'll also notice an enum value
 kSimpleComponent.  If set, then the Component is "simple" which means it
 transforms the data row-by-row as defined above.  Non-simple Components
 may allow inputs and outputs with different numbers of rows, and may need
 to know what indexes are used at the input and output.  The
<code>const ComponentPrecomputedIndexes *indexes</code> argument to Propgate and
 Backprop is only for use by non-simple Components.  For now, please
 assume all Components are simple, because we have not implemented any non-simple
 Components yet, and because they are not required for implementing any of the
 standard methods (RNNs, LSTMs and so on).  Unlike in the \ref dnn2 "nnet2" framework,
 Components are not responsible for implementing things like splicing across frames;
 instead we use \ref dnn3_dt_nnet_descriptor_code "Descriptors" to handle that, as will be explained below.


 \subsection dnn3_dt_nnet_node_outline Neural network nodes (outline)

We previously explained that a neural net is a collection of named Components
together with a graph on "network nodes", but we haven't yet explained what
a "network node" is.  NetworkNode is actually a struct.  A NetworkNode may be
one of four different varieties, defined by the \ref NodeType enum:
\verbatim
enum NodeType { kInput, kDescriptor, kComponent, kDimRange };
\endverbatim
The three most important ones are kInput, kDescriptor and kComponent (kDimRange
is included to support splitting up a node's output into various parts).  The
kComponent nodes are the "meat" of the network, and (at the risk of mixing
metaphors) the Descriptors are the "glue" that holds it together, supporting
things like frame splicing and recurrence.  The kInput nodes are very simple and
just provide a place to dump the provided input and to declare its dimension;
they don't really do anything.  You may be surprised that there is no kOutput
node.  The reason is that output nodes are simply Descriptors.  There is a rule
that each node of type kComponent must be immediately preceded in the list of
nodes, by its "own" node of type kDescriptor; this rule makes the graph
compilation easier.  Thus, a node of type kDescriptor that is not immediately
followed by a kComponent node is bound to be an output node; for convenience,
class Nnet has functions \ref Nnet::IsOutputNode() "IsOutputNode(int32 node_index)"
and \ref Nnet::IsComponentInputNode() "IsComponentInputNode(int32 node_index)"
that can tell these apart.

We will go into neural network nodes in more detail below in \ref dnn3_dt_nnet_node_detail.

 \subsection dnn3_dt_nnet_config Neural network config files

Neural networks can be created from configuration files.  We give a very simple example
here to show how the configuration files relate to the Descriptors.  This network has one
hidden layer and does splicing over time in the first node:
\verbatim
# First the components
component name=affine1 type=NaturalGradientAffineComponent input-dim=48 output-dim=65
component name=relu1 type=RectifiedLinearComponent dim=65
component name=affine2 type=NaturalGradientAffineComponent input-dim=65 output-dim=115
component name=logsoftmax type=LogSoftmaxComponent dim=115
# Next the nodes
input-node name=input dim=12
component-node name=affine1_node component=affine1 input=Append(Offset(input, -1), Offset(input, 0), Offset(input, 1), Offset(input, 2))
component-node name=nonlin1 component=relu1 input=affine1_node
component-node name=affine2 component=affine2 input=nonlin1
component-node name=output_nonlin component=logsoftmax input=affine2
output-node name=output input=output_nonlin
\endverbatim
In the config file there is no reference to descriptors (e.g. no "descriptor-node").
Instead, the "input" field (e.g. <code>Input=Append(....)</code>) is the descriptor.
Each component-node in the config file gets expanded to two nodes: a node of
type kComponent, and an immediately preceding node of type kDescriptor that is
defined by the "input" field.

The config file above doesn't give an example of a dim-range node.  The basic format
of a dim-range node is this (this example would take the first 50 dimensions from the 65 dimensions
of component affine1):
\verbatim
dim-range-node name=dim-range-node1 input-node=affine1_node dim-offset=0 dim=50
\endverbatim

 \subsection dnn3_dt_nnet_descriptor_config Descriptors in config files

  A Descriptor is a very limited type of expression that refers to quantities defined
  in other nodes in the graph.  Descriptors are part of the glue that attaches
  components together they are responsible for things like appending or summing the
  outputs of components so that they can be used as the input for later components.
  In this section we describe Descriptors from the perspective
  of their config file format; below we'll explain how they appear in code.

  The simplest type of Descriptor (the base-case) is just a node name,
  e.g. "affine1" (only nodes of type kComponent or kInput are allowed to appear here,
  to simplify implementation).
  We will list below some types of expression that may appear in Descriptors, but
  please bear in mind that this description will give you a picture of Descriptors that
  is a little more general than the reality; in reality these may only appear in a certain
  hierarchy, which we will describe more precisely further down this page.
\verbatim
# caution, this is a simplification that overgenerates descriptors.
<descriptor>  ::=   <node-name>      ;; node name of kInput or kComponent node.
<descriptor>  ::=   Append(<descriptor>, <descriptor> [, <descriptor> ... ] )
<descriptor>  ::=   Sum(<descriptor>, <descriptor>)
<descriptor>  ::=   Const(<value>, <dimension>)    ;; e.g. Const(1.0, 512)
<descriptor>  ::=   Scale(<scale>, <descriptor>)   ;; e.g. Scale(-1.0, tdnn2)
;; Failover or IfDefined might be useful for time t=-1 in a RNN, for instance.
<descriptor>  ::=   Failover(<descriptor>, <descriptor>)   ;; 1st arg if computable, else 2nd
<descriptor>  ::=   IfDefined(<descriptor>)     ;; the arg if defined, else zero.
<descriptor>  ::=   Offset(<descriptor>, <t-offset> [, <x-offset> ] ) ;; offsets are integers
;; Switch(...) is intended to be used in clockwork RNNs or similar schemes.  It chooses
;; one argument based on the value of t (in the requested Index) modulo the number of
;; arguments
<descriptor>  ::=   Switch(<descriptor>, <descriptor> [, <descriptor> ...])
;; For use in clockwork RNNs or similar, Round() rounds the time-index t of the
;; requested Index to the next-lowest multiple of the integer <t-modulus>,
;; and evaluates the input argument for the resulting Index.
<descriptor>  ::=   Round(<descriptor>, <t-modulus>)  ;; <t-modulus> is an integer
;; ReplaceIndex replaces some <variable-name> (t or x) in the requested Index
;; with a fixed integer <value>.  E.g. might be useful when incorporating
;; iVectors; iVector would always have time-index t=0.
<descriptor>  ::=   ReplaceIndex(<descriptor>, <variable-name>, <value>)
\endverbatim

Now we will describe the actual syntax which the code uses internally, which
differs from the above simplified version because expressions may appear only in
a certain hierarchy.  This syntax also corresponds more closely with the class
names in the real code.  The code that reads Descriptors attempts to normalize
them in as general as possible a way, so that almost all of the above syntax
can be read and converted to the internal representation.
\verbatim
;;; <descriptor> == class Descriptor
<descriptor> ::=  Append(<sum-descriptor>[, <sum-descriptor> ... ] )
<descriptor> ::=  <sum-descriptor>  ;; equivalent to Append() with one arg.
;;; <sum-descriptor> == class SumDescriptor
<sum-descriptor> ::= Sum(<sum-descriptor>, <sum-descriptor>)
<sum-descriptor> ::= Failover(<sum-descriptor>, <sum-descriptor>)
<sum-descriptor> ::= IfDefined(<sum-descriptor>)
<sum-descriptor> ::= Const(<value>, <dimension>)
<sum-descriptor> ::= <fwd-descriptor>
;;; <fwd-descriptor> == class ForwardingDescriptor
;; <t-offset> and <x-offset> are integers.
<fwd-descriptor>  ::=   Offset(<fwd-descriptor>, <t-offset> [, <x-offset> ] )
<fwd-descriptor>  ::=   Switch(<fwd-descriptor>, <fwd-descriptor> [, <fwd-descriptor> ...])
;; <t-modulus> is an integer
<fwd-descriptor>  ::=   Round(<fwd-descriptor>, <t-modulus>)
;; <variable-name> is t or x; <value> is an integer
<fwd-descriptor>  ::=   ReplaceIndex(<fwd-descriptor>, <variable-name>, <value>)
;; <node-name> is the name of a node of type kInput or kComponent.
<fwd-descriptor>  ::=   Scale(<scale>, <node-name>)
<fwd-descriptor>  ::=   <node-name>
\endverbatim
The design of the Descriptors is supposed to be restrictive enough that the
resulting expressions will be fairly easy to compute (and to produce backprop
code for).  They are only supposed to do heavy lifting when it comes to
connecting Components together, while any more interesting or nonlinear
operations are supposed to be carried out in the Components themselves.

Note: if it ever becomes necessary to do a sum or average over a variety of
indexes of unknown length (e.g. all the "t" values in a file), we intend to do
this in a Component - a non-simple Component- rather than using Descriptors.

\subsection dnn3_dt_nnet_descriptor_code Descriptors in code

We'll describe Descriptors in code from the bottom up.  The base-class ForwardingDescriptor
handles the types of Descriptor that will reference just a single value, without
any <code>Append(...)</code> or <code>Sum(...)</code> expressions or the like.  The most important
function in this interface is \ref ForwardingDescriptor::MapToInput() "MapToInput()":
\verbatim
class ForwardingDescriptor {
 public:
  virtual Cindex MapToInput(const Index &output) const = 0;
  ...
 }
\endverbatim
Given a particular requested Index, this function will return a \ref Cindex
(referencing some other node) corresponding to the input value.  The function argument
is an Index rather than a \ref Cindex because the value is never going to depend on
the node-index of the node corresponding to the Descriptor itself.  There
are several derived classes of ForwardingDescriptor, including
SimpleForwardingDescriptor (the base-case, holding just a node index),
OffsetForwardingDescriptor, ReplaceIndexForwardingDescriptor, and so on.

The next level up the hierarchy is class SumDescriptor, which exists to
support the expressions <code>Sum(<desc>, <desc>)</code>, <code>Failover(<desc>, <desc>)</code>,
and <code>IfDefined(<desc>)</code>.  Clearly a request for a given Index to a SumDescriptor
may return several different Cindexes, so the interface we used for
ForwardingDescriptor won't work.  We also need to support optional
dependencies.   Here is how we manage it at the code level:
\verbatim
class SumDescriptor {
 public:
  virtual void GetDependencies(const Index &ind,
                               std::vector<Cindex> *dependencies) const = 0;
  ...
};
\endverbatim
The function GetDependencies appends to <code>dependencies</code> all Cindexes that
are potentially involved in computing this quantity for this Index.  Next
we need to worry about what happens when some of the requested inputs may not be
computable (e.g. because of limited input data or edge effects).
The function \ref SumDescriptor::IsComputable()
"IsComputable()" handles this:
\verbatim
class SumDescriptor {
 public:
  ...
  virtual bool IsComputable(const Index &ind,
                            const CindexSet &cindex_set,
                            std::vector<Cindex> *input_terms) const = 0;
  ...
};
\endverbatim
Here, the <code>CindexSet</code> object is a representation of a set of Cindexes, which in
this context represents "the set of all Cindexes that we know are computable".
If the Descriptor is computable for this Index, the function will return true.
For instance, the expression <code>Sum(X, Y)</code> would only be computable
if <code>X</code> and <code>Y</code> are computable.  If this function is going to
return true, it will also append to "input_terms" only the input Cindexes that
actually appear in the evaluated expression.
For example (and speaking
loosely), in an expression of the form <code>Failover(X, Y)</code>, if <code>X</code> is computable then
only <code>X</code> would be appended to "input_terms", and not <code>Y</code>.

Class Descriptor is the top level of the hierarchy.  It can be thought of
as a vector of SumDescriptors, and note that this vector will usually be of
length one.  Its function is to append things (think of appending vectors), and
it is responsible for the <code>Append(...)</code> syntax.
It has functions \ref Descriptor::GetDependencies() "GetDependencies()"
and \ref Descriptor::IsComputable() "IsComputable()" with the same interface as
SumDescriptor, and also functions such as \ref Descriptor::NumParts() "NumParts()" and
\ref Descriptor::Part() "Part(int32 n)" that allow the user to access the
individual SumDescriptors in its vector.

\subsection dnn3_dt_nnet_node_detail Neural network nodes (detail)

We will now describe neural network nodes in more detail.  As mentioned above,
there are four types of node, as defined by the enum
\verbatim
enum NodeType { kInput, kDescriptor, kComponent, kDimRange };
\endverbatim
The actual NetworkNode is a struct.  To avoid the hassle of pointers and because C++ doesn't
allow unions containing classes, we have a slightly messy layout:
\verbatim
struct NetworkNode {
  NodeType node_type;
  // "descriptor" is relevant only for nodes of type kDescriptor.
  Descriptor descriptor;
  union {
    // For kComponent, the index into Nnet::components_
    int32 component_index;
    // for kDimRange, the node-index of the input node.
    int32 node_index;
  } u;
  // for kInput, the dimension of the input feature.  For kDimRange, the dimension
  // of the output (i.e. the length of the range)
  int32 dim;
  // for kDimRange, the dimension of the offset into the input component's feature.
  int32 dim_offset;
};
\endverbatim
Summarizing the different types of nodes and the members they actually use:
  - kInput nodes use only "dim"
  - kDescriptor nodes use only "descriptor"
  - kComponent nodes use only "component_index", which indexes the components_ array of the Nnet.
  - kDimRange nodes use only "node_index", "dim" and "dim_offset".


\subsection dnn3_dt_nnet_detail Neural network (detail)

We will give a little more detail on class Nnet itself, which stores the entire
neural net.  The easiest way to explain it is just to list the private data
members:
\verbatim
class Nnet {
public:
  ...
private:
  std::vector<std::string> component_names_;
  std::vector<Component*> components_;
  std::vector<std::string> node_names_;
  std::vector<NetworkNode> nodes_;

};
\endverbatim
The component_names_ should have the same size as components_ and the
node_names_ should have the same size as nodes_; these associate names with the
components and nodes.  Note that we automatically assign names to the nodes of
type kDescriptor which precede their corresponding nodes of type kComponent, by
appending "_input" to the corresponding component node's name.  These names
of kDescriptor nodes don't appear in the config-file representation of the neural net.

\section dnn3_dt_nnet_computation NnetComputation (detail)

Another important data type is struct NnetComputation.  This represents a compiled
neural-net computation, containing a sequence of commands together with other
information necessary to interpret them.  Internally it defines a number of types,
including the following enum value:
\verbatim
  enum CommandType {
    kAllocMatrixUndefined, kAllocMatrixZeroed,
    kDeallocMatrix, kPropagate, kStoreStats, kBackprop,
    kMatrixCopy, kMatrixAdd, kCopyRows, kAddRows,
    kCopyRowsMulti, kCopyToRowsMulti, kAddRowsMulti, kAddToRowsMulti,
    kAddRowRanges, kNoOperation, kNoOperationMarker };
\endverbatim
We would like to highlight <code>kPropagate</code>, <code>kBackprop</code> and <code>kMatrixCopy</code>
as self-explanatory examples of commands.
There is a struct Command which represents a single command together with
its arguments.   Most of the arguments are indexes into lists of matrices
and components.
\verbatim
  struct Command {
    CommandType command_type;
    int32 arg1;
    int32 arg2;
    int32 arg3;
    int32 arg4;
    int32 arg5;
    int32 arg6;
  };
\endverbatim
There are also a couple of struct types defined, which are used to store
size information for matrices and submatrices.  A submatrix is a possibly
restricted row and column range of a matrix, like the matlab syntax
<code>some_matrix(1:10, 1:20)</code>:
\verbatim
  struct MatrixInfo {
    int32 num_rows;
    int32 num_cols;
  };
  struct SubMatrixInfo {
    int32 matrix_index;  // index into "matrices": the underlying matrix.
    int32 row_offset;
    int32 num_rows;
    int32 col_offset;
    int32 num_cols;
  };
\endverbatim
The data members of struct NnetComputation include the following:
\verbatim
struct Command {
  ...
  std::vector<Command> commands;
  std::vector<MatrixInfo> matrices;
  std::vector<SubMatrixInfo> submatrices;
  // used in kAddRows, kAddToRows, kCopyRows, kCopyToRows.  contains row-indexes.
  std::vector<std::vector<int32> > indexes;
  // used in kAddRowsMulti, kAddToRowsMulti, kCopyRowsMulti, kCopyToRowsMulti.
  // contains pairs (sub-matrix index, row index)- or (-1,-1) meaning don't
  // do anything for this row.
  std::vector<std::vector<std::pair<int32,int32> > > indexes_multi;
  // Indexes used in kAddRowRanges commands, containing pairs (start-index,
  // end-index)
  std::vector<std::vector<std::pair<int32,int32> > > indexes_ranges;
  // Information about where the values and derivatives of inputs and outputs of
  // the neural net live.
  unordered_map<int32, std::pair<int32, int32> > input_output_info;
  bool need_model_derivative;
  // the following is only used in non-simple Components; ignore for now.
  std::vector<ComponentPrecomputedIndexes*> component_precomputed_indexes;
  ...
};
\endverbatim
The vectors with "indexes" in their name are arguments to matrix functions
such as CopyRows, AddRows and so on, that need vectors of indexes as input
(we will copy these to the GPU card before executing the computation).

More information about the individual commands and the meaning of their
arguments can be found \ref NnetComputation::CommandType "here".


 - Up: \ref dnn3
 - Next: \ref dnn3_code_compilation.


*/

}
}
