singa-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From build...@apache.org
Subject svn commit: r966886 - in /websites/staging/singa/trunk/content: ./ docs/data.html docs/distributed-training.html docs/examples.html docs/frameworks.html docs/model-config.html docs/programming-guide.html docs/train-one-batch.html
Date Sat, 26 Sep 2015 12:10:19 GMT
Author: buildbot
Date: Sat Sep 26 12:10:18 2015
New Revision: 966886

Log:
Staging update by buildbot for singa

Modified:
    websites/staging/singa/trunk/content/   (props changed)
    websites/staging/singa/trunk/content/docs/data.html
    websites/staging/singa/trunk/content/docs/distributed-training.html
    websites/staging/singa/trunk/content/docs/examples.html
    websites/staging/singa/trunk/content/docs/frameworks.html
    websites/staging/singa/trunk/content/docs/model-config.html
    websites/staging/singa/trunk/content/docs/programming-guide.html
    websites/staging/singa/trunk/content/docs/train-one-batch.html

Propchange: websites/staging/singa/trunk/content/
------------------------------------------------------------------------------
--- cms:source-revision (original)
+++ cms:source-revision Sat Sep 26 12:10:18 2015
@@ -1 +1 @@
-1705429
+1705434

Modified: websites/staging/singa/trunk/content/docs/data.html
==============================================================================
--- websites/staging/singa/trunk/content/docs/data.html (original)
+++ websites/staging/singa/trunk/content/docs/data.html Sat Sep 26 12:10:18 2015
@@ -489,6 +489,7 @@
         <div id="bodyColumn"  class="span10" >
                                   
             <h1>Data Preparation</h1>
+<hr />
 <p>To submit a training job, users need to convert raw data (e.g., images, text documents)
into SINGA recognizable <a href="../api/classsinga_1_1Record.html">Record</a>s.
SINGA uses <a href="layer#data-layers">data layers</a> to load these records into
memory and uses <a href="layer#parser-layers">parser layers</a> to parse features
(e.g., image pixels and labels) from these <tt>Record</tt>s. <tt>Record</tt>s
could be stored in a file, a database, or HDFS, as long as there is a corresponding <a
href="../api/classsinga_1_1DataLayer.html">DataLayer</a>.</p>
 <div class="section">
 <h2><a name="DataShard"></a>DataShard</h2>

Modified: websites/staging/singa/trunk/content/docs/distributed-training.html
==============================================================================
--- websites/staging/singa/trunk/content/docs/distributed-training.html (original)
+++ websites/staging/singa/trunk/content/docs/distributed-training.html Sat Sep 26 12:10:18
2015
@@ -489,6 +489,7 @@
         <div id="bodyColumn"  class="span10" >
                                   
             <h1>Distributed Training</h1>
+<hr />
 <p>SINGA is designed for distributed training of large deep learning models with huge
amount of training data.</p>
 <p>Here we introduce distrbuted SINGA in following aspects:</p>
 

Modified: websites/staging/singa/trunk/content/docs/examples.html
==============================================================================
--- websites/staging/singa/trunk/content/docs/examples.html (original)
+++ websites/staging/singa/trunk/content/docs/examples.html Sat Sep 26 12:10:18 2015
@@ -489,6 +489,7 @@
         <div id="bodyColumn"  class="span10" >
                                   
             <h1>Example Models</h1>
+<hr />
 <p>Different models are provided as examples to help users get familiar with SINGA.
<a href="neural-net.html">Neural Network</a> gives details on the models that
are supported by SINGA.</p>
 <div class="section">
 <div class="section">

Modified: websites/staging/singa/trunk/content/docs/frameworks.html
==============================================================================
--- websites/staging/singa/trunk/content/docs/frameworks.html (original)
+++ websites/staging/singa/trunk/content/docs/frameworks.html Sat Sep 26 12:10:18 2015
@@ -489,9 +489,9 @@
         <div id="bodyColumn"  class="span10" >
                                   
             <h1>Distributed Training Framework</h1>
+<hr />
 <div class="section">
-<div class="section">
-<h3><a name="Cluster_Topology_Configuration"></a>Cluster Topology Configuration</h3>
+<h2><a name="Cluster_Topology_Configuration"></a>Cluster Topology Configuration</h2>
 <p>Here we describe how to configure SINGA&#x2019;s cluster topology to support
different distributed training frameworks. The cluster topology is configured in the <tt>cluster</tt>
field in <tt>JobProto</tt>. The <tt>cluster</tt> is of type <tt>ClusterProto</tt>:</p>
 
 <div class="source">
@@ -520,12 +520,12 @@
 <li><tt>server_worker_separate</tt>:  separate servers and workers in different
processes.</li>
 </ul></div>
 <div class="section">
-<h3><a name="Different_Training_Frameworks"></a>Different Training Frameworks</h3>
+<h2><a name="Different_Training_Frameworks"></a>Different Training Frameworks</h2>
 <p>In SINGA, worker groups run asynchronously and workers within one group run synchronously.
Users can leverage this general design to run both <b>synchronous</b> and <b>asynchronous</b>
training frameworks. Here we illustrate how to configure popular distributed training frameworks
in SINGA.</p>
 <p><img src="../images/frameworks.png" style="width: 800px" alt="" /> 
 <p><b> Fig.1 - Training frameworks in SINGA</b></p>
 <div class="section">
-<h4><a name="Sandblaster"></a>Sandblaster</h4>
+<h3><a name="Sandblaster"></a>Sandblaster</h3>
 <p>This is a <b>synchronous</b> framework used by Google Brain. Fig.2(a)
shows the Sandblaster framework implemented in SINGA. Its configuration is as follows:</p>
 
 <div class="source">
@@ -539,7 +539,7 @@
 </pre></div></div>
 <p>A single server group is launched to handle all requests from workers. A worker
computes on its partition of the model, and only communicates with servers handling related
parameters.</p></div>
 <div class="section">
-<h4><a name="AllReduce"></a>AllReduce</h4>
+<h3><a name="AllReduce"></a>AllReduce</h3>
 <p>This is a <b>synchronous</b> framework used by Baidu&#x2019;s DeepImage.
Fig.2(b) shows the AllReduce framework implemented in SINGA. Its configuration is as follows:</p>
 
 <div class="source">
@@ -553,7 +553,7 @@
 </pre></div></div>
 <p>We bind each worker with a server on the same node, so that each node is responsible
for maintaining a partition of parameters and collecting updates from all other nodes.</p></div>
 <div class="section">
-<h4><a name="Downpour"></a>Downpour</h4>
+<h3><a name="Downpour"></a>Downpour</h3>
 <p>This is a <b>asynchronous</b> framework used by Google Brain. Fig.2(c)
shows the Downpour framework implemented in SINGA. Its configuration is as follows:</p>
 
 <div class="source">
@@ -567,7 +567,7 @@
 </pre></div></div>
 <p>Similar to the synchronous Sandblaster, all workers send requests to a global server
group. We divide workers into several worker groups, each running independently and working
on parameters from the last <i>update</i> response.</p></div>
 <div class="section">
-<h4><a name="Distributed_Hogwild"></a>Distributed Hogwild</h4>
+<h3><a name="Distributed_Hogwild"></a>Distributed Hogwild</h3>
 <p>This is a <b>asynchronous</b> framework used by Caffe. Fig.2(d) shows
the Distributed Hogwild framework implemented in SINGA. Its configuration is as follows:</p>
 
 <div class="source">
@@ -579,7 +579,7 @@
     server_worker_separate: false
 }
 </pre></div></div>
-<p>Each node contains a complete server group and a complete worker group. Parameter
updates are done locally, so that communication cost during each training step is minimized.
However, the server group must periodically synchronize with neighboring groups to improve
the training convergence.</p></div></div></div>
+<p>Each node contains a complete server group and a complete worker group. Parameter
updates are done locally, so that communication cost during each training step is minimized.
However, the server group must periodically synchronize with neighboring groups to improve
the training convergence.</p></div></div>
                   </div>
             </div>
           </div>

Modified: websites/staging/singa/trunk/content/docs/model-config.html
==============================================================================
--- websites/staging/singa/trunk/content/docs/model-config.html (original)
+++ websites/staging/singa/trunk/content/docs/model-config.html Sat Sep 26 12:10:18 2015
@@ -489,12 +489,12 @@
         <div id="bodyColumn"  class="span10" >
                                   
             <h1>Model Configuration</h1>
+<hr />
 <p>SINGA uses the stochastic gradient descent (SGD) algorithm to train parameters of
deep learning models. For each SGD iteration, there is a <a href="architecture.html">Worker</a>
computing gradients of parameters from the NeuralNet and a <a href="">Updater</a>
updating parameter values based on gradients. Hence the model configuration mainly consists
these three parts. We will introduce the NeuralNet, Worker and Updater in the following paragraphs
and describe the configurations for them. All model configuration is specified in the model.conf
file in the user provided workspace folder. E.g., the <a class="externalLink" href="https://github.com/apache/incubator-singa/tree/master/examples/cifar10">cifar10
example folder</a> has a model.conf file.</p>
 <div class="section">
+<h2><a name="NeuralNet"></a>NeuralNet</h2>
 <div class="section">
-<h3><a name="NeuralNet"></a>NeuralNet</h3>
-<div class="section">
-<h4><a name="Uniform_model_neuralnet_representation"></a>Uniform model
(neuralnet) representation</h4>
+<h3><a name="Uniform_model_neuralnet_representation"></a>Uniform model
(neuralnet) representation</h3>
 <p><img src="../images/model-categorization.png" style="width: 400px" alt="" />
Fig. 1: Deep learning model categorization</img></p>
 <p>Many deep learning models have being proposed. Fig. 1 is a categorization of popular
deep learning models based on the layer connections. The <a class="externalLink" href="https://github.com/apache/incubator-singa/blob/master/include/neuralnet/neuralnet.h">NeuralNet</a>
abstraction of SINGA consists of multiple directly connected layers. This abstraction is able
to represent models from all the three categorizations.</p>
 
@@ -540,7 +540,7 @@ LayerProto layer_proto_;  // layer confi
 <div class="source"><pre class="prettyprint">Param weight;
 </pre></div></div></div>
 <div class="section">
-<h4><a name="Configure_the_structure_of_a_NeuralNet_instance"></a>Configure
the structure of a NeuralNet instance</h4>
+<h3><a name="Configure_the_structure_of_a_NeuralNet_instance"></a>Configure
the structure of a NeuralNet instance</h3>
 <p>To train a deep learning model, the first step is to write the configurations for
the model structure, i.e., the layers and connections for the NeuralNet. Like <a class="externalLink"
href="http://caffe.berkeleyvision.org/">Caffe</a>, we use the <a class="externalLink"
href="https://developers.google.com/protocol-buffers/">Google Protocol Buffer</a>
to define the configuration protocol. The <a class="externalLink" href="https://github.com/apache/incubator-singa/blob/master/src/proto/model.proto">NetProto</a>
specifies the configuration fields for a NeuralNet instance,</p>
 <p>message NetProto {  repeated LayerProto layer = 1;  &#x2026; }</p>
 <p>The configuration is then</p>
@@ -604,7 +604,7 @@ layer {
 <p>Fig. 4 shows the model structure corresponding to the neural network configuration
for the <a class="externalLink" href="https://github.com/apache/incubator-singa/blob/master/examples/cifar10/model.conf">deep
convolutional model</a>.</p>
 <p><img src="../images/dcnn-cifar10.png" style="width: 200px" alt="" /> Fig.
4: Neural network structure for the example DCNN model (cifar10)</img></p></div></div>
 <div class="section">
-<h3><a name="Worker"></a>Worker</h3>
+<h2><a name="Worker"></a>Worker</h2>
 <p>At the beginning, the Work will initialize the values of Param instances of each
layer either randomly (according to user configured distribution) or loading from a <a
href="">checkpoint file</a>. For each training iteration, the worker visits layers
of the neural network to compute gradients of Param instances of each layer. Corresponding
to the three categories of models, there are three different algorithm to compute the gradients
of a neural network.</p>
 
 <ol style="list-style-type: decimal">
@@ -668,7 +668,7 @@ layer {
 </pre></div></div>
 <p>All <a href="">Layer implementations</a> must implement the above two
functions.</p></div>
 <div class="section">
-<h3><a name="Updater"></a>Updater</h3>
+<h2><a name="Updater"></a>Updater</h2>
 <p>Once the gradients of parameters are computed, the Updater will update parameter
values. There are many SGD variants for updating parameters, like <a class="externalLink"
href="http://arxiv.org/pdf/1212.5701v1.pdf">AdaDelta</a>, <a class="externalLink"
href="http://www.magicbroom.info/Papers/DuchiHaSi10.pdf">AdaGrad</a>, <a class="externalLink"
href="http://www.cs.toronto.edu/~tijmen/csc321/slides/lecture_slides_lec6.pdf">RMSProp</a>,
<a class="externalLink" href="http://scholar.google.com/citations?view_op=view_citation&amp;hl=en&amp;user=DJ8Ep8YAAAAJ&amp;citation_for_view=DJ8Ep8YAAAAJ:hkOj_22Ku90C">Nesterov</a>
and SGD with momentum. The core functions of the Updater is</p>
 
 <div class="source">
@@ -725,7 +725,7 @@ float GetLearningRate(int step);
 }
 </pre></div></div></div>
 <div class="section">
-<h3><a name="Other_model_configuration_fields"></a>Other model configuration
fields</h3>
+<h2><a name="Other_model_configuration_fields"></a>Other model configuration
fields</h2>
 <p>Some other important configuration fields for training a deep learning model is
listed:</p>
 
 <div class="source">
@@ -741,7 +741,7 @@ optional int32 checkpoint_frequency = 34
 // checkpoint path
 optional bool resume = 36 [default = false];
 </pre></div></div>
-<p>The pages of <a href="checkpoint.html">checkpoint and restore</a> has
more details on related fields.</p></div></div>
+<p>The pages of <a href="checkpoint.html">checkpoint and restore</a> has
more details on related fields.</p></div>
                   </div>
             </div>
           </div>

Modified: websites/staging/singa/trunk/content/docs/programming-guide.html
==============================================================================
--- websites/staging/singa/trunk/content/docs/programming-guide.html (original)
+++ websites/staging/singa/trunk/content/docs/programming-guide.html Sat Sep 26 12:10:18 2015
@@ -491,6 +491,7 @@
         <div id="bodyColumn"  class="span10" >
                                   
             <h1>Programming Guide</h1>
+<hr />
 <p>To submit a training job, users must provide the configuration of the four components
shown in Figure 1:</p>
 
 <ul>

Modified: websites/staging/singa/trunk/content/docs/train-one-batch.html
==============================================================================
--- websites/staging/singa/trunk/content/docs/train-one-batch.html (original)
+++ websites/staging/singa/trunk/content/docs/train-one-batch.html Sat Sep 26 12:10:18 2015
@@ -489,6 +489,7 @@
         <div id="bodyColumn"  class="span10" >
                                   
             <h1>Train-One-Batch</h1>
+<hr />
 <p>For each SGD iteration, every worker calls the <tt>TrainOneBatch</tt>
function to compute gradients of parameters associated with local layers (i.e., layers dispatched
to it). SINGA has implemented two algorithms for the <tt>TrainOneBatch</tt> function.
Users select the corresponding algorithm for their model in the configuration.</p>
 <div class="section">
 <h2><a name="Basic_user_guide"></a>Basic user guide</h2>



Mime
View raw message