Apache SINGA – Layers

SINGA setup

Please refer to the installation page for guidance on installing SINGA.

Starting Zookeeper

SINGA uses zookeeper to coordinate the training. Please make sure the zookeeper service is started before running SINGA.

If you installed the zookeeper using our thirdparty script, you can simply start it by:

- -

#goto top level folder
-cd  SINGA_ROOT
-./bin/zk-service.sh start
-

(./bin/zk-service.sh stop stops the zookeeper).

Otherwise, if you launched a zookeeper by yourself but not used the default port, please edit the conf/singa.conf:

- -

zookeeper_host: "localhost:YOUR_PORT"
-

Running in standalone mode

Running SINGA in standalone mode is on the contrary of running it using cluster managers like Mesos or YARN.

Training on a single node

For single node training, one process will be launched to run SINGA at local host. We train the CNN model over the CIFAR-10 dataset as an example. The hyper-parameters are set following cuda-convnet. More details is available at CNN example.

@@ -344,8 +335,8 @@ cp Makefile.example Makefile make download make create

A training dataset and a test dataset are created under cifar10-train-shard and cifar10-test-shard folder respectively. An image_mean.bin file is also generated, which contains the feature mean of all images.

Since all code used for training this CNN model is provided by SINGA as built-in implementation, there is no need to write any code. Instead, users just execute the running script (../../bin/singa-run.sh) by providing the job configuration file (job.conf). To code in SINGA, please refer to the programming guide.

A training dataset and a test dataset are created respectively. An image_mean.bin file is also generated, which contains the feature mean of all images.

Since all code used for training this CNN model is provided by SINGA as built-in implementation, there is no need to write any code. Instead, users just execute the running script by providing the job configuration file (job.conf). To code in SINGA, please refer to the programming guide.

Training without parallelism

By default, the cluster topology has a single worker and a single server. In other words, neither the training data nor the neural net is partitioned.

@@ -354,23 +345,8 @@ make create

# goto top level folder
 cd ../../
-./bin/singa-run.sh -conf examples/cifar10/job.conf
-

You can list the current running jobs by,

- -

./bin/singa-console.sh list
-
-JOB ID    |NUM PROCS
-----------|-----------
-24        |1
-

Jobs can be killed by,

- -

./bin/singa-console.sh kill JOB_ID
-

Logs and job information are available in /tmp/singa-log folder, which can be changed to other folders by setting log-dir in conf/singa.conf.

Asynchronous parallel training

@@ -391,7 +367,7 @@ cluster { neuralnet { layer { ... - sharddata_conf { + store_conf { random_skip: 5000 } } @@ -401,7 +377,7 @@ neuralnet {

The running command is:

./bin/singa-run.sh -conf examples/cifar10/job.conf
+./singa -conf examples/cifar10/job.conf

Synchronous parallel training

@@ -418,10 +394,26 @@ cluster {

In SINGA, asynchronous training is enabled by launching multiple workers within one worker group. For instance, we can change the original job.conf to have two workers in one worker group as shown above. The workers will run synchronously as they are from the same worker group. This framework is the in-memory sandblaster. The model is partitioned among the two workers. In specific, each layer is sliced over the two workers. The sliced layer is the same as the original layer except that it only has B/g feature instances, where B is the number of instances in a mini-batch, g is the number of workers in a group. It is also possible to partition the layer (or neural net) using other schemes. All other settings are the same as running without partitioning

./bin/singa-run.sh -conf examples/cifar10/job.conf
+./singa -conf examples/cifar10/job.conf

Training in a cluster

Starting Zookeeper

SINGA uses zookeeper to coordinate the training, and uses ZeroMQ for transferring messages. After installing zookeeper and ZeroMQ, you need to configure SINGA with --enable-dist before compiling. Please make sure the zookeeper service is started before running SINGA.

If you installed the zookeeper using our thirdparty script, you can simply start it by:

+ +

#goto top level folder
+cd  SINGA_ROOT
+./bin/zk-service.sh start
+

(./bin/zk-service.sh stop stops the zookeeper).

Otherwise, if you launched a zookeeper by yourself but not used the default port, please edit the conf/singa.conf:

+ +

zookeeper_host: "localhost:YOUR_PORT"
+

We can extend the above two training frameworks to a cluster by updating the cluster configuration with:

@@ -430,8 +422,8 @@ cluster {

Every process would then create only one worker thread. Consequently, the workers would be created in different processes (i.e., nodes). The hostfile must be provided under SINGA_ROOT/conf/ specifying the nodes in the cluster, e.g.,

logbase-a01
-logbase-a02
+192.168.0.1
+192.168.0.2

And the zookeeper location must be configured correctly, e.g.,

@@ -439,14 +431,29 @@ logbase-a02

#conf/singa.conf
 zookeeper_host: "logbase-a01"

The running command is the same as for single node training:

The running command is :

./bin/singa-run.sh -conf examples/cifar10/job.conf
-

You can list the current running jobs by,

./bin/singa-console.sh list
+
+JOB ID    |NUM PROCS
+----------|-----------
+24        |2
+

Jobs can be killed by,

./bin/singa-console.sh kill JOB_ID
+

Logs and job information are available in /tmp/singa-log folder, which can be changed to other folders by setting log-dir in conf/singa.conf.

Running with Mesos

working…

Training with GPUs

Please refer to the [GPU page][gpu.html] for details on training using GPUs.

Where to go next

The programming guide pages will describe how to submit a training job in SINGA.

Apache SINGA – RBM Example

Neuron Layers

ActivationLayer

ConvolutionLayer

CuDNN layers

Loss Layers

Functions

Implementing a new Layer subclass

SINGA setup

Starting Zookeeper

Running in standalone mode

Training on a single node

Training without parallelism

Asynchronous parallel training

Synchronous parallel training

Training in a cluster

Starting Zookeeper

Running with Mesos

Training with GPUs

Where to go next