singa-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From zhon...@apache.org
Subject [3/3] incubator-singa git commit: SINGA-193 Add Python layers
Date Fri, 08 Jul 2016 04:53:53 GMT
SINGA-193 Add Python layers

Add python classes for all layers implemented in src/model/layer/.

Update factory for layer registration, new layers can be registered using the
help function in layer.h; Add gtest for CreateLayer.

Update cmake file to include the .cc file for each layer as python_cxx
which are necessary to get all registered layers by the swig code.

Swig can automatically convert python list into vector for function args, and
convert vector to python tuple.

Update CreateLayer to return shared_ptr<Layer>; Add a function to list all registered layers.

Add py version of cifar10 alexnet model in alexnet.py; update download_data.py for the py version data.

Add the FeedForwardNet python class
Update py layer constructor to make name as a required field of each layer's constructor.

Create symbolic link (in build/python) to python source files, not copy

Replace Tensor* to Tensor (for cpp functions) to make it easier for creating python functions.
Otherwise, py Tensor needs to carry both cpp Tensor* and cpp Tensor,
which is ambiguous and hard to maintain.

Add py loss, metric

Fix bugs from default arg for W_specs and b_specs; cannot set the default value for args of type dict.
Otherwise the same args would be used by later functions.

running foodlg training using vgg model

Add net::save() and net::load() for params

Add predict.py

Add random shuffle

clean code and pass tests


Project: http://git-wip-us.apache.org/repos/asf/incubator-singa/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-singa/commit/790b7b4c
Tree: http://git-wip-us.apache.org/repos/asf/incubator-singa/tree/790b7b4c
Diff: http://git-wip-us.apache.org/repos/asf/incubator-singa/diff/790b7b4c

Branch: refs/heads/dev
Commit: 790b7b4cdec305356ce5e18ccf153832d7d25c13
Parents: 7333517
Author: Wei Wang <wangwei@comp.nus.edu.sg>
Authored: Wed Jun 15 19:13:51 2016 +0800
Committer: Wei Wang <wangwei@comp.nus.edu.sg>
Committed: Fri Jul 8 00:16:34 2016 +0800

----------------------------------------------------------------------
 cmake/Utils.cmake                      |  35 ++
 examples/cifar10/alexnet.py            |  47 +++
 examples/cifar10/download_data.py      |  54 ++-
 examples/cifar10/predict.py            |  82 ++++
 examples/cifar10/train.py              | 142 +++++++
 include/singa/core/device.h            |   4 +-
 include/singa/core/tensor.h            |   4 +
 include/singa/model/feed_forward_net.h |  11 +-
 include/singa/model/initializer.h      |  24 +-
 include/singa/model/layer.h            |  28 +-
 include/singa/model/loss.h             |  21 +-
 include/singa/model/metric.h           |   9 +-
 include/singa/model/optimizer.h        |  24 +-
 include/singa/utils/factory.h          |  63 ++-
 src/CMakeLists.txt                     |  23 +-
 src/core/device/device.cc              |   2 -
 src/io/binfile_writer.cc               |   4 +-
 src/model/feed_forward_net.cc          |  14 +-
 src/model/layer/activation.cc          |   2 +
 src/model/layer/activation.h           |   1 +
 src/model/layer/batchnorm.cc           |   5 +-
 src/model/layer/batchnorm.h            |   9 +-
 src/model/layer/convolution.cc         |  10 +-
 src/model/layer/convolution.h          |   5 +
 src/model/layer/cudnn_activation.cc    |   1 +
 src/model/layer/cudnn_batchnorm.cc     |   1 +
 src/model/layer/cudnn_convolution.cc   |   4 +-
 src/model/layer/cudnn_dropout.cc       |   1 +
 src/model/layer/cudnn_lrn.cc           |   1 +
 src/model/layer/cudnn_pooling.cc       |   1 +
 src/model/layer/cudnn_softmax.cc       |   2 +
 src/model/layer/dense.cc               |   3 +-
 src/model/layer/dense.h                |   4 +-
 src/model/layer/dropout.cc             |   1 +
 src/model/layer/flatten.cc             |   1 +
 src/model/layer/lrn.cc                 |   1 +
 src/model/layer/pooling.cc             |   9 +-
 src/model/layer/prelu.cc               |   3 +-
 src/model/layer/softmax.cc             |   1 +
 src/model/metric/accuracy.cc           |   2 +-
 src/model/optimizer/adagrad.cc         |   6 +-
 src/model/optimizer/nesterov.cc        |   6 +-
 src/model/optimizer/optimizer.cc       |  20 +-
 src/model/optimizer/rmsprop.cc         |   6 +-
 src/model/optimizer/sgd.cc             |   8 +-
 src/python/__init__.py                 |   3 -
 src/python/device.py                   |  74 ----
 src/python/initializer.py              |  37 --
 src/python/layer.py                    |  79 ----
 src/python/model.py                    |  21 -
 src/python/optimizer.py                | 330 ---------------
 src/python/setup.py.in                 |  52 ++-
 src/python/singa/__init__.py           |   3 +
 src/python/singa/device.py             |  75 ++++
 src/python/singa/initializer.py        |  43 ++
 src/python/singa/layer.py              | 600 ++++++++++++++++++++++++++++
 src/python/singa/loss.py               |  46 +++
 src/python/singa/metric.py             |  42 ++
 src/python/singa/model.py              |  21 +
 src/python/singa/net.py                | 120 ++++++
 src/python/singa/optimizer.py          | 343 ++++++++++++++++
 src/python/singa/tensor.py             | 521 ++++++++++++++++++++++++
 src/python/singa/utils.py              |  47 +++
 src/python/swig/core_tensor.i          |  33 +-
 src/python/swig/model_layer.i          |  52 +--
 src/python/swig/model_loss.i           |  62 +++
 src/python/swig/model_metric.i         |  43 ++
 src/python/swig/model_optimizer.i      |   6 +-
 src/python/swig/singa.i                |   2 +
 src/python/tensor.py                   | 496 -----------------------
 test/python/test_layer.py              | 194 +++++++++
 test/python/test_optimizer.py          |   2 +-
 test/python/test_tensor.py             | 153 +++----
 test/singa/test_accuracy.cc            |   2 +-
 test/singa/test_adagrad.cc             |   8 +-
 test/singa/test_initializer.cc         |  14 +-
 test/singa/test_layer.cc               |  27 +-
 test/singa/test_nesterov.cc            |   8 +-
 test/singa/test_rmsprop.cc             |   8 +-
 test/singa/test_sgd.cc                 |  16 +-
 80 files changed, 2895 insertions(+), 1393 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/790b7b4c/cmake/Utils.cmake
----------------------------------------------------------------------
diff --git a/cmake/Utils.cmake b/cmake/Utils.cmake
index c55cee3..73feabc 100644
--- a/cmake/Utils.cmake
+++ b/cmake/Utils.cmake
@@ -15,3 +15,38 @@ macro(swig_generate_cxx pylist_variable)
     set(${pylist_variable} "${CMAKE_SOURCE_DIR}/src/python/swig/singa_wrap.cxx")
 endmacro()
 
+function (create_symlinks)
+    # Do nothing if building in-source
+    if (${CMAKE_CURRENT_BINARY_DIR} STREQUAL ${CMAKE_CURRENT_SOURCE_DIR})
+        return()
+    endif()
+
+    foreach (path_file ${ARGN})
+        get_filename_component(folder ${path_file} PATH)
+
+        # Create REAL folder
+        file(MAKE_DIRECTORY "${CMAKE_BINARY_DIR}/${folder}")
+
+        # Delete symlink if it exists
+        file(REMOVE "${CMAKE_BINARY_DIR}/${path_file}")
+
+        # Get OS dependent path to use in `execute_process`
+        file(TO_NATIVE_PATH "${CMAKE_BINARY_DIR}/${path_file}" link)
+        file(TO_NATIVE_PATH "${CMAKE_CURRENT_SOURCE_DIR}/${path_file}" target)
+
+        if (UNIX)
+            set(command ln -s ${target} ${link})
+        else()
+            set(command cmd.exe /c mklink ${link} ${target})
+        endif()
+
+        execute_process(COMMAND ${command} 
+                        RESULT_VARIABLE result
+                        ERROR_VARIABLE output)
+
+        if (NOT ${result} EQUAL 0)
+            message(FATAL_ERROR "Could not create symbolic link for: ${target} --> ${output}")
+        endif()
+
+    endforeach(path_file)
+endfunction(create_symlinks)
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/790b7b4c/examples/cifar10/alexnet.py
----------------------------------------------------------------------
diff --git a/examples/cifar10/alexnet.py b/examples/cifar10/alexnet.py
new file mode 100644
index 0000000..4b3daec
--- /dev/null
+++ b/examples/cifar10/alexnet.py
@@ -0,0 +1,47 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# =============================================================================
+import sys
+import os
+
+sys.path.append(os.path.join(os.path.dirname(__file__), '../../build/python'))
+from singa import layer
+from singa import metric
+from singa import loss
+from singa import net as ffnet
+from singa.proto import core_pb2
+
+
+def create_net():
+    net = ffnet.FeedForwardNet(loss.SoftmaxCrossEntropy(), metric.Accuracy())
+    W0_specs = {'init': 'gaussian', 'mean': 0, 'std': 0.0001}
+    W1_specs = {'init': 'gaussian', 'mean': 0, 'std': 0.01}
+    W2_specs = {'init': 'gaussian', 'mean': 0, 'std': 0.01, 'decay_mult': 250}
+    b_specs = {'init': 'constant', 'value': 0, 'lt_mult': 2}
+    net.add(layer.Conv2D('conv1', 32, 5, 1, W_specs=W0_specs.copy(), b_specs=b_specs.copy(), pad=2, input_sample_shape=(3,32,32,)))
+    net.add(layer.MaxPooling2D('pool1', 3, 2, pad=1))
+    net.add(layer.Activation('relu1'))
+    net.add(layer.LRN(name='lrn1'))
+    net.add(layer.Conv2D('conv2', 32, 5, 1, W_specs=W1_specs.copy(), b_specs=b_specs.copy(), pad=2))
+    net.add(layer.Activation('relu2'))
+    net.add(layer.MaxPooling2D('pool2', 3, 2,  pad=1))
+    net.add(layer.LRN('lrn2'))
+    net.add(layer.Conv2D('conv3', 64, 5, 1, W_specs=W1_specs.copy(), b_specs=b_specs.copy(), pad=2))
+    net.add(layer.Activation('relu3'))
+    net.add(layer.MaxPooling2D('pool3', 3, 2, pad=1))
+    net.add(layer.Flatten('flat'))
+    net.add(layer.Dense('dense', 10, W_specs=W2_specs.copy(), b_specs=b_specs.copy()))
+    return net

http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/790b7b4c/examples/cifar10/download_data.py
----------------------------------------------------------------------
diff --git a/examples/cifar10/download_data.py b/examples/cifar10/download_data.py
old mode 100644
new mode 100755
index ffb9724..ce0ee4f
--- a/examples/cifar10/download_data.py
+++ b/examples/cifar10/download_data.py
@@ -3,28 +3,50 @@ import urllib
 import tarfile
 import os
 import sys
+import argparse
 
 
 def extract_tarfile(filepath):
-    with tarfile.open(filepath, 'r') as f:
-        f.extractall('.')
+    if os.path.exists(filepath):
+        print 'The tar file does exist. Extracting it now..'
+        with tarfile.open(filepath, 'r') as f:
+            f.extractall('.')
+        print 'Finished!'
+        sys.exit(0)
 
 
-dirpath = 'cifar-10-batches-bin'
-gzfile = 'cifar-10-binary' + '.tar.gz'
-if os.path.exists(dirpath):
-    print 'Directory %s does exist. To redownload the files, '\
-        'remove the existing directory and %s.tar.gz' % (dirpath, dirpath)
-    sys.exit(0)
+def check_dir_exist(dirpath):
+    if os.path.exists(dirpath):
+        print 'Directory %s does exist. To redownload the files, '\
+            'remove the existing directory and %s.tar.gz' % (dirpath, dirpath)
+        return True
+    else:
+        return False
 
-if os.path.exists(gzfile):
-    print 'The tar file does exist. Extracting it now..'
+
+def do_download(dirpath, gzfile, url):
+    if check_dir_exist(dirpath):
+        sys.exit(0)
+    print 'Downloading CIFAR10 from %s' % (url)
+    urllib.urlretrieve(url, gzfile)
     extract_tarfile(gzfile)
     print 'Finished!'
-    sys.exit(0)
 
-url = 'http://www.cs.toronto.edu/~kriz/cifar-10-binary.tar.gz'
-print 'Downloading CIFAR10 from %s' % (url)
-urllib.urlretrieve(url, gzfile)
-extract_tarfile(gzfile)
-print 'Finished!'
+
+if __name__ == '__main__':
+    parser = argparse.ArgumentParser(description='Download Cifar10 datasets')
+    parser.add_argument(
+        'file',
+        type=str,
+        choices=['py', 'bin'])
+    args = parser.parse_args()
+    if args.file == 'bin':
+        dirpath = 'cifar-10-batches-bin'
+        gzfile = 'cifar-10-binary' + '.tar.gz'
+        url = 'http://www.cs.toronto.edu/~kriz/cifar-10-binary.tar.gz'
+        do_download(dirpath, gzfile, url)
+    else:
+        dirpath = 'cifar-10-batches-py'
+        gzfile = 'cifar-10-python' + '.tar.gz'
+        url = 'http://www.cs.toronto.edu/~kriz/cifar-10-python.tar.gz'
+        do_download(dirpath, gzfile, url)

http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/790b7b4c/examples/cifar10/predict.py
----------------------------------------------------------------------
diff --git a/examples/cifar10/predict.py b/examples/cifar10/predict.py
new file mode 100644
index 0000000..d083d0b
--- /dev/null
+++ b/examples/cifar10/predict.py
@@ -0,0 +1,82 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# =============================================================================
+
+import numpy as np
+import sys
+import os
+
+sys.path.append(os.path.join(os.path.dirname(__file__), '../../build/python'))
+
+from singa import device
+from singa import tensor
+import net as ffnet
+
+
+def predict(net, images, cuda, topk=5):
+    x = tensor.from_numpy(images.astype(np.float32))
+    x.to_device(cuda)
+    y = net.predict(x)
+    y.to_host()
+    y = tensor.to_numpy(y)
+    prob = np.average(y, 0)
+    labels = np.flipud(np.argsort(prob))  # sort prob in descending order
+    return labels[0:topk], prob[labels[0:topk]]
+
+
+def load_dataset(filepath):
+    print 'Loading data file %s' % filepath
+    with open(filepath, 'rb') as fd:
+        cifar10 = cPickle.load(fd)
+    image = cifar10['data'].astype(dtype=np.uint8)
+    image = image.reshape((-1, 3, 32, 32))
+    label = np.asarray(cifar10['labels'], dtype=np.uint8)
+    label = label.reshape(label.size, 1)
+    return image, label
+
+
+def load_train_data(dir_path, num_batches=5):
+    labels = []
+    batchsize = 10000
+    images = np.empty((num_batches * batchsize, 3, 32, 32), dtype=np.uint8)
+    for did in range(1, num_batches + 1):
+        fname_train_data = dir_path + "/data_batch_{}".format(did)
+        image, label = load_dataset(fname_train_data)
+        images[(did - 1) * batchsize:did * batchsize] = image
+        labels.extend(label)
+    images = np.array(images, dtype=np.float32)
+    labels = np.array(labels, dtype=np.int32)
+    return images, labels
+
+
+def load_test_data(dir_path):
+    images, labels = load_dataset(dir_path + "/test_batch")
+    return np.array(images,  dtype=np.float32), np.array(labels, dtype=np.int32)
+
+
+def compute_image_mean(train_dir):
+    images = np.load(train_dir)
+    return np.average(images, 0)
+
+if __name__ == '__main__':
+    model = ffnet.create_alexnet()
+    model.load('model.bin')
+    cuda = device.create_cuda_gpu()
+    model.to_device(cuda)
+
+    mean = compute_image_mean('cifar-10-batches-py')
+    test_images, _ = load_test_data('cifar-10-batches-py')
+    print predict(model, test_images - mean, cuda)

http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/790b7b4c/examples/cifar10/train.py
----------------------------------------------------------------------
diff --git a/examples/cifar10/train.py b/examples/cifar10/train.py
new file mode 100644
index 0000000..f4caca4
--- /dev/null
+++ b/examples/cifar10/train.py
@@ -0,0 +1,142 @@
+# Licensed to the Apache Software Foundation (ASF) under one
+# or more contributor license agreements.  See the NOTICE file
+# distributed with this work for additional information
+# regarding copyright ownership.  The ASF licenses this file
+# to you under the Apache License, Version 2.0 (the
+# "License"); you may not use this file except in compliance
+# with the License.  You may obtain a copy of the License at
+#
+#   http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+# =============================================================================
+""" CIFAR10 dataset is at https://www.cs.toronto.edu/~kriz/cifar.html.
+It includes 5 binary dataset, each contains 10000 images. 1 row (1 image)
+includes 1 label & 3072 pixels.  3072 pixels are 3 channels of a 32x32 image
+"""
+
+import cPickle
+import numpy as np
+import os
+import sys
+
+sys.path.append(os.path.join(os.path.dirname(__file__), '../../build/python'))
+from singa import initializer
+from singa import utils
+from singa import optimizer
+from singa import device
+from singa import tensor
+from singa.proto import core_pb2
+
+import alexnet
+
+
+def load_dataset(filepath):
+    print 'Loading data file %s' % filepath
+    with open(filepath, 'rb') as fd:
+        cifar10 = cPickle.load(fd)
+    image = cifar10['data'].astype(dtype=np.uint8)
+    image = image.reshape((-1, 3, 32, 32))
+    label = np.asarray(cifar10['labels'], dtype=np.uint8)
+    label = label.reshape(label.size, 1)
+    return image, label
+
+
+def load_train_data(dir_path, num_batches=5):
+    labels = []
+    batchsize = 10000
+    images = np.empty((num_batches * batchsize, 3, 32, 32), dtype=np.uint8)
+    for did in range(1, num_batches + 1):
+        fname_train_data = dir_path + "/data_batch_{}".format(did)
+        image, label = load_dataset(fname_train_data)
+        images[(did - 1) * batchsize:did * batchsize] = image
+        labels.extend(label)
+    images = np.array(images, dtype=np.float32)
+    labels = np.array(labels, dtype=np.int32)
+    return images, labels
+
+
+def load_test_data(dir_path):
+    images, labels = load_dataset(dir_path + "/test_batch")
+    return np.array(images,  dtype=np.float32), np.array(labels, dtype=np.int32)
+
+
+def get_lr(epoch):
+    if epoch < 120:
+        return 0.001
+    elif epoch < 130:
+        return 0.0001
+    elif epoch < 140:
+        return 0.00001
+
+
+def train(data_dir, net, num_epoch=140, batch_size=100):
+    print 'Start intialization............'
+    cuda = device.create_cuda_gpu()
+    net.to_device(cuda)
+    opt = optimizer.SGD(momentum=0.9, weight_decay=0.004)
+    for (p, specs) in zip(net.param_values(), net.param_specs()):
+        filler = specs.filler
+        if filler.type == 'gaussian':
+            initializer.gaussian(p, filler.mean, filler.std)
+        else:
+            p.set_value(0)
+        opt.register(p, specs)
+        print specs.name, filler.type, p.l1()
+    print 'Loading data ..................'
+    train_x, train_y = load_train_data(data_dir)
+    test_x, test_y = load_test_data(data_dir)
+    mean = np.average(train_x, axis=0)
+    train_x -= mean
+    test_x -= mean
+
+    tx = tensor.Tensor((batch_size, 3, 32, 32), cuda)
+    ty = tensor.Tensor((batch_size,), cuda, core_pb2.kInt)
+    num_train_batch = train_x.shape[0] / batch_size
+    num_test_batch = test_x.shape[0] / batch_size
+    idx = np.arange(train_x.shape[0], dtype=np.int32)
+    for epoch in range(num_epoch):
+        np.random.shuffle(idx)
+        loss, acc = 0.0, 0.0
+        print 'Epoch %d' % epoch
+        for b in range(num_train_batch):
+            x = train_x[idx[b * batch_size: (b + 1) * batch_size]]
+            y = train_y[idx[b * batch_size: (b + 1) * batch_size]]
+            tx.copy_from_numpy(x)
+            ty.copy_from_numpy(y)
+            grads, (l, a) = net.train(tx, ty)
+            loss += l
+            acc += a
+            for (s, p, g) in zip(net.param_specs(), net.param_values(), grads):
+                opt.apply_with_lr(epoch, get_lr(epoch), g, p, str(s.name))
+            # update progress bar
+            utils.update_progress(b * 1.0 / num_train_batch,
+                                  'training loss = %f, accuracy = %f' % (l, a))
+        info = 'training loss = %f, training accuracy = %f' \
+            % (loss / num_train_batch, acc / num_train_batch)
+        print info
+
+        loss, acc = 0.0, 0.0
+        for b in range(num_test_batch):
+            x = test_x[b * batch_size: (b + 1) * batch_size]
+            y = test_y[b * batch_size: (b + 1) * batch_size]
+            tx.copy_from_numpy(x)
+            ty.copy_from_numpy(y)
+            l, a = net.evaluate(tx, ty)
+            loss += l
+            acc += a
+
+        print 'test loss = %f, test accuracy = %f' \
+            % (loss / num_test_batch, acc / num_test_batch)
+    net.save('model.bin')  # save model params into checkpoint file
+
+if __name__ == '__main__':
+    data_dir = 'cifar-10-batches-py'
+    assert os.path.exists(data_dir), \
+        'Pls download the cifar10 dataset via "download_data.py py"'
+    net = alexnet.create_net()
+    train(data_dir, net)

http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/790b7b4c/include/singa/core/device.h
----------------------------------------------------------------------
diff --git a/include/singa/core/device.h b/include/singa/core/device.h
index 17613bb..e0e8676 100644
--- a/include/singa/core/device.h
+++ b/include/singa/core/device.h
@@ -40,6 +40,8 @@ namespace singa {
 /// languages, namely cpp, cuda and opencl.
 class Device {
   public:
+  // Device() = default;
+  virtual ~Device() {}
   /// Constructor with device ID, num of executors (e.g., cuda streams),
   /// max mem size to use (in MB)
   Device(int id, int num_executors);
@@ -89,8 +91,6 @@ class Device {
 
   int id() const { return id_; }
 
-  virtual ~Device();
-
  private:
   Device() {};
 

http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/790b7b4c/include/singa/core/tensor.h
----------------------------------------------------------------------
diff --git a/include/singa/core/tensor.h b/include/singa/core/tensor.h
index 3b4304a..3420a0c 100644
--- a/include/singa/core/tensor.h
+++ b/include/singa/core/tensor.h
@@ -81,6 +81,8 @@ class Tensor {
     return static_cast<const SType *>(block()->data());
   }
 
+  /// used for swig code to convert Tensor into numpy array.
+  /// It gets data into 'value'
   template <typename SType>
   void GetValue(SType *value, const size_t num) {
     CHECK(device_ == defaultDevice);
@@ -213,6 +215,8 @@ class Tensor {
 typedef Shape::iterator ShapeIter;
 inline size_t Product(const Shape &shape, int start = 0, size_t len = 0) {
   if (len == 0) len = shape.size();
+  if (len == 0)
+    return 0;
   CHECK_LE(len, shape.size());
   size_t v = 1;
   for (unsigned int i = start; i < len; i++) v *= shape[i];

http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/790b7b4c/include/singa/model/feed_forward_net.h
----------------------------------------------------------------------
diff --git a/include/singa/model/feed_forward_net.h b/include/singa/model/feed_forward_net.h
index 87d2179..36cbe00 100644
--- a/include/singa/model/feed_forward_net.h
+++ b/include/singa/model/feed_forward_net.h
@@ -57,8 +57,8 @@ class FeedForwardNet {
   /// necessary; But for training, both 'opt' and 'loss' are necessary.
   /// 'shuffle' indicates shuffling training samples within one epoch it is
   /// valid using Train();
-  void Compile(bool shuffle, Optimizer* opt, Loss<Tensor>* loss,
-               Metric<Tensor>* metric);
+  void Compile(bool shuffle, Optimizer* opt, Loss* loss,
+               Metric* metric);
 
   /// Conduct the training giving the training data 'x' and label 'y'.
   /// 'val_split' of training data is used for
@@ -121,14 +121,13 @@ class FeedForwardNet {
   const vector<Layer*> layers() const { return layers_; }
   const vector<string> GetParamNames() const;
   const vector<ParamSpec> GetParamSpecs() const;
-  const vector<Tensor*> GetParamValues() const;
-  const vector<Tensor*> GetParamGrads() const;
+  const vector<Tensor> GetParamValues() const;
 
  protected:
   vector<Layer*> layers_;
   Optimizer* opt_;
-  Loss<Tensor>* loss_;
-  Metric<Tensor>* metric_;
+  Loss* loss_;
+  Metric* metric_;
 
   bool shuffle_ = true;
   Device* device_ = nullptr;

http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/790b7b4c/include/singa/model/initializer.h
----------------------------------------------------------------------
diff --git a/include/singa/model/initializer.h b/include/singa/model/initializer.h
index 5692185..4592af5 100644
--- a/include/singa/model/initializer.h
+++ b/include/singa/model/initializer.h
@@ -37,7 +37,7 @@ class Initializer {
   /// Set meta fields from user configurations.
   virtual void Setup(const InitializerConf& conf) {}
 
-  virtual void Fill(Tensor* t) = 0;
+  virtual void Fill(Tensor& t) = 0;
 };
 
 namespace init {
@@ -46,7 +46,7 @@ public:
   Constant() = default;
   Constant(const float x) : v_(x) {}
   void Setup(const InitializerConf& conf) override { v_ = conf.value(); }
-  void Fill(Tensor* t) override { t->SetValue(v_); }
+  void Fill(Tensor& t) override { t.SetValue(v_); }
 
  private:
   float v_ = 0;
@@ -60,7 +60,7 @@ public:
     min_ = conf.min();
     max_ = conf.max();
   }
-  void Fill(Tensor* t) override { singa::Uniform(min_, max_, t); }
+  void Fill(Tensor& t) override { singa::Uniform(min_, max_, &t); }
 
  private:
   float min_ = 0, max_ = 1;
@@ -74,7 +74,7 @@ public:
     mean_ = conf.mean();
     std_ = conf.std();
   }
-  void Fill(Tensor* t) override { singa::Gaussian(mean_, std_, t); }
+  void Fill(Tensor& t) override { singa::Gaussian(mean_, std_, &t); }
 
  private:
   float mean_ = 0, std_ = 1;
@@ -84,11 +84,11 @@ public:
 /// feedforward neural networks
 class Xavier : public Initializer {
 public:
-  void Fill(Tensor* t) override {
-    CHECK_EQ(t->nDim(), 2u);
-    float scale = sqrt(6.0f / (t->shape(0) + t->shape(1)));
+  void Fill(Tensor& t) override {
+    CHECK_EQ(t.nDim(), 2u);
+    float scale = sqrt(6.0f / (t.shape(0) + t.shape(1)));
     LOG(INFO) << "xavier scale " << scale;
-    singa::Uniform(-scale, scale, t);
+    singa::Uniform(-scale, scale, &t);
   }
 };
 
@@ -96,10 +96,10 @@ public:
 /// Surpassing Human-Level Performance on ImageNet Classification
 class MSRA : public Initializer {
  public:
-  void Fill(Tensor* t) override {
-    CHECK_EQ(t->nDim(), 2u);
-    float std = sqrt(2.0f / t->shape(0));
-    singa::Gaussian(0.0f, std, t);
+  void Fill(Tensor& t) override {
+    CHECK_EQ(t.nDim(), 2u);
+    float std = sqrt(2.0f / t.shape(0));
+    singa::Gaussian(0.0f, std, &t);
   }
 };
 

http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/790b7b4c/include/singa/model/layer.h
----------------------------------------------------------------------
diff --git a/include/singa/model/layer.h b/include/singa/model/layer.h
index 938b161..c35f9b8 100644
--- a/include/singa/model/layer.h
+++ b/include/singa/model/layer.h
@@ -1,4 +1,4 @@
-/**
+/*
  * Licensed to the Apache Software Foundation (ASF) under one
  * or more contributor license agreements.  See the NOTICE file
  * distributed with this work for additional information
@@ -23,8 +23,10 @@
 #include <string>
 #include <stack>
 #include <utility>
+#include <memory>
 #include "singa/core/tensor.h"
 #include "singa/proto/model.pb.h"
+#include "singa/utils/factory.h"
 
 namespace singa {
 
@@ -196,12 +198,14 @@ class Layer {
   }
 
   /// Return pointers to parameter Tensor s.
-  const vector<Tensor*> param_values() { return param_values_; }
+  virtual const vector<Tensor> param_values() {
+    return vector<Tensor>{};
+  }
 
   /// Return a pointer to the 'i'-th parameter Tensor.
-  Tensor* param_value(size_t i) {
+  Tensor param_value(size_t i) {
     CHECK_LT(i, param_values_.size());
-    return param_values_[i];
+    return param_values().at(i);
   }
 
   /// Return names of all parmaeters.
@@ -227,5 +231,21 @@ class Layer {
   vector<ParamSpec> param_specs_;
 };
 
+#define RegisterLayerClass(SubLayer) \
+  static Registra<Layer, SubLayer> _##SubLayer##Layer(#SubLayer);
+
+inline std::shared_ptr<Layer> CreateLayer(const std::string type) {
+  std::shared_ptr<Layer> layer(Factory<Layer>::Create(type));
+  return layer;
+}
+
+inline const std::vector<std::string> GetRegisteredLayers() {
+  vector<std::string> ret;
+  for (const string type : Factory<Layer>::GetIDs()) {
+    auto layer = CreateLayer(type);
+    ret.push_back("Register type: " + type + " --> " + layer->layer_type());
+  }
+  return ret;
+}
 }  // namespace singa
 #endif  // SINGA_MODEL_LAYER_H_

http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/790b7b4c/include/singa/model/loss.h
----------------------------------------------------------------------
diff --git a/include/singa/model/loss.h b/include/singa/model/loss.h
index f400768..951c477 100644
--- a/include/singa/model/loss.h
+++ b/include/singa/model/loss.h
@@ -27,28 +27,29 @@ namespace singa {
 /// score (loss) for a pair of prediction (from the model) and the target (i.e.
 /// the ground truth). It also computes the gradients of the objective w.r.t.
 /// the prediction. It has similar APIs as Layer.
-template <typename T = Tensor>
+// template <typename T = Tensor>
 class Loss {
- public:
+public:
   Loss() = default;
-  void Setup(const string& conf) {
+  void Setup(const string &conf) {
     LossConf loss;
     loss.ParseFromString(conf);
     Setup(loss);
   }
-	virtual ~Loss(){};
+  virtual ~Loss() {};
   virtual void ToDevice(std::shared_ptr<Device> device) {}
   /// Set meta fields from user configurations.
-  virtual void Setup(const LossConf& conf) {}
+  virtual void Setup(const LossConf &conf) {}
 
   /// Compute the loss values for each sample/instance given the prediction
   /// and the target.
-  virtual Tensor Forward(int flag, const Tensor& prediction, const T& target) = 0;
+  virtual Tensor Forward(int flag, const Tensor &prediction,
+                         const Tensor &target) = 0;
 
   /// Average loss values for all samples in the mini-batch
   /// It calls Forward() internally. The calling pattern should be
   /// [Evaluate|Forward] Backward.
-  float Evaluate(int flag, const Tensor& prediction, const T& target) {
+  float Evaluate(int flag, const Tensor &prediction, const Tensor &target) {
     Tensor loss = Forward(flag, prediction, target);
     loss.ToHost();
     return Sum<float>(loss) / (1.0f * loss.Size());
@@ -58,11 +59,9 @@ class Loss {
   virtual Tensor Backward() = 0;
 };
 
-
-
 // ============= Mean Squared Error ===========================================
 /// MSE is for mean squared error or squared euclidean distance.
-class MSE : public Loss<Tensor> {
+class MSE : public Loss {
  public:
   /// Compute the loss values for each sample/instance given the prediction
   /// and the target, which is 0.5/||prediction-target||^2
@@ -82,7 +81,7 @@ class MSE : public Loss<Tensor> {
 
 // ===============Softamx Cross Entropy =======================================
 /// Softmax + cross entropy for multi-category classification
-class SoftmaxCrossEntropy : public Loss<Tensor> {
+class SoftmaxCrossEntropy : public Loss {
  public:
   /// Compute the loss values for each sample/instance given the prediction
   /// and the target, which is -log(p[idx_truth]), idx_truth is the truth

http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/790b7b4c/include/singa/model/metric.h
----------------------------------------------------------------------
diff --git a/include/singa/model/metric.h b/include/singa/model/metric.h
index b100435..ad7f717 100644
--- a/include/singa/model/metric.h
+++ b/include/singa/model/metric.h
@@ -28,11 +28,12 @@ namespace singa {
 /// The target type is a template argument.  For data samples with a single
 /// label, T could be 1-d tenor (or vector<int>); If each data sample has
 /// multiple labels, T could be vector<vector<int>>, one vector per sample.
-template <typename T = Tensor>
+// template <typename T = Tensor>
 class Metric {
  public:
   // TODO(wangwei) call Setup using a default MetricConf.
   Metric() = default;
+  virtual ~Metric() {}
   virtual void ToDevice(std::shared_ptr<Device> device) {}
   void Setup(const string& conf) {
     MetricConf metric;
@@ -44,10 +45,10 @@ class Metric {
   virtual void Setup(const MetricConf& conf) {}
 
   /// Compute the metric for each data sample
-  virtual Tensor Forward(const Tensor& prediction, const T& target) = 0;
+  virtual Tensor Forward(const Tensor& prediction, const Tensor& target) = 0;
 
   /// Comptue the metric value averaged over all samples (in a batch)
-  float Evaluate(const Tensor& prediction, const T& target) {
+  float Evaluate(const Tensor& prediction, const Tensor& target) {
     const Tensor metric = Forward(prediction, target);
     return Sum<float>(metric) / (1.0f * metric.Size());
   }
@@ -55,7 +56,7 @@ class Metric {
 /// Compute the accuray of the prediction, which is matched against the
 /// ground truth labels.
 /// TODO(wangwei) consider multi-label cases.
-class Accuracy : public Metric<Tensor> {
+class Accuracy : public Metric {
  public:
   /// Set meta fields from user configurations.
   void Setup(const MetricConf& conf) override { top_k_ = conf.top_k(); }

http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/790b7b4c/include/singa/model/optimizer.h
----------------------------------------------------------------------
diff --git a/include/singa/model/optimizer.h b/include/singa/model/optimizer.h
index 454a387..e6e6d1c 100644
--- a/include/singa/model/optimizer.h
+++ b/include/singa/model/optimizer.h
@@ -63,13 +63,13 @@ class Optimizer {
   /// by Apply(int, const string&, Tensor*, Tensor*).
   /// All sub-classes should override this function.
   virtual void Apply(int step, float lr, const string& name, const Tensor& grad,
-                     Tensor* value) = 0;
+                     Tensor& value) = 0;
 
   /// Apply the updating algorithm.
   /// It will apply regularization and constraint to the parameters if
   /// configured during Register(). If will also scale the learning rate if
   /// configured in ParamSpecs (see Register).
-  void Apply(int step, const string& name, Tensor* grad, Tensor* value);
+  void Apply(int step, const string& name, Tensor& grad, Tensor& value);
 
   /// The argument is a function that returns the learning rate given the
   /// current step (i.e., curren running iteration).
@@ -113,11 +113,11 @@ class Constraint {
   /// e.g., clip each gradient if it is too large w.r.t the threshold,
   /// \ref
   /// https://www.reddit.com/r/MachineLearning/comments/31b6x8/gradient_clipping_rnns/
-  void Apply(int step, Tensor* grad, Tensor* value);
+  void Apply(int step, Tensor& grad, Tensor& value);
   /// Apply the constraint for multiple parameter objects together.
   /// \ref https://github.com/Lasagne/Lasagne/blob/master/lasagne/updates.py
-  void Apply(int step, const vector<Tensor*>& grads,
-             const vector<Tensor*>& values);
+  void Apply(int step, const vector<Tensor>& grads,
+             const vector<Tensor>& values);
 
  private:
   /// currently only support "L2" norm constraint, i.e., the norm should be less
@@ -150,11 +150,11 @@ class Regularizer {
   /// e.g., clip each gradient if it is too large w.r.t the threshold,
   /// \ref
   /// https://www.reddit.com/r/MachineLearning/comments/31b6x8/gradient_clipping_rnns/
-  void Apply(int step, Tensor* grad, Tensor* value, float scale = 1.0f);
+  void Apply(int step, Tensor& grad, Tensor& value, float scale = 1.0f);
   /// Apply the regularizer for multiple parameter objects together.
   /// \ref https://github.com/Lasagne/Lasagne/blob/master/lasagne/updates.py
-  void Apply(int step, const vector<Tensor*>& grads,
-             const vector<Tensor*>& values);
+  void Apply(int step, const vector<Tensor>& grads,
+             const vector<Tensor>& values);
 
  private:
   /// currently only support "L2" regularizer. type_ is case insensitive.
@@ -174,7 +174,7 @@ class SGD : public Optimizer {
   void Setup(const OptimizerConf& conf);
   /// Apply the updating algorithm.
   void Apply(int step, float lr, const string& name, const Tensor& grad,
-             Tensor* value) override;
+             Tensor& value) override;
 
   /// The argument function returns the momentum value given the current running
   /// step (i.e., iterations/mini-batches).
@@ -193,7 +193,7 @@ class Nesterov : public Optimizer {
   void Setup(const OptimizerConf& conf);
   /// Apply the updating algorithm.
   void Apply(int step, float lr, const string& name, const Tensor& grad,
-             Tensor* value) override;
+             Tensor& value) override;
 
   /// The argument function returns the momentum value given the current running
   /// step (i.e., iterations/mini-batches).
@@ -212,7 +212,7 @@ class AdaGrad : public Optimizer {
   void Setup(const OptimizerConf& conf);
   /// Apply the updating algorithm.
   void Apply(int step, float lr, const string& name, const Tensor& grad,
-             Tensor* value) override;
+             Tensor& value) override;
 
  private:
   std::unordered_map<string, Tensor> history_gradient_;
@@ -224,7 +224,7 @@ class RMSProp : public Optimizer {
   void Setup(const OptimizerConf& conf);
   /// Apply the updating algorithm.
   void Apply(int step, float lr, const string& name, const Tensor& grad,
-             Tensor* value) override;
+             Tensor& value) override;
   virtual ~RMSProp() = default;
 
  private:

http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/790b7b4c/include/singa/utils/factory.h
----------------------------------------------------------------------
diff --git a/include/singa/utils/factory.h b/include/singa/utils/factory.h
index de018a6..b53caef 100644
--- a/include/singa/utils/factory.h
+++ b/include/singa/utils/factory.h
@@ -41,7 +41,7 @@
  * 2. call Create() func to call the creation function and return
  * a pointer to the base calss.
  */
-template<typename T>
+template<typename T, typename ID = std::string>
 class Factory {
  public:
   /**
@@ -51,50 +51,47 @@ class Factory {
    * @param id Identifier of the creating function/class
    * @param func a function that creates a layer instance
    */
-  inline void Register(const std::string& id,
-                       const std::function<T*(void)>& func) {
-    CHECK(str2func_.find(id) == str2func_.end())
-      << "The id has been registered by another function";
-    str2func_[id] = func;
-  }
-  /**
-   * Register functions to create user defined classes.
-   * This function is called by the REGISTER_FACTORY macro.
-   *
-   * @param id Identifier of the creating function/class
-   * @param func a function that creates a layer instance
-   */
-  inline void Register(int id,
-                       const std::function<T*(void)>& func) {
-    CHECK(id2func_.find(id) == id2func_.end())
-      << "The id has been registered by another function";
-    id2func_[id] = func;
+  static void Register(const ID& id,
+                       const std::function<T*(void)>& creator) {
+    Registry* reg = GetRegistry();
+    // CHECK(reg->find(id) == reg->end())
+    //  << "The id " << id << " has been registered";
+    (*reg)[id] = creator;
   }
+
   /**
    * create an instance by providing its id
    *
    * @param id
    */
-  inline T* Create(const std::string& id) {
-    CHECK(str2func_.find(id) != str2func_.end())
+  static T* Create(const ID& id) {
+    Registry* reg = GetRegistry();
+    CHECK(reg->find(id) != reg->end())
       << "The creation function for " << id << " has not been registered";
-    return str2func_[id]();
+    return (*reg)[id]();
   }
-  /**
-   * create an instance by providing its id
-   *
-   * @param id
-   */
-  inline T* Create(int id) {
-    CHECK(id2func_.find(id) != id2func_.end())
-      << "The creation function for " << id << " has not been registered";
-    return id2func_[id]();
+
+  static const std::vector<ID> GetIDs() {
+    std::vector<ID> keys;
+    for (const auto entry : *GetRegistry())
+      keys.push_back(entry.first);
+    return keys;
   }
 
  private:
   // Map that stores the registered creation functions
-  std::map<std::string, std::function<T*(void)>> str2func_;
-  std::map<int, std::function<T*(void)>> id2func_;
+  typedef std::map<ID, std::function<T*(void)>> Registry;
+  static Registry* GetRegistry() {
+    static Registry reg;
+    return &reg;
+  }
 };
 
+template<typename Base, typename Sub, typename ID = std::string>
+class Registra {
+ public:
+  Registra(const ID& id) {
+    Factory<Base, ID>::Register(id, [](void) { return new Sub(); });
+  }
+};
 #endif  // SINGA_UTILS_FACTORY_H_

http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/790b7b4c/src/CMakeLists.txt
----------------------------------------------------------------------
diff --git a/src/CMakeLists.txt b/src/CMakeLists.txt
index c479252..38ef6cc 100644
--- a/src/CMakeLists.txt
+++ b/src/CMakeLists.txt
@@ -79,16 +79,31 @@ LIST(APPEND SINGA_LINKER_LIBS singa_io)
 #ADD_LIBRARY(singa SHARED ${SINGA_LINKER_LIBS})
 
 IF(USE_PYTHON)
+
     FILE(GLOB python_files python/swig/singa.i)
+    # delete old .cxx file
+    FILE(REMOVE "${CMAKE_CURRENT_SOURCE_DIR}/python/swig/singa_wrap.cxx")
+
+    # generate cxx and wrap.py
     swig_generate_cxx(python_srcs ${python_files})
-    FILE(COPY python/ DESTINATION ${CMAKE_BINARY_DIR}/python/singa FILES_MATCHING PATTERN "swig" EXCLUDE PATTERN "*.py")
-    SET(python_cxxs "${CMAKE_SOURCE_DIR}/src/core/tensor/tensor.cc;${CMAKE_SOURCE_DIR}/src/core/device/device.cc")
+
+    #FILE(COPY python/ DESTINATION ${CMAKE_BINARY_DIR}/python/singa FILES_MATCHING PATTERN "swig" EXCLUDE PATTERN "*.py")
+    #Create symlinks for all python source files  Do not omit !!!RELATIVE!!!
+    file(GLOB_RECURSE python_source_files RELATIVE ${CMAKE_CURRENT_SOURCE_DIR} *.py)
+
+    create_symlinks(${python_source_files})
+
+    FILE(GLOB_RECURSE layer_source_files ${CMAKE_SOURCE_DIR}/src/model/layer/*.cc)
+    # message(STATUS "layer_source_files ${layer_source_files}")
+    FILE(GLOB_RECURSE metric_source_files ${CMAKE_SOURCE_DIR}/src/model/metric/*.cc)
+    FILE(GLOB_RECURSE loss_source_files ${CMAKE_SOURCE_DIR}/src/model/loss/*.cc)
+    SET(python_cxxs "${CMAKE_SOURCE_DIR}/src/core/tensor/tensor.cc;${CMAKE_SOURCE_DIR}/src/core/device/device.cc;${layer_source_files};${metric_source_files};${loss_source_files}")
     ADD_LIBRARY(_singa_wrap SHARED ${python_srcs} ${python_cxxs})
     SET(WRAPPER_LINKER_LIBS "${SINGA_LINKER_LIBS};protobuf")
     TARGET_LINK_LIBRARIES(_singa_wrap ${WRAPPER_LINKER_LIBS})
     TARGET_INCLUDE_DIRECTORIES(_singa_wrap PRIVATE ${PYTHON_INCLUDE_DIRS})
     #message(STATUS "PYTHON_INCLUDE_DIRS ${PYTHON_INCLUDE_DIRS}")
-    
+
     SET_TARGET_PROPERTIES(_singa_wrap
         PROPERTIES PREFIX ""
         LIBRARY_OUTPUT_DIRECTORY ${CMAKE_BINARY_DIR}/python/singa
@@ -98,7 +113,7 @@ IF(USE_PYTHON)
     SET(SETUP_PY_IN "python/setup.py.in")
     set(SETUP_PY    "${CMAKE_BINARY_DIR}/python/setup.py")
     configure_file(${SETUP_PY_IN} ${SETUP_PY})
-    
+
     #COPY protobuf files to python/singa/proto
     FILE(COPY ${CMAKE_BINARY_DIR}/python/singa/__init__.py DESTINATION ${CMAKE_BINARY_DIR}/python/singa/proto )
 

http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/790b7b4c/src/core/device/device.cc
----------------------------------------------------------------------
diff --git a/src/core/device/device.cc b/src/core/device/device.cc
index 31d2b2a..0220df0 100644
--- a/src/core/device/device.cc
+++ b/src/core/device/device.cc
@@ -25,8 +25,6 @@ Device::Device(int id, int num_executors)
   host_ = defaultDevice;
 }
 
-Device::~Device() {}
-
 void Device::Exec(function<void(Context*)>&& fn, const vector<Block*> read_blocks,
                     const vector<Block*> write_blocks, bool use_rand_generator) {
   // TODO(wangwei) execute operations scheduled by the scheduler.

http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/790b7b4c/src/io/binfile_writer.cc
----------------------------------------------------------------------
diff --git a/src/io/binfile_writer.cc b/src/io/binfile_writer.cc
index e207453..adc910e 100644
--- a/src/io/binfile_writer.cc
+++ b/src/io/binfile_writer.cc
@@ -24,7 +24,6 @@ namespace io {
 bool BinFileWriter::Open(const std::string& path, Mode mode) {
   path_ = path;
   mode_ = mode;
-  buf_ = new char[capacity_];
   return OpenFile();
 }
 
@@ -39,7 +38,7 @@ bool BinFileWriter::Open(const std::string& path, Mode mode, int capacity) {
 void BinFileWriter::Close() {
   Flush();
   if (buf_ != nullptr) {
-    delete buf_;
+    delete [] buf_;
     buf_ = nullptr;
   }
   if (fdat_.is_open()) fdat_.close();
@@ -92,6 +91,7 @@ void BinFileWriter::Flush() {
 }
 
 bool BinFileWriter::OpenFile() {
+  CHECK(buf_ == nullptr);
   buf_ = new char[capacity_];
   switch (mode_) {
     case kCreate:

http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/790b7b4c/src/model/feed_forward_net.cc
----------------------------------------------------------------------
diff --git a/src/model/feed_forward_net.cc b/src/model/feed_forward_net.cc
index ebbe00c..09a06d7 100644
--- a/src/model/feed_forward_net.cc
+++ b/src/model/feed_forward_net.cc
@@ -59,8 +59,8 @@ const vector<string> FeedForwardNet::GetParamNames() const {
     for (const auto name : layer->param_names()) names.push_back(name);
   return names;
 }
-const vector<Tensor*> FeedForwardNet::GetParamValues() const {
-  vector<Tensor*> values;
+const vector<Tensor> FeedForwardNet::GetParamValues() const {
+  vector<Tensor> values;
   for (auto layer : layers_)
     for (const auto value : layer->param_values()) values.push_back(value);
   return values;
@@ -73,8 +73,8 @@ const vector<ParamSpec> FeedForwardNet::GetParamSpecs() const {
   return specs;
 }
 
-void FeedForwardNet::Compile(bool shuffle, Optimizer* opt, Loss<Tensor>* loss,
-                             Metric<Tensor>* metric) {
+void FeedForwardNet::Compile(bool shuffle, Optimizer* opt, Loss* loss,
+                             Metric* metric) {
   shuffle_ = shuffle;
   bool train = (opt != nullptr) && (loss != nullptr);
   bool test = metric != nullptr;
@@ -83,13 +83,13 @@ void FeedForwardNet::Compile(bool shuffle, Optimizer* opt, Loss<Tensor>* loss,
   loss_ = loss;
   metric_ = metric;
   const auto specs = GetParamSpecs();
-  const auto params = GetParamValues();
+  auto params = GetParamValues();
   CHECK_EQ(specs.size(), params.size());
   for (size_t k = 0; k < specs.size(); k++) {
     opt_->Register(specs[k].name(), specs[k]);
     auto init = CreateInitializer(specs[k].filler());
     init->Fill(params[k]);
-    LOG(INFO) << specs[k].name() << " : " << params[k]->L1();
+    LOG(INFO) << specs[k].name() << " : " << params[k].L1();
   }
 }
 
@@ -195,7 +195,7 @@ const std::pair<float, float> FeedForwardNet::TrainOnBatch(int epoch,
   auto names = GetParamNames();
   auto values = GetParamValues();
   for (size_t k = 0; k < grads.size(); k++) {
-    opt_->Apply(epoch, names[k], &grads[k], values.at(k));
+    opt_->Apply(epoch, names[k], grads[k], values.at(k));
   }
   return std::make_pair(loss, metric);
 }

http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/790b7b4c/src/model/layer/activation.cc
----------------------------------------------------------------------
diff --git a/src/model/layer/activation.cc b/src/model/layer/activation.cc
index b00834f..2497c31 100644
--- a/src/model/layer/activation.cc
+++ b/src/model/layer/activation.cc
@@ -20,6 +20,8 @@
 #include "./activation.h"
 namespace singa {
 
+RegisterLayerClass(Activation);
+
 void Activation::Setup(const Shape& in_sample, const LayerConf& conf) {
   Layer::Setup(in_sample, conf);
   mode_ = conf.type();

http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/790b7b4c/src/model/layer/activation.h
----------------------------------------------------------------------
diff --git a/src/model/layer/activation.h b/src/model/layer/activation.h
index db3a8f5..e3fb657 100644
--- a/src/model/layer/activation.h
+++ b/src/model/layer/activation.h
@@ -52,5 +52,6 @@ class Activation : public Layer {
   Shape out_sample_shape_;
   float neg_slope_;
 };
+
 }  // namespace singa
 #endif  // SINGA_MODEL_LAYER_ACTIVATION_H_

http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/790b7b4c/src/model/layer/batchnorm.cc
----------------------------------------------------------------------
diff --git a/src/model/layer/batchnorm.cc b/src/model/layer/batchnorm.cc
index ab666aa..e583f8e 100644
--- a/src/model/layer/batchnorm.cc
+++ b/src/model/layer/batchnorm.cc
@@ -21,6 +21,7 @@
 #include "batchnorm.h"
 
 namespace singa {
+RegisterLayerClass(BatchNorm);
 void BatchNorm::Setup(const Shape& in_sample, const LayerConf& conf) {
   Layer::Setup(in_sample, conf);
   out_sample_shape_ = in_sample;
@@ -39,10 +40,6 @@ void BatchNorm::Setup(const Shape& in_sample, const LayerConf& conf) {
   // Push back params into param_values_
   // Assume the order of param is: bnScale, bnBias, runningMean, runningVariance
   for (const auto &spec : conf.param()) param_specs_.push_back(spec);
-  param_values_.push_back(&bnScale_);
-  param_values_.push_back(&bnBias_);
-  param_values_.push_back(&runningMean_);
-  param_values_.push_back(&runningVariance_);
 }
 
 void BatchNorm::ToDevice(std::shared_ptr<Device> device) {

http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/790b7b4c/src/model/layer/batchnorm.h
----------------------------------------------------------------------
diff --git a/src/model/layer/batchnorm.h b/src/model/layer/batchnorm.h
index c2b43e9..6ff818b 100644
--- a/src/model/layer/batchnorm.h
+++ b/src/model/layer/batchnorm.h
@@ -29,9 +29,7 @@ namespace singa {
 class BatchNorm : public Layer {
  public:
   /// \copydoc Layer::layer_type()
-  const std::string layer_type() const override {
-    return "Batch Normalization";
-  }
+  const std::string layer_type() const override { return "BatchNorm"; }
 
   /// \copydoc Layer::Setup(const LayerConf&);
   void Setup(const Shape& in_sample, const LayerConf& conf) override;
@@ -46,7 +44,10 @@ class BatchNorm : public Layer {
   /// \copydoc Layer::Backward(int, const Tensor&, const Tensor&);
   const std::pair<Tensor, vector<Tensor>> Backward(
       int flag, const Tensor& grad) override;
-
+  const std::vector<Tensor> param_values() override {
+    return std::vector<Tensor> { bnScale_, bnBias_, runningMean_,
+                                 runningVariance_ };
+  }
   const float factor() const { return factor_; }
   const Tensor& bnScale() const { return bnScale_; }
   const Tensor& bnBias() const { return bnBias_; }

http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/790b7b4c/src/model/layer/convolution.cc
----------------------------------------------------------------------
diff --git a/src/model/layer/convolution.cc b/src/model/layer/convolution.cc
index e4991a4..a43014d 100644
--- a/src/model/layer/convolution.cc
+++ b/src/model/layer/convolution.cc
@@ -23,6 +23,7 @@
 namespace singa {
 using std::vector;
 
+RegisterLayerClass(Convolution);
 void Convolution::Setup(const Shape& in_sample, const LayerConf &conf) {
   Layer::Setup(in_sample, conf);
   ConvolutionConf conv_conf = conf.convolution_conf();
@@ -67,7 +68,7 @@ void Convolution::Setup(const Shape& in_sample, const LayerConf &conf) {
     stride_h_ = conv_conf.stride_h();
   }
   CHECK_GT(stride_w_, 0u);
-  CHECK_GT(stride_h_, 0u);
+  CHECK_GE(stride_h_, 0u);  // 0 for 1D conv
 
   num_filters_ = conv_conf.num_output();
   bias_term_ = conv_conf.bias_term();
@@ -78,7 +79,9 @@ void Convolution::Setup(const Shape& in_sample, const LayerConf &conf) {
   height_ = in_sample.at(1);
   width_ = in_sample.at(2);
 
-  conv_height_ = (height_ + 2 * pad_h_ - kernel_h_) / stride_h_ + 1;
+  conv_height_ = 1;
+  if (stride_h_ > 0)
+    conv_height_ = (height_ + 2 * pad_h_ - kernel_h_) / stride_h_ + 1;
   conv_width_ = (width_ + 2 * pad_w_ - kernel_w_) / stride_w_ + 1;
   out_sample_shape_ = vector<size_t>{num_filters_, conv_height_, conv_width_};
 
@@ -88,11 +91,8 @@ void Convolution::Setup(const Shape& in_sample, const LayerConf &conf) {
   // Setup shape of weight_ and bias_
   weight_.Reshape(Shape{num_filters_, col_height_});
   bias_.Reshape(Shape{num_filters_});
-  // Push back params into param_values_
   // Assume the order of param is: weight, bias
   for (const auto &spec : conf.param()) param_specs_.push_back(spec);
-  param_values_.push_back(&weight_);
-  param_values_.push_back(&bias_);
 }
 
 /// \copydoc Layer::Forward(int flag, const Tensor&)

http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/790b7b4c/src/model/layer/convolution.h
----------------------------------------------------------------------
diff --git a/src/model/layer/convolution.h b/src/model/layer/convolution.h
index 0e0b160..b3d0c42 100644
--- a/src/model/layer/convolution.h
+++ b/src/model/layer/convolution.h
@@ -45,6 +45,11 @@ class Convolution : public Layer {
                                                    const Tensor &grad) override;
 
   void ToDevice(std::shared_ptr<Device> device) override;
+
+  const std::vector<Tensor> param_values() override {
+    return std::vector<Tensor>{weight_, bias_};
+  }
+
   size_t kernel_w() const { return kernel_w_; }
   size_t kernel_h() const { return kernel_h_; }
   size_t pad_w() const { return pad_w_; }

http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/790b7b4c/src/model/layer/cudnn_activation.cc
----------------------------------------------------------------------
diff --git a/src/model/layer/cudnn_activation.cc b/src/model/layer/cudnn_activation.cc
index df728ce..c86539d 100644
--- a/src/model/layer/cudnn_activation.cc
+++ b/src/model/layer/cudnn_activation.cc
@@ -25,6 +25,7 @@
 #include "singa/utils/logging.h"
 
 namespace singa {
+RegisterLayerClass(CudnnActivation);
 CudnnActivation::~CudnnActivation() {
   if (acti_desc_ != nullptr)
     CUDNN_CHECK(cudnnDestroyActivationDescriptor(acti_desc_));

http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/790b7b4c/src/model/layer/cudnn_batchnorm.cc
----------------------------------------------------------------------
diff --git a/src/model/layer/cudnn_batchnorm.cc b/src/model/layer/cudnn_batchnorm.cc
index a1e9e50..9e1e892 100644
--- a/src/model/layer/cudnn_batchnorm.cc
+++ b/src/model/layer/cudnn_batchnorm.cc
@@ -23,6 +23,7 @@
 
 namespace singa {
 
+RegisterLayerClass(CudnnBatchNorm);
 CudnnBatchNorm::~CudnnBatchNorm() {
   if (has_init_cudnn_) {
     CUDNN_CHECK(cudnnDestroyTensorDescriptor(shape_desc_));

http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/790b7b4c/src/model/layer/cudnn_convolution.cc
----------------------------------------------------------------------
diff --git a/src/model/layer/cudnn_convolution.cc b/src/model/layer/cudnn_convolution.cc
index 82cf4e5..e5efec0 100644
--- a/src/model/layer/cudnn_convolution.cc
+++ b/src/model/layer/cudnn_convolution.cc
@@ -23,6 +23,7 @@
 #include "singa/utils/logging.h"
 
 namespace singa {
+RegisterLayerClass(CudnnConvolution);
 CudnnConvolution::~CudnnConvolution() {
   if (bias_desc_ != nullptr)
     CUDNN_CHECK(cudnnDestroyTensorDescriptor(bias_desc_));
@@ -47,8 +48,7 @@ void CudnnConvolution::Setup(const Shape& in_sample, const LayerConf &conf) {
 }
 
 void CudnnConvolution::ToDevice(std::shared_ptr<Device> device) {
-  weight_.ToDevice(device);
-  bias_.ToDevice(device);
+  Convolution::ToDevice(device);
   workspace_.ToDevice(device);
 }
 

http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/790b7b4c/src/model/layer/cudnn_dropout.cc
----------------------------------------------------------------------
diff --git a/src/model/layer/cudnn_dropout.cc b/src/model/layer/cudnn_dropout.cc
index ab83226..e6950ca 100644
--- a/src/model/layer/cudnn_dropout.cc
+++ b/src/model/layer/cudnn_dropout.cc
@@ -27,6 +27,7 @@
 #include "singa/utils/logging.h"
 
 namespace singa {
+RegisterLayerClass(CudnnDropout);
 CudnnDropout::~CudnnDropout() {
   if (drop_desc_ != nullptr)
     CUDNN_CHECK(cudnnDestroyDropoutDescriptor(drop_desc_));

http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/790b7b4c/src/model/layer/cudnn_lrn.cc
----------------------------------------------------------------------
diff --git a/src/model/layer/cudnn_lrn.cc b/src/model/layer/cudnn_lrn.cc
index 4dbf426..540beb1 100644
--- a/src/model/layer/cudnn_lrn.cc
+++ b/src/model/layer/cudnn_lrn.cc
@@ -23,6 +23,7 @@
 #include "cudnn_utils.h"
 
 namespace singa {
+RegisterLayerClass(CudnnLRN);
 CudnnLRN::~CudnnLRN() {
   if (has_init_cudnn_) {
     CUDNN_CHECK(cudnnDestroyLRNDescriptor(lrn_desc_));

http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/790b7b4c/src/model/layer/cudnn_pooling.cc
----------------------------------------------------------------------
diff --git a/src/model/layer/cudnn_pooling.cc b/src/model/layer/cudnn_pooling.cc
index e49a1ec..984427c 100644
--- a/src/model/layer/cudnn_pooling.cc
+++ b/src/model/layer/cudnn_pooling.cc
@@ -25,6 +25,7 @@
 #include "singa/utils/logging.h"
 
 namespace singa {
+RegisterLayerClass(CudnnPooling);
 CudnnPooling::~CudnnPooling() {
   if (pool_desc_ != nullptr)
     CUDNN_CHECK(cudnnDestroyPoolingDescriptor(pool_desc_));

http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/790b7b4c/src/model/layer/cudnn_softmax.cc
----------------------------------------------------------------------
diff --git a/src/model/layer/cudnn_softmax.cc b/src/model/layer/cudnn_softmax.cc
index 1d9e0b8..6dce68f 100644
--- a/src/model/layer/cudnn_softmax.cc
+++ b/src/model/layer/cudnn_softmax.cc
@@ -22,6 +22,8 @@
 #include "./cudnn_utils.h"
 #include "singa/utils/logging.h"
 namespace singa {
+
+RegisterLayerClass(CudnnSoftmax);
 CudnnSoftmax::~CudnnSoftmax() {
   if (desc_ != nullptr) CUDNN_CHECK(cudnnDestroyTensorDescriptor(desc_));
 }

http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/790b7b4c/src/model/layer/dense.cc
----------------------------------------------------------------------
diff --git a/src/model/layer/dense.cc b/src/model/layer/dense.cc
index 338409c..557d8bd 100644
--- a/src/model/layer/dense.cc
+++ b/src/model/layer/dense.cc
@@ -23,6 +23,7 @@
 namespace singa {
 using std::vector;
 
+RegisterLayerClass(Dense);
 Dense::~Dense() {
   // delete weight_;
   // delete bias_;
@@ -39,8 +40,6 @@ void Dense::Setup(const Shape& in_sample, const LayerConf &conf) {
   else
     weight_.Reshape(Shape{hdim_, vdim_});
   bias_.Reshape(Shape{hdim_});
-  param_values_.push_back(&weight_);
-  param_values_.push_back(&bias_);
   for (auto specs: conf.param())
     param_specs_.push_back(specs);
 }

http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/790b7b4c/src/model/layer/dense.h
----------------------------------------------------------------------
diff --git a/src/model/layer/dense.h b/src/model/layer/dense.h
index 8b30d80..bb5db66 100644
--- a/src/model/layer/dense.h
+++ b/src/model/layer/dense.h
@@ -45,7 +45,9 @@ class Dense : public Layer {
                                                    const Tensor& grad) override;
 
   void ToDevice(std::shared_ptr<Device> device) override;
-
+  const std::vector<Tensor> param_values() override {
+    return std::vector<Tensor>{weight_, bias_};
+  }
   size_t num_output() const { return hdim_; }
   size_t num_input() const { return vdim_; }
   bool transpose() const { return transpose_; }

http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/790b7b4c/src/model/layer/dropout.cc
----------------------------------------------------------------------
diff --git a/src/model/layer/dropout.cc b/src/model/layer/dropout.cc
index bd18e51..0a4b1df 100644
--- a/src/model/layer/dropout.cc
+++ b/src/model/layer/dropout.cc
@@ -20,6 +20,7 @@
 #include "./dropout.h"
 namespace singa {
 
+RegisterLayerClass(Dropout);
 void Dropout::Setup(const Shape& in_sample, const LayerConf& conf) {
   Layer::Setup(in_sample, conf);
   dropout_ratio_ = conf.dropout_conf().dropout_ratio();

http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/790b7b4c/src/model/layer/flatten.cc
----------------------------------------------------------------------
diff --git a/src/model/layer/flatten.cc b/src/model/layer/flatten.cc
index bc175a5..e7d8fa0 100644
--- a/src/model/layer/flatten.cc
+++ b/src/model/layer/flatten.cc
@@ -20,6 +20,7 @@
 #include "./flatten.h"
 namespace singa {
 
+RegisterLayerClass(Flatten);
 void Flatten::Setup(const Shape& in_sample, const LayerConf &conf) {
   Layer::Setup(in_sample, conf);
   axis_ = conf.flatten_conf().axis();

http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/790b7b4c/src/model/layer/lrn.cc
----------------------------------------------------------------------
diff --git a/src/model/layer/lrn.cc b/src/model/layer/lrn.cc
index 2694fe9..f17b743 100644
--- a/src/model/layer/lrn.cc
+++ b/src/model/layer/lrn.cc
@@ -21,6 +21,7 @@
 #include "lrn.h"
 
 namespace singa{
+RegisterLayerClass(LRN);
 void LRN::Setup(const Shape& in_sample, const LayerConf& conf) {
   Layer::Setup(in_sample, conf);
   out_sample_shape_ = in_sample;

http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/790b7b4c/src/model/layer/pooling.cc
----------------------------------------------------------------------
diff --git a/src/model/layer/pooling.cc b/src/model/layer/pooling.cc
index 094177a..9968c2c 100644
--- a/src/model/layer/pooling.cc
+++ b/src/model/layer/pooling.cc
@@ -20,6 +20,7 @@
 #include "singa/model/layer.h"
 namespace singa {
 
+RegisterLayerClass(Pooling);
 void Pooling::Setup(const Shape& in_sample, const LayerConf& conf) {
   Layer::Setup(in_sample, conf);
   PoolingConf pool_conf = conf.pooling_conf();
@@ -48,7 +49,7 @@ void Pooling::Setup(const Shape& in_sample, const LayerConf& conf) {
     stride_h_ = pool_conf.stride_h();
   }
   CHECK_GT(stride_w_, 0u);
-  CHECK_GT(stride_h_, 0u);
+  CHECK_GE(stride_h_, 0u);  // 0 for 1D pooling
 
   pool_ = pool_conf.pool();
   CHECK(pool_ == PoolingConf_PoolMethod_AVE ||
@@ -60,10 +61,12 @@ void Pooling::Setup(const Shape& in_sample, const LayerConf& conf) {
   channels_ = in_sample.at(0);
   height_ = in_sample.at(1);
   width_ = in_sample.at(2);
-  pooled_height_ =
+  pooled_height_ = 1;
+  if (stride_h_ > 0)
+    pooled_height_ =
       static_cast<size_t>((height_ + 2 * pad_h_ - kernel_h_) / stride_h_) + 1;
   pooled_width_ =
-      static_cast<size_t>((width_ + 2 * pad_w_ - kernel_w_) / stride_w_) + 1;
+    static_cast<size_t>((width_ + 2 * pad_w_ - kernel_w_) / stride_w_) + 1;
   out_sample_shape_ = vector<size_t>{channels_, pooled_height_, pooled_width_};
 }
 

http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/790b7b4c/src/model/layer/prelu.cc
----------------------------------------------------------------------
diff --git a/src/model/layer/prelu.cc b/src/model/layer/prelu.cc
index 6eb09d9..421bcaa 100644
--- a/src/model/layer/prelu.cc
+++ b/src/model/layer/prelu.cc
@@ -20,6 +20,7 @@
 #include "./prelu.h"
 namespace singa {
 
+RegisterLayerClass(PReLU);
 void PReLU::Setup(const Shape& in_sample, const LayerConf &conf) {
   Layer::Setup(in_sample, conf);
   out_sample_shape_ = in_sample;
@@ -27,7 +28,7 @@ void PReLU::Setup(const Shape& in_sample, const LayerConf &conf) {
   format_ = conf.prelu_conf().format();
   // Push back params into param_values_
   for (const auto &spec : conf.param()) param_specs_.push_back(spec);
-  param_values_.push_back(&a_);
+//  param_values_.push_back(a_);
 }
 
 const Tensor PReLU::Forward(int flag, const Tensor &input) {

http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/790b7b4c/src/model/layer/softmax.cc
----------------------------------------------------------------------
diff --git a/src/model/layer/softmax.cc b/src/model/layer/softmax.cc
index cccb06b..6b1785c 100644
--- a/src/model/layer/softmax.cc
+++ b/src/model/layer/softmax.cc
@@ -19,6 +19,7 @@
 #include "./softmax.h"
 namespace singa {
 
+RegisterLayerClass(Softmax);
 void Softmax::Setup(const Shape& in_sample, const LayerConf& conf) {
   Layer::Setup(in_sample, conf);
   CHECK_EQ(in_sample.size(), 1u);

http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/790b7b4c/src/model/metric/accuracy.cc
----------------------------------------------------------------------
diff --git a/src/model/metric/accuracy.cc b/src/model/metric/accuracy.cc
index a95486b..789e4f6 100644
--- a/src/model/metric/accuracy.cc
+++ b/src/model/metric/accuracy.cc
@@ -44,7 +44,7 @@ Tensor Accuracy::Match(const Tensor& predict, const vector<int>& target) {
   }
   Tensor ret(Shape{batchsize});
   ret.CopyDataFromHostPtr(score, batchsize);
-  delete score;
+  delete [] score;
   return ret;
 }
 

http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/790b7b4c/src/model/optimizer/adagrad.cc
----------------------------------------------------------------------
diff --git a/src/model/optimizer/adagrad.cc b/src/model/optimizer/adagrad.cc
index 0f2119b..3ed1855 100644
--- a/src/model/optimizer/adagrad.cc
+++ b/src/model/optimizer/adagrad.cc
@@ -26,16 +26,16 @@ void AdaGrad::Setup(const OptimizerConf& conf) { delta_ = conf.delta(); }
 // history += grad*grad;
 // value = value - lr*grad/sqrt(history+delta)
 void AdaGrad::Apply(int step, float lr, const string& name, const Tensor& grad,
-                    Tensor* value) {
+                    Tensor& value) {
   if (history_gradient_.find(name) == history_gradient_.end())
-    history_gradient_[name].ResetLike(*value);
+    history_gradient_[name].ResetLike(value);
   Tensor& history = history_gradient_[name];
   Tensor tmp = Square(grad);
   history += tmp;
   Add(history, delta_, &tmp);
   Sqrt(tmp, &tmp);
   Div(grad, tmp, &tmp);
-  Axpy(-lr, tmp, value);
+  Axpy(-lr, tmp, &value);
 }
 }  // namespace singa
 #endif  // SRC_MODEL_OPTIMIZER_ADAGRAD_H_

http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/790b7b4c/src/model/optimizer/nesterov.cc
----------------------------------------------------------------------
diff --git a/src/model/optimizer/nesterov.cc b/src/model/optimizer/nesterov.cc
index eaaf445..e5354b1 100644
--- a/src/model/optimizer/nesterov.cc
+++ b/src/model/optimizer/nesterov.cc
@@ -31,18 +31,18 @@ void Nesterov::Setup(const OptimizerConf& conf) {
 // tmp = (1+mom) * history - tmp * mom;
 // value = value - tmp;
 void Nesterov::Apply(int step, float lr, const string& name, const Tensor& grad,
-                     Tensor* value) {
+                     Tensor& value) {
   if (momentum_generator_) {
     float mom = momentum_generator_(step);
     if (history_gradient_.find(name) == history_gradient_.end())
-      history_gradient_[name].ResetLike(*value);
+      history_gradient_[name].ResetLike(value);
     Tensor& history = history_gradient_[name];
     Tensor tmp = history.Clone();
     history *= mom;
     Axpy(lr, grad, &history);
     tmp *= -mom;
     Axpy(1 + mom, history, &tmp);
-    (*value) -= tmp;
+    value -= tmp;
   }
 }
 }  // namespace singa

http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/790b7b4c/src/model/optimizer/optimizer.cc
----------------------------------------------------------------------
diff --git a/src/model/optimizer/optimizer.cc b/src/model/optimizer/optimizer.cc
index 9fee18d..d098249 100644
--- a/src/model/optimizer/optimizer.cc
+++ b/src/model/optimizer/optimizer.cc
@@ -60,8 +60,8 @@ void Optimizer::Register(const string& name, const ParamSpec& specs) {
   */
 }
 
-void Optimizer::Apply(int step, const string& name, Tensor* grad,
-                      Tensor* param) {
+void Optimizer::Apply(int step, const string& name, Tensor& grad,
+                      Tensor& param) {
   // TODO(wangwei) need to consider the order of constraint and regularizer
   if (regularizers_.find(name) != regularizers_.end()) {
     regularizers_.at(name)->Apply(step, param, grad);
@@ -78,7 +78,7 @@ void Optimizer::Apply(int step, const string& name, Tensor* grad,
   float lr = learning_rate_generator_(step);
   if (learning_rate_multplier_.find(name) != learning_rate_multplier_.end())
     lr *= learning_rate_multplier_.at(name);
-  Apply(step, lr, name, *grad, param);
+  Apply(step, lr, name, grad, param);
 }
 
 void Regularizer::Setup(const RegularizerConf& conf) {
@@ -89,16 +89,16 @@ void Regularizer::Setup(const RegularizerConf& conf) {
   }
 }
 
-void Regularizer::Apply(int step, Tensor* value, Tensor* grad, float scale) {
+void Regularizer::Apply(int step, Tensor& value, Tensor& grad, float scale) {
   if (type_ == "L2" || type_ == "l2") {
-    Axpy(coefficient_ * scale, *value, grad);
+    Axpy(coefficient_ * scale, value, &grad);
   } else {
     CHECK(type_ == "NotSet") << "Unknown regularizer type = " << type_;
   }
 }
 
-void Regularizer::Apply(int step, const vector<Tensor*>& values,
-                        const vector<Tensor*>& grads) {
+void Regularizer::Apply(int step, const vector<Tensor>& values,
+                        const vector<Tensor>& grads) {
   LOG(FATAL) << "Not implemented yet";
 }
 
@@ -107,13 +107,13 @@ void Constraint::Setup(const ConstraintConf& conf) {
   threshold_ = conf.threshold();
 }
 
-void Constraint::Apply(int step, Tensor* value, Tensor* grad) {
+void Constraint::Apply(int step, Tensor& value, Tensor& grad) {
   // TODO(wangwei) implement L2 and hard constraint
   CHECK(type_ == "NotSet") << "Unknown regularizer type = " << type_;
 }
 
-void Constraint::Apply(int step, const vector<Tensor*>& values,
-                       const vector<Tensor*>& grads) {
+void Constraint::Apply(int step, const vector<Tensor>& values,
+                       const vector<Tensor>& grads) {
   LOG(FATAL) << "Not implemented yet";
 }
 

http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/790b7b4c/src/model/optimizer/rmsprop.cc
----------------------------------------------------------------------
diff --git a/src/model/optimizer/rmsprop.cc b/src/model/optimizer/rmsprop.cc
index 6a74b8e..6d77ccd 100644
--- a/src/model/optimizer/rmsprop.cc
+++ b/src/model/optimizer/rmsprop.cc
@@ -29,9 +29,9 @@ void RMSProp::Setup(const OptimizerConf& conf) {
 // history = history * rho + grad * grad * (1 - rho)
 // value = value - lr * grad / sqrt(history + delta)
 void RMSProp::Apply(int step, float lr, const string& name, const Tensor& grad,
-                    Tensor* value) {
+                    Tensor& value) {
   if (history_gradient_.find(name) == history_gradient_.end()) {
-    history_gradient_[name].ResetLike(*value);
+    history_gradient_[name].ResetLike(value);
   }
   Tensor& history = history_gradient_[name];
   history *= rho_;
@@ -39,7 +39,7 @@ void RMSProp::Apply(int step, float lr, const string& name, const Tensor& grad,
   Axpy(1 - rho_, tmp, &history);
   Sqrt(history + delta_, &tmp);
   Div(grad, tmp, &tmp);
-  Axpy(-lr, tmp, value);
+  Axpy(-lr, tmp, &value);
 }
 }  // namespace singa
 #endif  // SRC_MODEL_OPTIMIZER_ADAGRAD_H_

http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/790b7b4c/src/model/optimizer/sgd.cc
----------------------------------------------------------------------
diff --git a/src/model/optimizer/sgd.cc b/src/model/optimizer/sgd.cc
index 71071ff..2797fc6 100644
--- a/src/model/optimizer/sgd.cc
+++ b/src/model/optimizer/sgd.cc
@@ -32,20 +32,20 @@ void SGD::Setup(const OptimizerConf& conf) {
 // history = history * momentum + grad * lr
 // value = value - history
 void SGD::Apply(int step, float lr, const string& name, const Tensor& grad,
-                Tensor* value) {
+                Tensor& value) {
   if (momentum_generator_) {
     float mom = momentum_generator_(step);
     if (mom != 0) {
       if (history_gradient_.find(name) == history_gradient_.end())
-        history_gradient_[name].ResetLike(*value);
+        history_gradient_[name].ResetLike(value);
       Tensor& history = history_gradient_[name];
       history *= mom;
       Axpy(lr, grad, &history);
-      (*value) -= history;
+      value -= history;
       return;
     }
   } else {
-    Axpy(-lr, grad, value);
+    Axpy(-lr, grad, &value);
   }
 }
 }  // namespace singa

http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/790b7b4c/src/python/__init__.py
----------------------------------------------------------------------
diff --git a/src/python/__init__.py b/src/python/__init__.py
deleted file mode 100644
index c32dc40..0000000
--- a/src/python/__init__.py
+++ /dev/null
@@ -1,3 +0,0 @@
-def main():
-    """Entry point for the application script"""
-    print("Welcome to SINGA!")
\ No newline at end of file

http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/790b7b4c/src/python/device.py
----------------------------------------------------------------------
diff --git a/src/python/device.py b/src/python/device.py
deleted file mode 100644
index 0472d8d..0000000
--- a/src/python/device.py
+++ /dev/null
@@ -1,74 +0,0 @@
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements.  See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership.  The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License.  You may obtain a copy of the License at
-#
-#   http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied.  See the License for the
-# specific language governing permissions and limitations
-# under the License.
-# =============================================================================
-'''
-This script includes Device class and its subclasses for python users
-to call singa::Device and its methods
-'''
-
-from . import singa_wrap as singa
-
-
-class Device(object):
-    """ Class and member functions for singa::Device.
-
-    Create Device instances using the CreateXXXDevice.
-    """
-    def __init__(self, id, device):
-        """Device constructor given device ID.
-
-        Args:
-            id (int): device ID.
-            device: swig shared_ptr<Device>
-        """
-        self.id = id
-        self.singa_device = device
-
-    def set_rand_seed(self, seed):
-        self.singa_device.SetRandSeed(seed)
-
-    def get_host(self):
-        return self.singa_device.host()
-
-    def get_id(self):
-        return self.singa_device.id()
-
-
-class Platform(object):
-    @staticmethod
-    def get_num_gpus():
-        return singa.Platform.GetNumGPUs()
-
-    @staticmethod
-    def get_gpu_ids():
-        return singa.Platform.GetGPUIDs()
-
-    @staticmethod
-    def get_gpu_mem_size(id):
-        return singa.Platform.GetGPUMemSize(id)
-
-    @staticmethod
-    def device_query(id, verbose=False):
-        return singa.Platform.DeviceQuery(id, verbose)
-
-    @staticmethod
-    def create_raw_cuda_gpus(num):
-        return singa.Platform.CreateCudaGPUs(num)
-
-    @staticmethod
-    def create_cuda_gpu():
-        return singa.Platform.CreateCudaGPUs(1)[0]

http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/790b7b4c/src/python/initializer.py
----------------------------------------------------------------------
diff --git a/src/python/initializer.py b/src/python/initializer.py
deleted file mode 100644
index bc1e8a0..0000000
--- a/src/python/initializer.py
+++ /dev/null
@@ -1,37 +0,0 @@
-# Licensed to the Apache Software Foundation (ASF) under one
-# or more contributor license agreements.  See the NOTICE file
-# distributed with this work for additional information
-# regarding copyright ownership.  The ASF licenses this file
-# to you under the Apache License, Version 2.0 (the
-# "License"); you may not use this file except in compliance
-# with the License.  You may obtain a copy of the License at
-#
-#   http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied.  See the License for the
-# specific language governing permissions and limitations
-# under the License.
-# =============================================================================
-"""Popular initialization methods for parameter values (Tensor ojects)"""
-
-import math
-
-
-def uniform(t, low=0, high=1):
-    t.uniform(low, high)
-
-
-def gaussian(t, mean=0, std=0.01):
-    t.gaussian(mean, std)
-
-
-def xavier(t):
-    scale = math.sqrt(6.0 / (t.shape[0] + t.shape[1]))
-    t.uniform(-scale, scale)
-
-
-def msra(t):
-    t.gaussian(0, math.sqrt(2.0 / t.shape[0]))

http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/790b7b4c/src/python/layer.py
----------------------------------------------------------------------
diff --git a/src/python/layer.py b/src/python/layer.py
deleted file mode 100644
index 2151766..0000000
--- a/src/python/layer.py
+++ /dev/null
@@ -1,79 +0,0 @@
-#!/usr/bin/env python
-
-# /************************************************************
-# *
-# * Licensed to the Apache Software Foundation (ASF) under one
-# * or more contributor license agreements.  See the NOTICE file
-# * distributed with this work for additional information
-# * regarding copyright ownership.  The ASF licenses this file
-# * to you under the Apache License, Version 2.0 (the
-# * "License"); you may not use this file except in compliance
-# * with the License.  You may obtain a copy of the License at
-# *
-# *   http://www.apache.org/licenses/LICENSE-2.0
-# *
-# * Unless required by applicable law or agreed to in writing,
-# * software distributed under the License is distributed on an
-# * "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# * KIND, either express or implied.  See the License for the
-# * specific language governing permissions and limitations
-# * under the License.
-# *
-# *************************************************************/
-
-import sys
-import os
-import numpy as np
-
-from . import singa_wrap as singa
-
-from .proto.core_pb2 import *
-from .proto.model_pb2 import *
-
-
-class Layer(object):
-
-    def __init__(self, name='default', **kwargs):
-        self.singa_layer = singa.Layer()
-        self.conf = LayerConf()
-        self.conf.name = name
-        # other initialization
-        # ...
-
-    def setup(self, proto):
-        self.singa_layer.Setup(proto.SerializeToString())
-
-    def forward(self, flag, inputs):
-        return self.singa_layer.Forward(flag, inputs)
-
-    def backward(self, flag, grads):
-        return self.singa_layer.Backward(flag, grads)
-
-    def to_device(self, device):
-        self.singa_layer.ToDevice(device)
-
-    def as_type(self, dtype):
-        self.singa_layer.AsType(dtype)
-
-    def name(self):
-        return self.singa_layer.name()
-
-
-class Conv2D(Layer):
-
-    def __init__(self, in_channels, out_channels, kernel=3, stride=1,
-                 border_mode='valid',  engine='cudnn', cudnn_prefer='fatest',
-                 data_format='NCHW', use_bias=True, pad=None, W_specs=None,
-                 b_specs=None, name=None):
-
-        super(Conv2D, self).__init__(name)
-
-        conf = ConvolutionConf()
-        conf.channels = in_channels
-        conf.num_output = out_channels
-        # other fields
-        # ...
-
-        self.conf.convolution_conf.CopyFrom(conf)
-
-        self.setup(self.conf)

http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/790b7b4c/src/python/model.py
----------------------------------------------------------------------
diff --git a/src/python/model.py b/src/python/model.py
deleted file mode 100644
index 38d9950..0000000
--- a/src/python/model.py
+++ /dev/null
@@ -1,21 +0,0 @@
-#/**
-# * Licensed to the Apache Software Foundation (ASF) under one
-# * or more contributor license agreements.  See the NOTICE file
-# * distributed with this work for additional information
-# * regarding copyright ownership.  The ASF licenses this file
-# * to you under the Apache License, Version 2.0 (the
-# * "License"); you may not use this file except in compliance
-# * with the License.  You may obtain a copy of the License at
-# *
-# *     http://www.apache.org/licenses/LICENSE-2.0
-# *
-# * Unless required by applicable law or agreed to in writing, software
-# * distributed under the License is distributed on an "AS IS" BASIS,
-# * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
-# * See the License for the specific language governing permissions and
-# * limitations under the License.
-# */
-
-class Model(Object):
-    pass
-


Mime
View raw message