singa-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From kaip...@apache.org
Subject [7/8] incubator-singa git commit: SINGA-305 - Add jupyter notebooks for SINGA V1 tutorial
Date Wed, 15 Mar 2017 09:16:33 GMT
SINGA-305 - Add jupyter notebooks for SINGA V1 tutorial

Add notebooks for tensor, device, model classes; and mlp, linear regression, cnn, rbm examples;
TODO RNN examples, and distributed training, Rafiki.


Project: http://git-wip-us.apache.org/repos/asf/incubator-singa/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-singa/commit/5144bcf1
Tree: http://git-wip-us.apache.org/repos/asf/incubator-singa/tree/5144bcf1
Diff: http://git-wip-us.apache.org/repos/asf/incubator-singa/diff/5144bcf1

Branch: refs/heads/master
Commit: 5144bcf1a4c970f9165d9a164f23686eceaba749
Parents: 0be3e03
Author: wangwei <wangwei@comp.nus.edu.sg>
Authored: Tue Mar 14 18:23:59 2017 +0800
Committer: wangwei <wangwei@comp.nus.edu.sg>
Committed: Wed Mar 15 16:15:43 2017 +0800

----------------------------------------------------------------------
 doc/en/docs/installation.md                |    4 +-
 doc/en/docs/notebook/cnn.ipynb             |  907 ++++++++++++++++++
 doc/en/docs/notebook/core.ipynb            | 1151 +++++++++++++++++++++++
 doc/en/docs/notebook/index.ipynb           |  108 ++-
 doc/en/docs/notebook/installation.ipynb    |   93 ++
 doc/en/docs/notebook/mlp.ipynb             |   98 +-
 doc/en/docs/notebook/model.ipynb           |  536 +++++++++++
 doc/en/docs/notebook/rbm.ipynb             |   79 +-
 doc/en/docs/notebook/regression.ipynb      |   70 +-
 doc/en/docs/notebook/requirements.txt      |    5 +
 doc/en/docs/notebook/rnn.ipynb             |  257 +++++
 doc/en/docs/notebook/static/bp.PNG         |  Bin 0 -> 22579 bytes
 doc/en/docs/notebook/static/digit.jpg      |  Bin 0 -> 4746 bytes
 doc/en/docs/notebook/static/models.PNG     |  Bin 0 -> 43188 bytes
 doc/en/docs/notebook/static/sgd.png        |  Bin 0 -> 18725 bytes
 doc/en/docs/notebook/static/singav1-sw.png |  Bin 0 -> 24326 bytes
 python/singa/layer.py                      |   13 +-
 python/singa/tensor.py                     |    2 +
 18 files changed, 3180 insertions(+), 143 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/5144bcf1/doc/en/docs/installation.md
----------------------------------------------------------------------
diff --git a/doc/en/docs/installation.md b/doc/en/docs/installation.md
index 7772cc0..3a596d6 100755
--- a/doc/en/docs/installation.md
+++ b/doc/en/docs/installation.md
@@ -9,11 +9,11 @@ Currently, SINGA has conda packages (Python2.7) for Linux and MacOSX.
 
 1. CPU only
 
-        conda install -c nusdbsystem singa
+      conda install -c nusdbsystem singa
 
 2. GPU via CUDA+cuDNN
 
-        conda install -c nusdbsystem singa-cudax.y-cudnnz
+      conda install -c nusdbsystem singa-cudax.y-cudnnz
 
     where `x.y,z` is one of <8.0, 5>, <7.5, 5> and <7.5, 4>.
     Users need to install CUDA and cuDNN before installing SINGA.

http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/5144bcf1/doc/en/docs/notebook/cnn.ipynb
----------------------------------------------------------------------
diff --git a/doc/en/docs/notebook/cnn.ipynb b/doc/en/docs/notebook/cnn.ipynb
new file mode 100644
index 0000000..64a3a25
--- /dev/null
+++ b/doc/en/docs/notebook/cnn.ipynb
@@ -0,0 +1,907 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Classify images from MNIST using LeNet"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Dataset\n",
+    "\n",
+    "Download the [dataset](http://deeplearning.net/data/mnist/mnist.pkl.gz) to your workspace (i.e. the notebook folder)."
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "metadata": {
+    "collapsed": true
+   },
+   "outputs": [],
+   "source": [
+    "import cPickle, gzip\n",
+    "\n",
+    "# Load the dataset\n",
+    "f = gzip.open('mnist.pkl.gz', 'rb')\n",
+    "train_set, valid_set, _ = cPickle.load(f)\n",
+    "f.close()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "(50000, 784) (50000,)\n",
+      "(10000, 784) (10000,)\n"
+     ]
+    }
+   ],
+   "source": [
+    "print train_set[0].shape, train_set[1].shape\n",
+    "print valid_set[0].shape, valid_set[1].shape"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [],
+   "source": [
+    "import numpy as np\n",
+    "train_x = np.reshape(train_set[0], (50000, 1, 28, 28)).astype(np.float32, copy=False)\n",
+    "train_y = np.array(train_set[1]).astype(np.int32, copy=False)\n",
+    "valid_x = np.reshape(valid_set[0], (10000, 1, 28, 28))"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "<matplotlib.image.AxesImage at 0x7f0747e21410>"
+      ]
+     },
+     "execution_count": 4,
+     "metadata": {},
+     "output_type": "execute_result"
+    },
+    {
+     "data": {
+      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAP8AAAD8CAYAAAC4nHJkAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAADn9JREFUeJzt3X9sXfV5x/HPU8dxlhDauCmeSzMSIC3QsIbtKoCIgImR\npQgpoKqhUVWljDVdC3RsmQTLpjWb2JRNLVXKGJJZsyQVv0oLIn+wVmBV0GrgYbIQfpVfwV0TjE1w\nIYHSxLGf/eGTygXf73XuPfeeaz/vl2T53vOcc8+jk3x87r3fe8/X3F0A4vlA0Q0AKAbhB4Ii/EBQ\nhB8IivADQRF+ICjCDwRF+IGgCD8Q1IxG7mymtfkszWnkLoFQfq13dNgP2WTWrSn8ZrZS0mZJLZL+\nw903pdafpTk62y6qZZcAEnq8e9LrVv2038xaJN0i6dOSzpC0xszOqPbxADRWLa/5l0l6yd33uPth\nSXdJWpVPWwDqrZbwnyjpF+Pu782W/RYzW2dmvWbWO6xDNewOQJ7q/m6/u3e5e8ndS61qq/fuAExS\nLeHfJ2nBuPsfy5YBmAJqCf/jkhab2SIzmynpc5J25NMWgHqreqjP3Y+Y2TWSfqSxob4t7v5Mbp0B\nqKuaxvnd/QFJD+TUC4AG4uO9QFCEHwiK8ANBEX4gKMIPBEX4gaAIPxAU4QeCIvxAUIQfCIrwA0ER\nfiAowg8ERfiBoAg/EBThB4Ii/EBQhB8IivADQRF+ICjCDwRF+IGgCD8QFOEHgiL8QFCEHwiK8ANB\nEX4gKMIPBFXTLL1m1ifpoKQRSUfcvZRHU8iPzUj/E7d8ZH5d9//8Xy8sWxuZPZrc9qRTBpP12V+1\nZP21m2aWre0s3Z3cdv/IO8n62fesT9ZP/avHkvVmUFP4M3/k7vtzeBwADcTTfiCoWsPvkh4ysyfM\nbF0eDQFojFqf9i93931mdoKkB83sZ+7+yPgVsj8K6
 yRplmbXuDsAeanpzO/u+7Lfg5Luk7RsgnW6\n3L3k7qVWtdWyOwA5qjr8ZjbHzOYevS1phaSn82oMQH3V8rS/Q9J9Znb0ce5w9x/m0hWAuqs6/O6+\nR9Kncuxl2mo5fXGy7m2tyfqrF3woWX/3nPJj0u0fTI9X/+RT6fHuIv3Xr+Ym6//ybyuT9Z4z7yhb\ne2X43eS2mwYuTtY/+hNP1qcChvqAoAg/EBThB4Ii/EBQhB8IivADQeXxrb7wRi78g2T9pq23JOsf\nby3/1dPpbNhHkvW/v/mLyfqMd9LDbefec03Z2tx9R5Lbtu1PDwXO7u1J1qcCzvxAUIQfCIrwA0ER\nfiAowg8ERfiBoAg/EBTj/Dloe/7VZP2JXy9I1j/eOpBnO7la339Osr7n7fSlv7ee8v2ytbdG0+P0\nHd/+72S9nqb+F3Yr48wPBEX4gaAIPxAU4QeCIvxAUIQfCIrwA0GZe+NGNI+3dj/bLmrY/prF0JXn\nJusHVqYvr92y+7hk/cmv3nzMPR114/7fT9YfvyA9jj/y5lvJup9b/urufV9LbqpFa55Mr4D36fFu\nHfCh9NzlGc78QFCEHwiK8ANBEX4gKMIPBEX4gaAIPxBUxXF+M9si6VJJg+6+JFvWLuluSQsl9Ula\n7e6/rLSzqOP8lbTM/3CyPvLGULL+yh3lx+qfOX9Lcttl/3xtsn7CLcV9px7HLu9x/q2S3jsR+g2S\nut19saTu7D6AKaRi+N39EUnvPfWskrQtu71N0mU59wWgzqp9zd/h7v3Z7dckdeTUD4AGqfkNPx97\n06DsGwdmts7Mes2sd1iHat0dgJxUG/4BM+uUpOz3YLkV3b3L3UvuXmpVW5W7A5C3asO/Q9La7PZa\nSffn0w6ARqkYfjO7U9Kjkj5hZnvN7CpJmyRdbGYvSvrj7D6AKaTidfvdfU2ZEgP2ORnZ/0ZN2w8f\nmFn1tp/8/LPJ+uu3tqQfYHSk
 6n2jWHzCDwiK8ANBEX4gKMIPBEX4gaAIPxAUU3RPA6df/0LZ2pVn\npkdk//Ok7mT9gs9enazPvfuxZB3NizM/EBThB4Ii/EBQhB8IivADQRF+ICjCDwTFOP80kJom+42v\nnJ7c9v92vJus33Dj9mT9b1Zfnqz7/36wbG3BPz2a3FYNnD4+Is78QFCEHwiK8ANBEX4gKMIPBEX4\ngaAIPxBUxSm688QU3c1n6E/PTdZv//o3kvVFM2ZVve9Pbr8mWV98W3+yfmRPX9X7nq7ynqIbwDRE\n+IGgCD8QFOEHgiL8QFCEHwiK8ANBVRznN7Mtki6VNOjuS7JlGyV9SdLr2Wob3P2BSjtjnH/q8fOW\nJuvHb9qbrN958o+q3vdpP/6zZP0T/1D+OgaSNPLinqr3PVXlPc6/VdLKCZZ/y92XZj8Vgw+guVQM\nv7s/ImmoAb0AaKBaXvNfa2a7zWyLmc3LrSMADVFt+G+VdLKkpZL6JX2z3Ipmts7Mes2sd1iHqtwd\ngLxVFX53H3D3EXcflXSbpGWJdbvcveTupVa1VdsngJxVFX4z6xx393JJT+fTDoBGqXjpbjO7U9KF\nkuab2V5JX5d0oZktleSS+iR9uY49AqgDvs+PmrR0nJCsv3rFqWVrPddvTm77gQpPTD//yopk/a3l\nbyTr0xHf5wdQEeEHgiL8QFCEHwiK8ANBEX4gKIb6UJjv7U1P0T3bZibrv/LDyfql115X/rHv60lu\nO1Ux1AegIsIPBEX4gaAIPxAU4QeCIvxAUIQfCKri9/kR2+jy9KW7X/5seoruJUv7ytYqjeNXcvPQ\nWcn67Pt7a3r86Y4zPxAU4QeCIvxAUIQfCIrwA0ERfiAowg8ExTj/NGelJcn6C19Lj7Xfdt62ZP38\nWenv1NfikA8n648NLUo/wGh/jt1MP5z5gaAIPxAU4QeCIvxAUIQfCIrwA0ERfiCoiuP8ZrZA0nZJ\nHZJcUpe
 7bzazdkl3S1ooqU/Sanf/Zf1ajWvGopOS9Zev/GjZ2sYr7kpu+5nj9lfVUx42DJSS9Yc3\nn5Osz9uWvu4/0iZz5j8iab27nyHpHElXm9kZkm6Q1O3uiyV1Z/cBTBEVw+/u/e6+M7t9UNJzkk6U\ntErS0Y9/bZN0Wb2aBJC/Y3rNb2YLJZ0lqUdSh7sf/fzkaxp7WQBgiph0+M3sOEk/kHSdux8YX/Ox\nCf8mnPTPzNaZWa+Z9Q7rUE3NAsjPpMJvZq0aC/7t7n5vtnjAzDqzeqekwYm2dfcudy+5e6lVbXn0\nDCAHFcNvZibpO5Kec/ebxpV2SFqb3V4r6f782wNQL5P5Su95kr4g6Skz25Ut2yBpk6TvmdlVkn4u\naXV9Wpz6Ziz8vWT9rT/sTNav+McfJut//qF7k/V6Wt+fHo579N/LD+e1b/2f5LbzRhnKq6eK4Xf3\nn0oqN9/3Rfm2A6BR+IQfEBThB4Ii/EBQhB8IivADQRF+ICgu3T1JMzp/t2xtaMuc5LZfWfRwsr5m\n7kBVPeXhmn3Lk/Wdt6an6J7//aeT9faDjNU3K878QFCEHwiK8ANBEX4gKMIPBEX4gaAIPxBUmHH+\nw3+Svkz04b8cStY3nPpA2dqK33mnqp7yMjDybtna+TvWJ7c97e9+lqy3v5kepx9NVtHMOPMDQRF+\nICjCDwRF+IGgCD8QFOEHgiL8QFBhxvn7Lkv/nXvhzHvqtu9b3jwlWd/88Ipk3UbKXTl9zGk3vlK2\ntnigJ7ntSLKK6YwzPxAU4QeCIvxAUIQfCIrwA0ERfiAowg8EZe6eXsFsgaTtkjokuaQud99sZhsl\nfUnS69mqG9y9/JfeJR1v7X62Mas3UC893q0DPpT+YEhmMh/yOSJpvbvvNLO5kp4wswez2rfc/RvV\nNgqgOBXD7+79kvqz2wfN7DlJJ9a7MQD1dUyv+c1soaSzJB39zOi1ZrbbzLaY2bwy26wz
 s14z6x3W\noZqaBZCfSYffzI6T9ANJ17n7AUm3SjpZ0lKNPTP45kTbuXuXu5fcvdSqthxaBpCHSYXfzFo1Fvzb\n3f1eSXL3AXcfcfdRSbdJWla/NgHkrWL4zcwkfUfSc+5+07jlneNWu1xSerpWAE1lMu/2nyfpC5Ke\nMrNd2bINktaY2VKNDf/1SfpyXToEUBeTebf/p5ImGjdMjukDaG58wg8IivADQRF+ICjCDwRF+IGg\nCD8QFOEHgiL8QFCEHwiK8ANBEX4gKMIPBEX4gaAIPxBUxUt357ozs9cl/XzcovmS9jesgWPTrL01\na18SvVUrz95OcvePTGbFhob/fTs363X3UmENJDRrb83al0Rv1SqqN572A0ERfiCoosPfVfD+U5q1\nt2btS6K3ahXSW6Gv+QEUp+gzP4CCFBJ+M1tpZs+b2UtmdkMRPZRjZn1m9pSZ7TKz3oJ72WJmg2b2\n9Lhl7Wb2oJm9mP2ecJq0gnrbaGb7smO3y8wuKai3BWb2YzN71syeMbO/yJYXeuwSfRVy3Br+tN/M\nWiS9IOliSXslPS5pjbs/29BGyjCzPkkldy98TNjMzpf0tqTt7r4kW/avkobcfVP2h3Oeu1/fJL1t\nlPR20TM3ZxPKdI6fWVrSZZK+qAKPXaKv1SrguBVx5l8m6SV33+PuhyXdJWlVAX00PXd/RNLQexav\nkrQtu71NY/95Gq5Mb03B3fvdfWd2+6CkozNLF3rsEn0VoojwnyjpF+Pu71VzTfntkh4ysyfMbF3R\nzUygI5s2XZJek9RRZDMTqDhzcyO9Z2bppjl21cx4nTfe8Hu/5e6+VNKnJV2dPb1tSj72mq2Zhmsm\nNXNzo0wws/RvFHnsqp3xOm9FhH+fpAXj7n8sW9YU3H1f9ntQ0n1qvtmHB45Okpr9Hiy4n99oppmb\nJ5pZWk1w7Jppxusiwv+4pMVmtsjMZkr6nKQdBfTxPmY2J3sjRmY
 2R9IKNd/swzskrc1ur5V0f4G9\n/JZmmbm53MzSKvjYNd2M1+7e8B9Jl2jsHf+XJf1tET2U6etkSU9mP88U3ZukOzX2NHBYY++NXCXp\nw5K6Jb0o6SFJ7U3U23clPSVpt8aC1llQb8s19pR+t6Rd2c8lRR+7RF+FHDc+4QcExRt+QFCEHwiK\n8ANBEX4gKMIPBEX4gaAIPxAU4QeC+n8DZI6NXofNrQAAAABJRU5ErkJggg==\n",
+      "text/plain": [
+       "<matplotlib.figure.Figure at 0x7f077003c810>"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    }
+   ],
+   "source": [
+    "%matplotlib inline\n",
+    "import matplotlib.pyplot as plt\n",
+    "plt.imshow(train_x[0][0])"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Create the CNN model\n",
+    "\n",
+    "TODO: plot the net structure"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "conv1 (32, 14, 14)\n",
+      "relu1 (32, 14, 14)\n",
+      "conv2 (32, 7, 7)\n",
+      "relu2 (32, 7, 7)\n",
+      "pool (32, 4, 4)\n",
+      "flat (512,)\n",
+      "dense (10,)\n"
+     ]
+    },
+    {
+     "data": {
+      "text/plain": [
+       "<singa.layer.Dense at 0x7f0735355990>"
+      ]
+     },
+     "execution_count": 5,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "from singa import net as ffnet\n",
+    "from singa.layer import Conv2D, MaxPooling2D, Dropout, Activation, Flatten, Dense\n",
+    "from singa import optimizer, loss, metric\n",
+    "from singa import layer\n",
+    "layer.engine = 'singacpp'\n",
+    "net = ffnet.FeedForwardNet(loss.SoftmaxCrossEntropy(), metric.Accuracy())\n",
+    "net.add(Conv2D('conv1', 32, 3, 2, input_sample_shape=(1,28,28)))\n",
+    "net.add(Activation('relu1'))\n",
+    "net.add(Conv2D('conv2', 32, 3, 2))\n",
+    "net.add(Activation('relu2'))\n",
+    "net.add(MaxPooling2D('pool', 3, 2))\n",
+    "net.add(Flatten('flat'))\n",
+    "net.add(Dense('dense', 10))\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Initialize the parameters\n",
+    "\n",
+    "* weight matrix - guassian distribution\n",
+    "* bias - 0"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 6,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "conv1_weight (32, 9) 0.0764843672514\n",
+      "conv1_bias (32,) 0.0\n",
+      "conv2_weight (32, 288) 0.0803024768829\n",
+      "conv2_bias (32,) 0.0\n",
+      "dense_weight (512, 10) 0.0795410946012\n",
+      "dense_bias (10,) 0.0\n"
+     ]
+    }
+   ],
+   "source": [
+    "for pname, pval in zip(net.param_names(), net.param_values()):\n",
+    "    if len(pval.shape) > 1:\n",
+    "        pval.gaussian(0, 0.1)\n",
+    "    else:\n",
+    "        pval.set_value(0)\n",
+    "    print pname, pval.shape, pval.l1()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Set up the optimizer and tensors"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 7,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [],
+   "source": [
+    "from singa import tensor\n",
+    "#from singa.proto import core_pb2\n",
+    "from singa import device\n",
+    "from singa import utils\n",
+    "cpu = device.get_default_device()\n",
+    "\n",
+    "opt = optimizer.SGD(momentum=0.9, weight_decay=1e-4)\n",
+    "batch_size = 32\n",
+    "num_train_batch = train_x.shape[0] / batch_size\n",
+    "\n",
+    "tx = tensor.Tensor((batch_size, 1, 28, 28))\n",
+    "ty = tensor.Tensor((batch_size,), cpu , tensor.int32)\n",
+    "\n",
+    "# for progress bar\n",
+    "from tqdm import tnrange\n",
+    "idx = np.arange(train_x.shape[0], dtype=np.int32)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Conduct SGD\n",
+    "\n",
+    "1. process the training data multile time, each time is called on epoch; \n",
+    "2. for each epoch, read the data as mini-batches in random order\n",
+    "3. for each mini-batch, do BP and update the parameters  "
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 8,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "\n",
+      "Epoch = 0, training loss = 0.292792, training accuracy = 0.906530\n",
+      "\n",
+      "Epoch = 1, training loss = 0.109958, training accuracy = 0.965589\n"
+     ]
+    }
+   ],
+   "source": [
+    "for epoch in range(2):\n",
+    "    np.random.shuffle(idx)\n",
+    "    loss, acc = 0.0, 0.0\n",
+    "    \n",
+    "    bar = tnrange(num_train_batch, desc='Epoch %d' % epoch)\n",
+    "    for b in bar:\n",
+    "        x = train_x[idx[b * batch_size: (b + 1) * batch_size]]\n",
+    "        y = train_y[idx[b * batch_size: (b + 1) * batch_size]]\n",
+    "        tx.copy_from_numpy(x)\n",
+    "        ty.copy_from_numpy(y)\n",
+    "        grads, (l, a) = net.train(tx, ty)\n",
+    "        loss += l\n",
+    "        acc += a\n",
+    "        for (s, p, g) in zip(net.param_names(), net.param_values(), grads):\n",
+    "            opt.apply_with_lr(epoch, 0.01, g, p, str(s), b)\n",
+    "        # update progress bar\n",
+    "        bar.set_postfix(train_loss=l, train_accuracy=a)\n",
+    "    print 'Epoch = %d, training loss = %f, training accuracy = %f' % (epoch, loss / num_train_batch, acc / num_train_batch)    \n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Save model to disk"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 9,
+   "metadata": {
+    "collapsed": true
+   },
+   "outputs": [],
+   "source": [
+    "net.save('checkpoint')"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Load model from disk"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 10,
+   "metadata": {
+    "collapsed": false,
+    "scrolled": true
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "NOTE: If your model was saved using pickle, then set use_pickle=True for loading it\n",
+      "conv2_bias\n",
+      "conv2_weight\n",
+      "dense_weight\n",
+      "conv1_bias\n",
+      "dense_bias\n",
+      "conv1_weight\n"
+     ]
+    }
+   ],
+   "source": [
+    "for pval in net.param_values():\n",
+    "    pval.set_value(0)\n",
+    "net.load('checkpoint')"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Do prediction"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 11,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [],
+   "source": [
+    "from PIL import Image\n",
+    "img = Image.open('static/digit.jpg').convert('L')\n",
+    "img = img.resize((28,28))\n",
+    "img = np.array(img, dtype=np.float32)/255\n",
+    "img = tensor.from_numpy(img)\n",
+    "img.reshape((1,1,28,28))\n",
+    "y=net.predict(img)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 12,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "[<matplotlib.lines.Line2D at 0x7f07356e4210>]"
+      ]
+     },
+     "execution_count": 12,
+     "metadata": {},
+     "output_type": "execute_result"
+    },
+    {
+     "data": {
+      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAXcAAAD8CAYAAACMwORRAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAG7pJREFUeJzt3WuMY+dZB/D/Y3s8F/vsbOZ2nJ29zG5iuyyX0HZIy0VQ\nKKVJC4RKICVcKipQiGigIKQ2IAEfyhdUQFARulqVghCIqGojCNXS9EOh/YBasoHSNEntmU72mswZ\nz+zteO62Hz7YZ9bjjMdnxsc+9jn/nxLtnOMzPs9a4/+cfc/j9xVVBRERBUvE7wKIiMh7DHciogBi\nuBMRBRDDnYgogBjuREQBxHAnIgoghjsRUQAx3ImIAojhTkQUQDG/TjwxMaEzMzN+nZ6IqC+9+OKL\ny6o62eo438J9ZmYGFy9e9Ov0RER9SUQuuzmOwzJERAHEcCciCiCGOxFRADHciYgCqGW4i8hnRGRJ\nRL7V5HERkU+KyLyIfFNE3uZ9mUREdBBurtz/HsBD+zz+MIB07f/HAXyq/bKIiKgdLcNdVb8K4MY+\nhzwC4B+06msAjorIvV4VSEREB+fFmPs0gKt129dq+4hCqVSu4Jn/voLtcsXvUijEunpDVUQeF5GL\nInKxUCh089REXfPVuQKeevYlfPnbS36XQiHmRbhfB3Cibvt4bd+bqOp5VZ1V1dnJyZafniXqS99e\ntAEA+dqfRH7wItyfA/DBWtfMOwHcVtU3PHheor7khHrOYriTf1rOLSMi/wzgXQAmROQagD8GMAAA\nqnoOwAUA7wMwD2ANwIc6VSxRP8hZRQBAnuFOPmoZ7qr6WIvHFcCHPauIqI+VyhV8Z6mIaESwUFjF\nVqmCeIyfFaTu408dkYcu31jDVrmCH7pvHKWK4tLKqt8lUUgx3Ik85Iy3/8wDxwAAOd5UJZ8w3Ik8\nlLNsiADvPZtCRDjuTv5huBN5KG/ZODU2gtGRAcxMJ
 HjlTr5huBN5KLdoI2MaAICsafDKnXzDcCfy\nyGapjEsra8imquGeMQ1cvrGGje2yz5VRGDHciTyyUFhFuaJ3r9xTBlSB+aWiz5VRGDHciTziDME4\n4Z4xkwDYMUP+YLgTeSS3aCMWEZyeSAAATo0nEI9GOO5OvmC4E3kkb9k4M5nY+UTqQDSCM5MJzjFD\nvmC4E3kkbxV3hmQc2ZSBOYtj7tR9DHciD6xtlXDlxhqyDeGeMQ1cv7UOe2Pbp8oorBjuRB5wrs7T\ne4Q7UL2qJ+omhjuRB5xxdafH3ZHdCXeOu1N3MdyJPDBn2RiMRXBybGTX/uP3DGN4IMpwp65juBN5\nIGcVkTaTiEZk1/5IRJAxkwx36jqGO5EH8nVzyjTKmAZyixxzp+5iuBO16fbaNhbvbOwb7svFTawU\nN7tcGYUZw52oTfml2s3UZuGeYscMdR/DnahNO3PKpPYOdyf055Y47k7dw3AnalN+0UZyMIZjo0N7\nPm4eGcSRoRgnEKOuYrgTtSln2ciYSYjIno+LCLIpLtxB3cVwJ2qDqu5afamZtGkgt2hDVbtUGYUd\nw52oDcvFLdxc224Z7lnTwJ2NEqw77Jih7mC4E7Vhrsm0A40ynIaAuozhTtSGXMPqS804qzIx3Klb\nGO5EbchbNsYScUwk4/seN54cxERykB0z1DUMd6I25BZtpKead8rU4xwz1E0Md6JDUlXMWcWW4+2O\njGkgbxVRqbBjhjqP4U50SG/c3oC9WWo53u7Ipgysb5dx/dZ6hysjYrgTHVqzBTqacX4JcNydusFV\nuIvIQyKSE5F5EXlqj8dHReTfROT/RORlEfmQ96US9ZZ8LaQzU27Dvdoxk+O4O3VBy3AXkSiApwE8\nDOAsgMdE5GzDYR8G8IqqPgDgXQD+XET2bx8g6nM5y4Z5ZBCjIwOujjeGBnBsdIg3Vakr3Fy5Pwhg\nXlUXVHULwDMAHmk4RgEYUm0Z
 SAK4AaDkaaVEPSZvtZ52oFEmZXBYhrrCTbhPA7hat32ttq/eXwP4\nLgCvA3gJwEdUteJJhUQ9qFxRzC8Vm87h3kzWNLBQWEWpzLcHdZZXN1TfC+AbAI4B+H4Afy0iRxoP\nEpHHReSiiFwsFAoenZqo+67eWMPGdqXpHO7NZEwDW+UKLq2sdagyoio34X4dwIm67eO1ffU+BOBZ\nrZoH8BqAtzQ+kaqeV9VZVZ2dnJw8bM1EvnM77UCjbIpzzFB3uAn3FwCkReR07SbpowCeazjmCoB3\nA4CImACyABa8LJSolzidMump5IG+777JJETYDkmdF2t1gKqWRORJAM8DiAL4jKq+LCJP1B4/B+Dj\nAP5eRF4CIAA+pqrLHaybyFc5y8aJsWEkBlu+hXYZjkdxamyEV+7Uca5+MlX1AoALDfvO1X39OoCf\n8rY0ot41Zx38ZqqjOg0Bw506i59QJTqgrVIF3ykUDzze7simDFxaWcPGdtnjyojuYrgTHdCllVWU\nKnrocM+YBsoVxUJh1ePKiO5iuBMdkHMztJ1wB9gxQ53FcCc6oLxlIxoRnJlMHOr7T08kEIsI55ih\njmK4Ex1Q3rIxMz6CoYHoob4/HovgzGRiZ/1Vok5guBMdUP4AC3Q0kzENXrlTRzHciQ5gY7uMSyur\nSLuc5reZrGng6o11rG5yfj3qDIY70QHMLxWh6n6BjmbStZuqc0tFL8oiehOGO9EBtNsp49iZY4bT\nEFCHMNyJDiC/ZCMejWBmfKSt5zk5NoLBWITtkNQxDHeiA8gv2rhvKolYtL23TjQiSJtJ3lSljmG4\nEx1A3irurIXaLs4xQ53EcCdyyd7YxvVb622PtzsypgHrziZurW158nxE9RjuRC7lrWpny2Fng2yU\n3ZmGgB0z5D2GO5FLzidK222DdGS4KhN1EMOdyKWcZWMkHsX00WFPnu/Y6BCSgzGGO3UEw53Ipbxl\nIz2VRCQ
 injyfiCBjJrnkHnUEw53Ipdzi4RfoaMbpmFFVT5+XiOFO5MKN1S0sFzc9G293ZEwDN9e2\nUShuevq8RAx3IheccXGvr9ydXxZz7JghjzHciVzIe9wp43B+WXDcnbzGcCdyIbdo48hQDFPGoKfP\nO5GMYywRZ8cMeY7hTuRC3rKRTRkQ8aZTxiEiSE9xjhnyHsOdqAVVrc0p4+2QjCObMjBnFdkxQ55i\nuBO1sGRv4vb6tufj7Y6MaaC4WcLrtzc68vwUTgx3oha8WqCjGS7cQZ3AcCdqoVNtkI5MbT1WjruT\nlxjuRC3kFm1MJAcxloh35PlHRwZgHhnklTt5iuFO1EJ+qYhsypsFOprJmAbySwx38g7DnWgflYpi\nzrI7NiTjyJrVjplyhR0z5A2GO9E+rt9ax9pW2bMFOprJpAxsliq4cmOto+eh8HAV7iLykIjkRGRe\nRJ5qcsy7ROQbIvKyiHzF2zKJ/OF0yqS7cOVefz6idrUMdxGJAngawMMAzgJ4TETONhxzFMDfAPhZ\nVf1uAL/QgVqJui630ynT2TH3+6eqz89pCMgrbq7cHwQwr6oLqroF4BkAjzQc84sAnlXVKwCgqkve\nlknkjznLxvTRYRhDAx09T2IwhhNjwwx38oybcJ8GcLVu+1ptX70MgHtE5D9F5EUR+eBeTyQij4vI\nRRG5WCgUDlcxURflrGLHr9od2drCHURe8OqGagzA2wG8H8B7AfyhiGQaD1LV86o6q6qzk5OTHp2a\nqDNK5Qq+s1TcWci60zKmgYXCKrZKla6cj4LNTbhfB3Cibvt4bV+9awCeV9VVVV0G8FUAD3hTIpE/\nLq2sYatc2fkEaadlUwZKFcVry6tdOR8Fm5twfwFAWkROi0gcwKMAnms45l8B/IiIxERkBMA7ALzq\nbalE3dWpBTqaSXMaAvJQrNUBqloSkScBPA8gCuAzqvqyiDxRe/ycqr4qIl8E8E0AFQCf
 VtVvdbJw\nok7LWzZE7naydNqZyQSiEcEcw5080DLcAUBVLwC40LDvXMP2JwB8wrvSiPyVt2zMjCcwNBDtyvmG\nBqKYGR9hrzt5gp9QJWoit2gj3aWrdkc2xY4Z8gbDnWgPG9tlXFpZ69p4uyNjGrh8Yw3rW+WunpeC\nh+FOtIeFwirKFe34hGGNsqYBVWB+qdjV81LwMNyJ9jC31N1OGYczhw2HZqhdDHeiPeQWbQxEBTPj\nia6ed2Z8BPFohOFObWO4E+0hb9k4PZFAPNbdt0gsGsF9U0n2ulPbGO5Ee8h1YYGOZrJmkkvuUdsY\n7kQNVjdLuHpjveMLdDSTSRl4/fYG7mxs+3J+CgaGO1EDp1OlWxOGNXLmspmz2DFDh8dwJ2rgjHf7\ndeXudOjwpiq1g+FO1CC/aGMwFsGJsRFfzj99dBgj8SinIaC2MNyJGuQsG2kziWhEfDl/JCJIc+EO\nahPDnahB3sdOGUfWTDLcqS0Md6I6t9e2Yd3Z9G283ZExDSwXt7BS3PS1DupfDHeiOvnatAN+dco4\nMjvTELBjhg6H4U5Ux7mJ6fuwDDtmqE0Md6I6ectGcjCGY6NDvtYxZQxidHiA0xDQoTHcierkFm1k\nzCRE/OmUcYgIsqbBaQjo0BjuRDWqirxld32a32bStY4ZVfW7FOpDDHeimuXiFm6ubfs+3u7Ipgzc\n2SjBusOOGTo4hjtRjXPzslfC3amD4+50GAx3oppe6ZRx7LRDctydDoHhTlSTt2yMJeKYSMb9LgUA\nMJaIY9IY5JU7HQrDnaimOu2A/50y9TJmEnMMdzoEhjsRnE6Zou/TDjTKmAbyVhGVCjtm6GAY7kQA\nXr+9geJmCekeC/esaWB9u4xrN9f9LoX6DMOdCHdvWvZKj7vDmeOG4+50UAx3ItS1QU71Vrinp5IA\nOMcMHRzDnQjVK+PUkSGMjgz4XcouxtAApo8OM9zpwBjuRKh1yvT
 YkIwjYya55B4dmKtwF5GHRCQn\nIvMi8tQ+x/2AiJRE5Oe9K5Gos8oVxZxVRKY2BNJrMikDC4VVbJcrfpdCfaRluItIFMDTAB4GcBbA\nYyJytslxfwrgS14XSdRJV26sYbNU6dkr96xpYKtcweWVVb9LoT7i5sr9QQDzqrqgqlsAngHwyB7H\n/RaAzwNY8rA+oo5zxrN7rcfdwVWZ6DDchPs0gKt129dq+3aIyDSADwD4lHelEXWH0waZNntzWOb+\nqSQiAo6704F4dUP1LwF8TFX3HRQUkcdF5KKIXCwUCh6dmqg9OcvGybERjMRjfpeyp6GBKE6NJ9gx\nQwfi5qf5OoATddvHa/vqzQJ4pjYnxwSA94lISVX/pf4gVT0P4DwAzM7O8vPU1BOcOWV6WcZM8oNM\ndCBurtxfAJAWkdMiEgfwKIDn6g9Q1dOqOqOqMwA+B+A3G4OdqBdtlSpYKKz2zDS/zWRNA5eWV7Gx\nXfa7FOoTLcNdVUsAngTwPIBXAXxWVV8WkSdE5IlOF0jUSZdWVlGqaM9NO9AokzJQUWChwI4ZcsfV\nIKOqXgBwoWHfuSbH/mr7ZRF1R68t0NHM3Y4ZG2ePHfG5GuoH/IQqhVreshGNCM5MJvwuZV8z4wkM\nRIXj7uQaw51CLbdoY2Z8BIOxqN+l7Csei+DMRJJL7pFrDHcKtbxl9/x4uyOTMnjlTq4x3Cm0NrbL\nuHxjrefH2x1ZM4lrN9exulnyuxTqAwx3Cq35pSJUe3fagUbOKlFzS5yGgFpjuFNo5XamHeiPcHd+\nCXHcndxguFNo5S0b8WgEM+MjfpfiyomxEQwNRDjuTq4w3Cm0cpaN+6aSiEX7420QjQjSUwbnmCFX\n+uOnmqgD5qwisj0+p0yjjMlwJ3cY7hRK9sY2rt9a79kFOprJmElYdzZxa23L71KoxzHcKZSchS8y\nU30W7iku3EHuMNwplHZWX+qzK3enY4Y3Va
 kVhjuFUm7Rxkg8iumjw36XciD3jg7BGIyxHZJaYrhT\nKM0t2UibBiIR8buUAxERZFK8qUqtMdwplHKL/dcp48iYSeQtG6pczIyaY7hT6KwUN7Fc3OybOWUa\nZUwDN9e2UShu+l0K9TCGO4XOTqdMn4b73WkI2DFDzTHcKXT6tVPG4bRDsmOG9sNwp9DJWzZGhwcw\nZQz6XcqhTCQHMZ6IY47hTvtguFPo5C0bWdOASH91ytRLm0leudO+GO4UKqqK3KKNdJ92yjiypoH8\nIjtmqDmGO4WKdWcTdzZKfTve7sikDKxulXH91rrfpVCPYrhTqDhDGf3aKePY6Zjh0Aw1wXCnUJkL\nSLinTU4gRvtjuFOo5BZtTBqDGEvE/S6lLaPDA0gdGeIcM9QUw51CJW/ZyPT5zVRHJmWwY4aaYrhT\naFQqirxV7PshGUfWTGJuqYhyhR0z9GYMdwqNazfXsb5d3rkZ2e8ypoGtUgWXV1b9LoV6EMOdQsPp\nLOm3pfWayXJVJtoHw51CwxmfTk8FY8z9/trfg+2QtBeGO4VG3rIxfXQYxtCA36V4YiQew8mxEd5U\npT25CncReUhEciIyLyJP7fH4L4nIN0XkJRH5LxF5wPtSidqTWwxOp4wjU5uGgKhRy3AXkSiApwE8\nDOAsgMdE5GzDYa8B+DFV/V4AHwdw3utCidpRKlewUFgNzHi7I5tK4rXlVWyVKn6XQj3GzZX7gwDm\nVXVBVbcAPAPgkfoDVPW/VPVmbfNrAI57WyZRey6trGGrXAlMp4wjYxooVRSvLbNjhnZzE+7TAK7W\nbV+r7Wvm1wD8+14PiMjjInJRRC4WCgX3VRK1KR+QaQcaOX8fjrtTI09vqIrIj6Ma7h/b63FVPa+q\ns6o6Ozk56eWpifaVW7QRkbsdJkFxZjKBaEQ47k5vEnNxzHUAJ+q2j9f27SIi3wfg0wAeVtUVb8oj\n8kbesnFqPIGhgajfp
 XhqMBbF6YkEr9zpTdxcub8AIC0ip0UkDuBRAM/VHyAiJwE8C+BXVDXvfZlE\n7QnSnDKNsqbBJffoTVqGu6qWADwJ4HkArwL4rKq+LCJPiMgTtcP+CMA4gL8RkW+IyMWOVUx0QBvb\nZVxaWQvczVRHxjRw+cYa1rfKfpdCPcTNsAxU9QKACw37ztV9/esAft3b0oi8sVBYRbmigWuDdGTM\nJFSB+aUivvf4qN/lUI/gJ1Qp8ILaKeNwfmlx3J3qMdwp8HKWjYGoYGY84XcpHXFqbATxWIRzzNAu\nDHcKvDnLxpmJJOKxYP64x6IR3D+ZZLjTLsH8aSeqk7PswI63O7IpzjFDuzHcKdBWN0u4emMd2YC2\nQTrSZhKv397AnY1tv0uhHsFwp0CbW6ouZJEO6M1Uh9PmyX53cjDcKdCcoYqg9rg7duaYWeSqTFTF\ncKdAy1s2hgYiODE24ncpHTV9dBiJeJQ3VWkHw50CLWfZSE8ZiEbE71I6KhIRpE2D4U47GO4UaNU5\nZYI9JOPImGyHpLsY7hRYt9a2YN3ZDOyEYY0ypoHl4haWi5t+l0I9gOFOgZW3qjcXg97j7sjW/p68\neieA4U4B5oRc0DtlHHfbIdkxQwx3CrC8ZcMYjOHe0SG/S+mKSWMQR0cGOIEYAWC4U4DlFm2kzSRE\ngt0p4xARZKY4DQFVMdwpkFQVecveGYcOi0wqiZxlQ1X9LoV8xnCnQCoUN3FzbTs0bZCOrGnA3ihh\n8c6G36WQzxjuFEjOTcWw3Ex1OL/M8rypGnoMdwqkXG3cOSxtkI6dcOe4e+gx3CmQ8paNsUQcE8lB\nv0vpqnsScUwag+yYIYY7BVPOskPzydRGWc4xQ2C4UwCpKuasYujG2x0Z08CcVUSlwo6ZMGO4U+C8\nfnsDxc1S6MbbHdlUEuvbZVy7ue53KeQjhjsFTlgW6GhmZ+EODs2EGsOdAscJtaAvrddM2uQEYsRw\n
 pwDKL9pIHRnC6PCA36X4IjkYw/TR4Z12UAonhjsFTs6yQzve7sim2DETdgx3CpRyRTG/VEQ2pG2Q\njoxpYKGwiu1yxe9SyCcMdwqUKzfWsFmqhG5OmUbZVBJb5Qour6z6XQr5hOFOgbIz7UDIwz09VeuY\nWeQcM2HlKtxF5CERyYnIvIg8tcfjIiKfrD3+TRF5m/elErWW3+mUCfewzP1TSUSE7ZBh1jLcRSQK\n4GkADwM4C+AxETnbcNjDANK1/x8H8CmP6yRyJWfZODk2gpF4zO9SfDU0EMXMeIITiIWYmyv3BwHM\nq+qCqm4BeAbAIw3HPALgH7TqawCOisi9HtdK1NKcZYd+SMaRMQ3klxjuYeXm8mYawNW67WsA3uHi\nmGkAb7RV3R6+ki/gT77witdP29fanUGkl1btaXdJvIVCEe85a3pUTX/LpAw8/8oi3vMXX/G7lJ6g\nqP6sa22jflsVUCgqteaixv1af/xe39vs6ybf+xs/egYffegtHf37dvXfriLyOKrDNjh58uShniM5\nGAv9eOpeBG2uE9oLy4x68Dvmu+49gg+8dbr9JwqAn/v+Y7i0vIpShe2QDoGg9h9EpPZndTsizmOy\ns0/qt6XJftQ/1nDMzjl273/w9HjH/65uwv06gBN128dr+w56DFT1PIDzADA7O3uot/LbT92Dt596\n+2G+lShUzkwm8cnH3up3GeQTN2PuLwBIi8hpEYkDeBTAcw3HPAfgg7WumXcCuK2qng/JEBGROy2v\n3FW1JCJPAngeQBTAZ1T1ZRF5ovb4OQAXALwPwDyANQAf6lzJRETUiqsxd1W9gGqA1+87V/e1Aviw\nt6UREdFh8ROqREQBxHAnIgoghjsRUQAx3ImIAojhTkQUQOLXR89FpADg8iG/fQLAsofl9Du+Hrvx\n9biLr8VuQXg9TqnqZKuDfAv3dojIRVWd9buOXsHXYze+HnfxtdgtTK8Hh2WIi
 AKI4U5EFED9Gu7n\n/S6gx/D12I2vx118LXYLzevRl2PuRES0v369cicion30Xbi3Wqw7TETkhIj8h4i8IiIvi8hH/K7J\nbyISFZH/FZEv+F2L30TkqIh8TkS+LSKvisgP+l2TX0Tkd2vvkW+JyD+LyJDfNXVaX4W7y8W6w6QE\n4PdU9SyAdwL4cMhfDwD4CIBX/S6iR/wVgC+q6lsAPICQvi4iMg3gtwHMqur3oDp1+aP+VtV5fRXu\ncLdYd2io6huq+j+1r21U37yhXWNORI4DeD+AT/tdi99EZBTAjwL4WwBQ1S1VveVvVb6KARgWkRiA\nEQCv+1xPx/VbuDdbiDv0RGQGwFsBfN3fSnz1lwA+CoCLhgKnARQA/F1tmOrTIpLwuyg/qOp1AH8G\n4AqAN1BdKe5L/lbVef0W7rQHEUkC+DyA31HVO37X4wcR+WkAS6r6ot+19IgYgLcB+JSqvhXAKoBQ\n3qMSkXtQ/Rf+aQDHACRE5Jf9rarz+i3cXS3EHSYiMoBqsP+Tqj7rdz0++mEAPysil1AdrvsJEflH\nf0vy1TUA11TV+Zfc51AN+zD6SQCvqWpBVbcBPAvgh3yuqeP6LdzdLNYdGiIiqI6pvqqqf+F3PX5S\n1d9X1eOqOoPqz8WXVTXwV2fNqOoigKsikq3tejeAV3wsyU9XALxTREZq75l3IwQ3l12todormi3W\n7XNZfvphAL8C4CUR+UZt3x/U1rwl+i0A/1S7EFpASBeuV9Wvi8jnAPwPqh1m/4sQfFKVn1AlIgqg\nfhuWISIiFxjuREQBxHAnIgoghjsRUQAx3ImIAojhTkQUQAx3IqIAYrgTEQXQ/wNXckf+AMbmGAAA\nAABJRU5ErkJggg==\n",
+      "text/plain": [
+       "<matplotlib.figure.Figure at 0x7f0747e45890>"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    }
+   ],
+   "source": [
+    "prob=tensor.to_numpy(y)[0]\n",
+    "plt.plot(range(10), prob)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "## Debug\n",
+    "\n",
+    "Print l1 norm or parameter and layer feature\n",
+    "\n",
+    "1. parameter initialization\n",
+    "2. learning rate\n",
+    "3. weight decay\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 13,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "conv1_weight (32, 9) 7.97165679932\n",
+      "conv1_bias (32,) 0.0\n",
+      "conv2_weight (32, 288) 8.00566577911\n",
+      "conv2_bias (32,) 0.0\n",
+      "dense_weight (512, 10) 7.92119693756\n",
+      "dense_bias (10,) 0.0\n",
+      "\n",
+      "\n",
+      "Epoch 0\n",
+      "-->conv1: 3.886751\n",
+      "conv1-->relu1: 2.299547\n",
+      "relu1-->conv2: 603.797852\n",
+      "conv2-->relu2: 295.446167\n",
+      "relu2-->pool: 955.442017\n",
+      "pool-->flat: 955.442017\n",
+      "flat-->dense: 284901.062500\n",
+      "-->dense: 0.311904\n",
+      "dense-->flat: 0.311904\n",
+      "flat-->pool: 0.086828\n",
+      "pool-->relu2: 0.050518\n",
+      "relu2-->conv2: 9.024708\n",
+      "conv2-->relu1: 2.790344\n",
+      "relu1-->conv1: 336.182007\n",
+      "\n",
+      " loss = 79.148743, params\n",
+      "conv1_weight 10.1786603928\n",
+      "conv1_bias 12.2705039978\n",
+      "conv2_weight 8.05525493622\n",
+      "conv2_bias 0.13293106854\n",
+      "dense_weight 8.18883609772\n",
+      "dense_bias 0.00134002871346\n",
+      "\n",
+      "\n",
+      "Epoch 1\n",
+      "-->conv1: 20.362625\n",
+      "conv1-->relu1: 4.780766\n",
+      "relu1-->conv2: 2066.139404\n",
+      "conv2-->relu2: 488.272614\n",
+      "relu2-->pool: 1144.980225\n",
+      "pool-->flat: 1144.980225\n",
+      "flat-->dense: 1392742.250000\n",
+      "-->dense: 0.286131\n",
+      "dense-->flat: 0.286131\n",
+      "flat-->pool: 0.084796\n",
+      "pool-->relu2: 0.034586\n",
+      "relu2-->conv2: 7.695237\n",
+      "conv2-->relu1: 2.117975\n",
+      "relu1-->conv1: 487.341522\n",
+      "\n",
+      " loss = 70.960945, params\n",
+      "conv1_weight 16.7161483765\n",
+      "conv1_bias 34.0985298157\n",
+      "conv2_weight 8.74045848846\n",
+      "conv2_bias 0.33584010601\n",
+      "dense_weight 9.48363685608\n",
+      "dense_bias 0.00340566551313\n",
+      "\n",
+      "\n",
+      "Epoch 2\n",
+      "-->conv1: 52.302490\n",
+      "conv1-->relu1: 10.072969\n",
+      "relu1-->conv2: 12706.870117\n",
+      "conv2-->relu2: 1381.310059\n",
+      "relu2-->pool: 1888.169067\n",
+      "pool-->flat: 1888.169067\n",
+      "flat-->dense: 4740897.500000\n",
+      "-->dense: 0.350905\n",
+      "dense-->flat: 0.350905\n",
+      "flat-->pool: 0.111108\n",
+      "pool-->relu2: 0.023244\n",
+      "relu2-->conv2: 7.156909\n",
+      "conv2-->relu1: 1.548402\n",
+      "relu1-->conv1: 516.079651\n",
+      "\n",
+      " loss = 73.690216, params\n",
+      "conv1_weight 30.0410804749\n",
+      "conv1_bias 60.256187439\n",
+      "conv2_weight 11.2280254364\n",
+      "conv2_bias 0.536676049232\n",
+      "dense_weight 13.0333833694\n",
+      "dense_bias 0.00553257204592\n",
+      "\n",
+      "\n",
+      "Epoch 3\n",
+      "-->conv1: 96.670937\n",
+      "conv1-->relu1: 2.220010\n",
+      "relu1-->conv2: 16051.786133\n",
+      "conv2-->relu2: 705.551208\n",
+      "relu2-->pool: 846.462280\n",
+      "pool-->flat: 846.462280\n",
+      "flat-->dense: 4615773.000000\n",
+      "-->dense: 0.454912\n",
+      "dense-->flat: 0.454912\n",
+      "flat-->pool: 0.146553\n",
+      "pool-->relu2: 0.012394\n",
+      "relu2-->conv2: 6.071361\n",
+      "conv2-->relu1: 0.411632\n",
+      "relu1-->conv1: 328.999054\n",
+      "\n",
+      " loss = 81.878014, params\n",
+      "conv1_weight 44.8700485229\n",
+      "conv1_bias 111.767730713\n",
+      "conv2_weight 14.0987415314\n",
+      "conv2_bias 0.919560790062\n",
+      "dense_weight 18.0457553864\n",
+      "dense_bias 0.00845981575549\n",
+      "\n",
+      "\n",
+      "Epoch 4\n",
+      "-->conv1: 165.898941\n",
+      "conv1-->relu1: 0.000018\n",
+      "relu1-->conv2: 1.004575\n",
+      "conv2-->relu2: 0.049885\n",
+      "relu2-->pool: 0.214831\n",
+      "pool-->flat: 0.214831\n",
+      "flat-->dense: 1963.049194\n",
+      "-->dense: 0.560196\n",
+      "dense-->flat: 0.560196\n",
+      "flat-->pool: 0.182800\n",
+      "pool-->relu2: 0.026711\n",
+      "relu2-->conv2: 10.339769\n",
+      "conv2-->relu1: 0.006092\n",
+      "relu1-->conv1: 11.064970\n",
+      "\n",
+      " loss = 45.199768, params\n",
+      "conv1_weight 58.6796875\n",
+      "conv1_bias 162.814819336\n",
+      "conv2_weight 16.8376598358\n",
+      "conv2_bias 1.47756028175\n",
+      "dense_weight 23.0595436096\n",
+      "dense_bias 0.01112665236\n",
+      "\n",
+      "\n",
+      "Epoch 5\n",
+      "-->conv1: 241.562866\n",
+      "conv1-->relu1: 0.000000\n",
+      "relu1-->conv2: 1.477560\n",
+      "conv2-->relu2: 0.000000\n",
+      "relu2-->pool: 0.000000\n",
+      "pool-->flat: 0.000000\n",
+      "flat-->dense: 0.011127\n",
+      "-->dense: 0.668576\n",
+      "dense-->flat: 0.668576\n",
+      "flat-->pool: 0.218311\n",
+      "pool-->relu2: 0.000000\n",
+      "relu2-->conv2: 0.000000\n",
+      "conv2-->relu1: 0.000000\n",
+      "relu1-->conv1: 0.000000\n",
+      "\n",
+      " loss = 2.299366, params\n",
+      "conv1_weight 71.2618331909\n",
+      "conv1_bias 208.757034302\n",
+      "conv2_weight 19.3444824219\n",
+      "conv2_bias 2.00746393204\n",
+      "dense_weight 27.6571712494\n",
+      "dense_bias 0.0137531962246\n",
+      "\n",
+      "\n",
+      "Epoch 6\n",
+      "-->conv1: 296.653534\n",
+      "conv1-->relu1: 0.000000\n",
+      "relu1-->conv2: 2.007464\n",
+      "conv2-->relu2: 0.000000\n",
+      "relu2-->pool: 0.000000\n",
+      "pool-->flat: 0.000000\n",
+      "flat-->dense: 0.013753\n",
+      "-->dense: 0.910015\n",
+      "dense-->flat: 0.910015\n",
+      "flat-->pool: 0.297148\n",
+      "pool-->relu2: 0.000000\n",
+      "relu2-->conv2: 0.000000\n",
+      "conv2-->relu1: 0.000000\n",
+      "relu1-->conv1: 0.000000\n",
+      "\n",
+      " loss = 2.301979, params\n",
+      "conv1_weight 82.6378097534\n",
+      "conv1_bias 250.104797363\n",
+      "conv2_weight 21.6229515076\n",
+      "conv2_bias 2.484375\n",
+      "dense_weight 31.8290367126\n",
+      "dense_bias 0.0165074821562\n",
+      "\n",
+      "\n",
+      "Epoch 7\n",
+      "-->conv1: 348.356262\n",
+      "conv1-->relu1: 0.000000\n",
+      "relu1-->conv2: 2.484375\n",
+      "conv2-->relu2: 0.000000\n",
+      "relu2-->pool: 0.000000\n",
+      "pool-->flat: 0.000000\n",
+      "flat-->dense: 0.016507\n",
+      "-->dense: 0.927903\n",
+      "dense-->flat: 0.927903\n",
+      "flat-->pool: 0.302989\n",
+      "pool-->relu2: 0.000000\n",
+      "relu2-->conv2: 0.000000\n",
+      "conv2-->relu1: 0.000000\n",
+      "relu1-->conv1: 0.000000\n",
+      "\n",
+      " loss = 2.302492, params\n",
+      "conv1_weight 92.8762283325\n",
+      "conv1_bias 287.317565918\n",
+      "conv2_weight 23.6872882843\n",
+      "conv2_bias 2.9135928154\n",
+      "dense_weight 35.6038131714\n",
+      "dense_bias 0.0192515775561\n",
+      "\n",
+      "\n",
+      "Epoch 8\n",
+      "-->conv1: 394.178162\n",
+      "conv1-->relu1: 0.000000\n",
+      "relu1-->conv2: 2.913593\n",
+      "conv2-->relu2: 0.000000\n",
+      "relu2-->pool: 0.000000\n",
+      "pool-->flat: 0.000000\n",
+      "flat-->dense: 0.019252\n",
+      "-->dense: 1.218609\n",
+      "dense-->flat: 1.218609\n",
+      "flat-->pool: 0.397913\n",
+      "pool-->relu2: 0.000000\n",
+      "relu2-->conv2: 0.000000\n",
+      "conv2-->relu1: 0.000000\n",
+      "relu1-->conv1: 0.000000\n",
+      "\n",
+      " loss = 2.300582, params\n",
+      "conv1_weight 102.090713501\n",
+      "conv1_bias 320.808746338\n",
+      "conv2_weight 25.5548191071\n",
+      "conv2_bias 3.29988527298\n",
+      "dense_weight 39.0138893127\n",
+      "dense_bias 0.0218270029873\n",
+      "\n",
+      "\n",
+      "Epoch 9\n",
+      "-->conv1: 430.156555\n",
+      "conv1-->relu1: 0.000000\n",
+      "relu1-->conv2: 3.299885\n",
+      "conv2-->relu2: 0.000000\n",
+      "relu2-->pool: 0.000000\n",
+      "pool-->flat: 0.000000\n",
+      "flat-->dense: 0.021827\n",
+      "-->dense: 1.221773\n",
+      "dense-->flat: 1.221773\n",
+      "flat-->pool: 0.398946\n",
+      "pool-->relu2: 0.000000\n",
+      "relu2-->conv2: 0.000000\n",
+      "conv2-->relu1: 0.000000\n",
+      "relu1-->conv1: 0.000000\n",
+      "\n",
+      " loss = 2.302244, params\n",
+      "conv1_weight 110.383636475\n",
+      "conv1_bias 350.950500488\n",
+      "conv2_weight 27.2405776978\n",
+      "conv2_bias 3.64754581451\n",
+      "dense_weight 42.0904846191\n",
+      "dense_bias 0.0241855494678\n"
+     ]
+    }
+   ],
+   "source": [
+    "np.random.shuffle(idx)\n",
+    "ffnet.verbose=True\n",
+    "for pname, pval in zip(net.param_names(), net.param_values()):\n",
+    "    if len(pval.shape) > 1:\n",
+    "        pval.gaussian(0, 10)\n",
+    "    else:\n",
+    "        pval.set_value(0)\n",
+    "    print pname, pval.shape, pval.l1()\n",
+    "for b in range(10):\n",
+    "    print \"\\n\\nEpoch %d\" % b\n",
+    "    x = train_x[idx[b * batch_size: (b + 1) * batch_size]]\n",
+    "    y = train_y[idx[b * batch_size: (b + 1) * batch_size]]\n",
+    "    tx.copy_from_numpy(x)\n",
+    "    ty.copy_from_numpy(y)\n",
+    "    grads, (l, a) = net.train(tx, ty)\n",
+    "    print '\\n loss = %f, params' % l\n",
+    "    for (s, p, g) in zip(net.param_names(), net.param_values(), grads):\n",
+    "        opt.apply_with_lr(epoch, 0.01, g, p, str(s), b)\n",
+    "        print s, p.l1()\n"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 14,
+   "metadata": {
+    "collapsed": true
+   },
+   "outputs": [],
+   "source": [
+    "def vis_square(data):\n",
+    "    \"\"\"Take an array of shape (n, height, width) or (n, height, width, 3)\n",
+    "       and visualize each (height, width) thing in a grid of size approx. sqrt(n) by sqrt(n)\"\"\"\n",
+    "    \n",
+    "    # normalize data for display\n",
+    "    data = (data - data.min()) / (data.max() - data.min())\n",
+    "    \n",
+    "    # force the number of filters to be square\n",
+    "    n = int(np.ceil(np.sqrt(data.shape[0])))\n",
+    "    padding = (((0, n ** 2 - data.shape[0]),\n",
+    "               (0, 1), (0, 1))                 # add some space between filters\n",
+    "               + ((0, 0),) * (data.ndim - 3))  # don't pad the last dimension (if there is one)\n",
+    "    data = np.pad(data, padding, mode='constant', constant_values=1)  # pad with ones (white)\n",
+    "    \n",
+    "    # tile the filters into an image\n",
+    "    data = data.reshape((n, n) + data.shape[1:]).transpose((0, 2, 1, 3) + tuple(range(4, data.ndim + 1)))\n",
+    "    data = data.reshape((n * data.shape[1], n * data.shape[3]) + data.shape[4:])\n",
+    "    \n",
+    "    plt.imshow(data); plt.axis('off')"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 15,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "NOTE: If your model was saved using pickle, then set use_pickle=True for loading it\n",
+      "conv2_bias\n",
+      "conv2_weight\n",
+      "dense_weight\n",
+      "conv1_bias\n",
+      "dense_bias\n",
+      "conv1_weight\n"
+     ]
+    }
+   ],
+   "source": [
+    "np.random.shuffle(idx)\n",
+    "ffnet.verbose=False\n",
+    "net.load('checkpoint')\n",
+    "b=1\n",
+    "x = train_x[idx[b * batch_size: (b + 1) * batch_size]]    \n",
+    "tx.copy_from_numpy(x)\n",
+    "\n",
+    "r = net.forward(False, tx, ['relu1', 'relu2'])\n",
+    "\n",
+    "\n",
+    "    "
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 16,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [
+    {
+     "data": {
+      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAP8AAAD8CAYAAAC4nHJkAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAAIABJREFUeJztnWl0HNd1529VNYDGDhA7AYLgAoCLuEiUKFp2bEsek44k\nJ854mTO2Mx88drxEmjnOzJmcyWTiLJOTyXHG8thJTjKyYseWJpHO+EgxJdlwrMV2xEUkRYmSuIIL\nSCxcQOxAo4Guqvlwq957QDcagFBV3dT7/770w6vXXRfV9freeu8uhuu6BADQDzPXAgAAcgMmPwCa\ngskPgKZg8gOgKZj8AGgKJj8AmoLJD4CmYPIDoCmY/ABoSizKk33Y+lReuBN29R0nIqJ9q3fmWBKm\nq/910c43mSBPZuZ8Zy27ciiJpKv3GBERmY3njKWMh+YHQFMw+QHQFEx+ADQFkx8ATcHkB0BTMPkB\n0JRIt/oyEmQyEWNJOxzZPyImL4mbSq3484IgtnYNERGd/UoLERGlym1xzJjl32/DkePdAr6m8QGL\niIjW/OmBUOQyd2wW7WRDKTcc+X0aXtMuYhljCSm39fM3vPGyL5fEWppFO9Xbt+LPsyoriIjIbW4g\nIiJzaloccy1P5xYo029mloiI7O6L6R9mWiuWJxPQ/ABoCiY/AJqSc7PfN7fGdreIvviNGSIicmPS\njDen2TycqSokIiKnUB4r/qdXA5PHrKoUbbeFTTbn9ZPcoTxWGLECHl9aLPqm79xIREQFPzsWmDxE\nRJTi/720l8/f9OgJcciZnk4f78l58U/3EBFRrK1VftSly4GJ5bxxSrQLsozLdJOZt23iz3jrdGDy\nqJz9691ERFR/SJrMZb3p91XBT48SEVHvJ9aKvsZvrtzsp5oqIiLq27uKiIjWPNUjDhkxTyZbPqvZ\n1fzY1PNH9xAR0dqvhfOopgLND4Cm5Fzzp/r6iYior
 GtE9LnJJL/a6YtBRd6rVVsr+oJcMrIHb8o/\nhkbmHlQWJ91Z1iL2yIzos5IOhYF/jRr+6hoRETmLLZJ5chobJvn9AWr7wOi+FOrHb/69M0REZI+M\nLmn82G3ye2wM4Pz2Bb7mzY8OEhFRanxcHsy0gHeJX2IfvjuAsy8NaH4ANAWTHwBNybnZ7+NMTaV3\nZvEBsG/cEG0zHufP8B4XghNqrnltlpTIQ768igln/vJ4sOdfRB4iIud9HOI60i4XHgfv5nG1Pwln\nfzgbZnm5aBsN/GiWquM9b+PgG+JYxoXKAFmquR9r5EXd+l9kW7J85ziT6fdJrLmJiIjsq9flwB0d\nRETU+jT3ReH9AM0PgKbkjebPyO5tommX8i+za/I2TewFuZ2WbbsrSJztG0XbOj/ApymUGsNfmIuS\nyRa2emJJaSWVdfPXOtrOfdZn94hjlY8fClUeR13Y8tpGN/9p7twiDrkWfz/usbdDlUcldR8n3SgY\nl4t7zizr2Jofd0vZSnnbzZmcDEWO4ffytraVlF6F01Wsh2vezPAG3+IL2NMPmh8ATclrzW+eOCfb\nJv9OmdXsPJG89w5xrKhniIiIUhcuhSvQYfmzbPvbaQWFos+qYYcOqq+R407J/yEMyp86wg1lPaDC\n0xCmZ5Wc+V87xLHKx0MVJyvCWUrBaqgXbfva9bTjQVJ45CwRzYvfWMPP/Ooa0rWH2dGm4dvhONpU\n/oidtNxZGTtS7jmXOW280Xj1q/eIY42PhBSbEcqnAgDyHkx+ADQlr83+TFt3TiJBRESWurjmbdeE\nToatR9/Tj4jIvsmPH9TekjYuNDJ5+3l9jhcPEW8IZ+EqCNypRGTnmrMY6TM8nNY1upXN8bDuKmc6\n/b4W9473morA0w+aHwBNyWvNv1Sc8Ylci0BE0sHFOZxpvyZ6jF1biYgo1V2+yMhoUBdHfYspozbO\nMeVnczctfAegyvPhxImoQPMD
 oCmY/ABoyq1n9mdYdDMsz/NJ9YBywzebiObuGZte3rZcm7J+gpST\nX2F//84vHhHHclkvzSyOi7atLJTmA5f/UO6rr/s2JxiJMrugVVFGREQje/i7K38yXE9MImh+ALQl\nvzV/lqg+a+M60XaueNt+IWv7TAtWaobfILK+Lhc/s+/JP5AbU+Y4f61b/rCX5cpRFmL3PexZaE15\niU+UtF/5wvWHWOO3Pi+tNbHtFhJWNXvz2cqWsDvB16jqEN9Dc74xZO8FAARJXmt+4StPRKMf4njn\n+CDnNzd+KbfT3NRstIIRUax5NRHlJpJPxY8J3/SwTD/m5xrIddUBP34/mtWXxbE/yPEgw51Foq/p\np5wazT57PnJ5ZiqlJVnS5zn3XGFrLSxtrwLND4CmYPIDoCmGG2S5rEVwrrbncqcJAC0wG88tKZMN\nND8AmhLpgt/92+6L8nQL8vybLxIR0b7VO3MsCdPV/7po72u+PYeSSLr6OBnpvpZdOZaE6erltG35\ndn2I8vM+WgrQ/ABoCiY/AJqCyQ+ApmDyA6ApmPwAaAomPwCagskPgKbkPLAniPBJv1CnUVqyyMjF\nsaqrRdvu5HBZOnRixZ+7EoxCDgAZ+DLvubd//Kw4trWCy4YNz8r/Pel45bpmOZnH+cc6xbFV3115\nkgijgD8/sVcWAxnq9G4lxbfM8LJhOF5FM0fGsdCaLi+E9mh6IY9lk8lL1S9csqVddKVq+HoUXFXC\nd890UxoBlnqzKrwipauq0j9/NkPolVdoxR0dkzKGFGIMzQ+ApuRc8/tpsNSkGJf/gBMslPYrxScH\n+LhfqDO+/1VxLPEhLuhZcmjlYZm2msf9UHpO91zgevULCsb5ehx7e704NvUIaxRjVGoz/1om7uRx\nL33nEXHs4999T2DyqN/B6v3L+4yB/8TfcdPRFYtDsbZW0b74F6xpYzE2O1JHpCVXcpWvX3K3LGc+\ntYMzGzf
 sl2G+5U8dXrlQvnZv5iQrU62V4lB8gLNNm+OyZoHrF3yNscUyuUUWhY0/K69zkEDzA6Ap\nOdf8ftJL9Tlr/eOcIEN97pmPmlxxaBP/apaEn/MwK1YtF+i0B28uMnKZeNem5rGDRERU9w/y+d5O\nLFzxJpZgjfj1mwH75/uJJjJVC8qWhEIZX/NWcAk8Uz1XRLvtc16NAm9dwh46nfW9yfvvJCKiG9ul\nHgykyoG3DmGfuUBEREXd8ro4fs2CDG+zvIKdVnP4tRag+QHQFEx+ADQl52a/QNmu8c39pW5xFI5G\nkyMktr5N/uHJ61ry99PuvhiuAJ757yimvuU9Nl3+0lbRl6hng/K3PvwCERF1ffX94lgBvRacPBlM\nfKusVLTH9m4mIiK7kOWOJeT3VPL04QU/YyXY82smLJKsZriDHxnrXg8pD6T3qONmeESy6urSxs1s\nbyMiouLuQXEorFyM0PwAaEr+aP5FMO7i7Txz0itvfENuw9V8hxfC3HK5SGL4WycBkrpwKfDPXCnu\nDGusxFZpDRSX8oJSSyFbTi1/ck4cuz7CFoJ77O1Q5LHH5CJt2X5OLuFvDSY+tlsc87dz1z96QfSl\nrt0IXJ7k/XeJ9vRDfM+sr5QLsmdPsF5tfOQIpRGgs0+mz7XXNYoup5inopHyLMqYtIh8JzZnejpQ\nMaD5AdAUTH4ANCWvzX5/35yIyBicu+ef2tAk/7jB5uL0PdKHvfjoBdIBv0BH++cVH3mbF48eL+Tr\n0fvQbeJQ8R/xtap+MHzZfHNfnPsZ6anW+gy/Dv6m9DiseiJ4s7/45/K6lBxh/4iRVfK+qtvD5vXZ\nv5GPJFv+51Uimus/EAbGGzJGI+YVmzVXsUeis0rZ5+/0StMFXO4Mmh8ATclrzT9n+29s7haOpfzt\nb6IMbpNhY2sC8Bl/p/iRXERzF8BCwVs8cmfSPeZcb0uw+ozcLDL/VcilzZa6decVVS3r
 C7dUt28Z\nERGR3x6U22jVXpkua+Zu0Tf4fi6TXfWDkDS/v02sWEb+ne7LayXlNuBsJ8sTtKaG5gdAUzD5AdCU\n/Db7s5DJ+69gMrfVwGJNvG+bGriaUzl8zBJe4Or9DeldVvkMVxeup57o5YnLsFljLZuysXPXRF9O\nqgp7JvhYm9SDDUeSC42ODGdtg2gXXPM8XgM+BzQ/AJpyy2n+TBp/7NN7iIio8YXrUYsjQjCJMi+6\n5QI/7Vf/5znNVn29vC7Vf89pvzKFk4Ymjxde6/pbVkRkxz2PtlPS+zCKmvQsiLQQzR0cf2AoRmP8\nOMdoBK1pl4IIC49Jvexc7gvlXND8AGhKXmv+bFF9yQekz3bRsPcbfTPctFuqlnc2cnJP++hboZ5z\nMXwtf/G/3yH6HvxVjpjrPsPbRi2/Lbf3UurWVxjyFMhbauJBLmA52cgavalrQBxzPe0aurZXtLyf\nMo5u3yz6zn+8jIiI1j85KvrsoWjSt1k1q+Qftdy2KznFWOySshaSDGcNApofAE3B5AdAU/La7Fd9\n+2e2rSUioptbOLyx7vikOBY7E64Pto89Ik1DOjq68MAI8b3E1v+ZrC1w8h87iIio4wbnQkxdD95n\nfjF5iIhKf8iPH356jzkLaBEt7qkJM3o+zzn87bg8vvEx3pYNPRGLgv/4YVRKT1AnziHo1tnLRESU\nGgn//oLmB0BTDHeRNEdB4lxtz60XDgAaYDaeW1IWEmh+ADQFkx8ATYl0we+Be34tytMtyHMHfkRE\nRPtW78yxJExX/+uinW8y5Zs8D9x1f44lYZ478rxo72u+PYeSSLr6ji9rPDQ/AJqCyQ+ApmDyA6Ap\nmPwAaAomPwCagskPgKbk3Lff9cpqjW2vFX12ATsolV2R5YkKhjkTbbKBQzCHNsmUUDWneFzRRZmV\n9d2EH0o8+LEtREQ0dJt0lLSrOPmVUSDTc7gp/k0v6uNru+F7MpQ
 2iJJjfsHSs1+UtRNcT40UDUnn\nMstz85/1XNhNJTK15c8OrFgOgSP/99n1jZ48LIfhKIk7pjm02ZiVUQauly/fuh5sGK9ZxPfn5Ee2\n82u9jGUovcbnLx6QJdZS5RyaPdzOrzVvyWPmK28EKpv43FA+FQCQ9+Re83spikq9/OkL4f9Wx7z6\nkvUvymPWlo4QJMsf/GKc5ixrsTX/LFNdFnYtXKDAL1J58dOrRd+a/3Fp5fKMc0Rl27PSMjN/ubiD\niVqoM0jclNTkxivsDJTJud2d96pi3ymrGlkDSysNnw2/qKZfpah4kfH+RKzjquo0/VF5reLpwwMB\nmh8ATcHkB0BTcm72m431/DqtlC6a9RZmyssWfJ/d2y/bJ7ngYaytNQwRl43IFUdEbmrl2ej9Ek6V\nTxxa1vum6liOiovB5uq1vcKo1rBcJFtKrHbvv5bXov2Z4OQxvEVjIiKrge8nw/T0mrmIfovxQlxK\nzcXYvHqBwcsRynvwyBQybywccevnZIzvf1XpXFKE7rKB5gdAU3Ku+QVqNRev7VSUiL4bu7l0cdEo\na7GqpMyRny8VcnyC0PZLJbaWswhPbJPbbiPt/LUmq1nrrPv62+JYkDZApv8zdd8u0R7uZC022sly\nND8Tfi4X1eoiIlGunIjIXsNWgV0kt90KzrEFOSeTbqACpWttyyvDffN+WVLe8YyXmh8cC0eODEDz\nA6ApmPwAaEr+mP0ZcE6cFu2aE3OPqQbn0OfeQ0REta/JjKfmyESYouUNTiXnxh3ukF/lrLdOmmzg\nq3T54W3i2Nq/5IUte2wsFHlcRZ00/D17pjW2tfA5y+Sj3cSnuMRa2VPLW8RctjzK46F75E0imqfx\nvOy+Tqt8bLKu3uTGYouF7xC/GE3VDw6mHfMfjEY/s0f01RxgD83UpcuBygHND4Cm5LXmjylbLu4U\n+zo7E+xd5s7KX/RVf8e/oM5uqeF0+VXzra
 OmEwuPuf7b94j2xL2biIio+J9eXWj4iih8WfqhO/6C\noLcVq5Lcztbawpu5wWAUFYp2rJHLXjuTsmSZv21pqHnya6pDlmpx1G3diQfZ2y8OzQ8ACIK81vyk\nFH00KsuJiMjyXlXEs9Crb8rOPHH4EZVpnFwUfGaspBLZlgp3u22p25xDv8JOXTXfCVMaIrLSKwOZ\nFeXp7Zgcl+rhClCxIJx9MrEUpx3FOajsaA8RzV3nCgJofgA0BZMfAE3Jb7M/C67iz53X5NDcjzVx\nYovkKmlm1r/MCU9yIZVxl1yQLTkVVqDqMvE8AN1SGXRrlqc/WkaFWcZLoM74uOhzp6cXGr6yc4Xy\nqQCAvOeW0/yuvwiYyWc6LP/sZWIUSWcWtWR1VFgVnDer/zfWExFR+WXp0R9lKWofP/7g1GdLRV/7\nfwwwjddyUdJ+pdp4+48Oyb3SWEszNyIsYuun/XI7uRQ9KVGG9lg4DmvQ/ABoCiY/AJqS12a/q+zz\nT3XUEBFR4YiX6OOV19PGZ0v+EQV+OGmUpr6f2ffKF7aKvsk2XsSqP8Dmbfk/hus/rxJbt1a0T/4u\nm9TFV/i65MTUV0z86S1szltTcsfcOMgeiXP29EM2930TP/EhuQBqF7IeLnn6cPob3GCTsQg5QvlU\nAEDek9ean0bkdkfRcxfmHjOlR1astTkqibISZRIPH9vzSV/99RwuoCk4126IdseXenIoCeOnhCMi\nir2QnigjNC++LPiZfYueO7LwoJBSd6lA8wOgKZj8AGiK4Ua4l+lcbY/uZABoitl4bknPDND8AGhK\npAt++1bvjPJ0C9LVz9uED+x+IMeSMM+9+pxo72u+PYeSSLr6uPxWvn1nH1kbTsmv5fKTHpkMJd++\ns6UCzQ+ApmDyA6ApmPwAaAomPwCagskPgKZg8gOgKTn37fdLKp/76gbR13Q7F94sK5TRcVWFnLe/\nuXiEiIiuJGRu9ZGHu
 NqKe1wWpHyn2DcGRfudRueZ2zfJ9tB4lpFLwy/b7MtjlsgCpqZX9NEtkWmx\njASPS13pzfBhwfmM+0lDiIiMao4unN5YL/oStZxqrWiEYx6Kj8lEIvbgzcDkUFHrORARmaUygYi4\nVlNK3n6veo6KUVCY1rdczGJOC+Zu4YQqyRr5/Uw28XVJVsrvomCC/d9qX+NKSnPuZZToBgAECSY/\nAJqSc7PfvnadiIiaDraJvt4SNh07H5PFJEf6OTRzaIxNsmufl6b1f33yCSIieqxj3YrlMdWEIBtZ\npr69nEikrE8mVfBNWScmTbLiF7loyFSrNIfLAjD7TS9kue9BzsZb1S1Dh0uuePndlBiN6dYqIiIa\n/CQXLmn6RrDhvn4Y7Kn/IgujFI6wHmn+hXxUKrvCj2oz1fyd9f2mrEdfd5zDWq2XX1uxPKZXCJSI\n6PTv8/9uWHw9is7IrLzF17kvUSu/M+MODolu/ZrMZ+yemRc+/k5kauACoDN/zp9/5arMNl1+kK9H\n6TUl0UgVy2R+Y5iIiMa/dbc4VvJMOKXVoPkB0JSca34ftXBk54ucN13NXT6fyksyScP/HfDLGV9b\nsRxGXMknP8ELQ83PeqnDEtkXAFNekoahzfKylqVnG1s2fsbd5u/yIpk9Ki2iTAmefB1T0sjaw0/1\nNf+975RUXz8REXX+nixu6S9GZkpo4ucyLiuVZaftItY76cW0lo9zSS5sbvrP3j3jpb5yRrNbXqOf\nuIOIiG7uktbAqjMrl8kvIRf7GFuS7TQi5Z1Iz8brVwroqeYCptZ6eawkbXQwQPMDoCmY/ABoSt6Y\n/Sq+ua/ul/d8lAtyJJo9s1L52Ur+DS/0VQRg9s/BZtNR7Jv39mUd7ryPw18bjoZTXimTye7v+TuT\nk2l9JVf5ccVWa88HuGesnjMbpvcoNdYmv7TW73Ml3KCzHjrq/zqP+T4ARESu99xR0RNOxuVMJr7l\n
 +xtMy3PaOzYSEVGqjBcl1/w/+V2HlQEHmh8ATclLze/jnDgt2omvcBKHgirWqmff/31x7GOb9xER\n0cyPlYWtLBogCJL338XyTErdVTDgeWcNXBd9bhNvWxpT4VgDE3tvIyKiRK38HZ8tZe2++kX2Xgsn\n63tm7HvvEO3zn+Tby6pijVuoOK25ZWEtY81leu8O0b78Kd7Oq6mR2njyEF+3ysfllqOvaYPw9MvE\n8N4OIiJq+JL0eNxdzbUV/vkaW7vGk3KLUqTaC9jTD5ofAE3Ja82v0vkQ/zL7W0n7SKaXOvftNj72\nDemo0fG5o4HLIAo4ElHsNOend+NSO8w0s+VhnT0v+pLv41/y0jf7A5eHiKjsZyf5VclPb3oFS5Pt\n7BSU+JTcYit7yqveE5K/eMHBk6K96TDrFrOaHW+mO5vEscuf4Paabw6IPieEUtQlB7pFe/NrvOno\n1FWJPvse1qpn/1qmB9v0O1y007XDsZmqnub935kfy4Ku/1K7hYiIEndzlaPCR+T2Zeyz/D2mrga7\npgXND4CmYPIDoCm3jNmfrRRWwwE2Ycc/mYhKHKIZz+tvRprbRWO89aVKOr6GL3Hpm+GIkWkryeln\nU9ryXu3PSD9xw68DP5O+7RWIPBlMd8cLoY31yUef+Dr2ZDPWrZEDT50LXp4JZTvSbw/KMN76U/xq\n/rtdom/6A7yIWvTiicDlISJy/FBxNWTc28atPH+JiIgubJePak27+XG2+Ecw+wEAAXDLaP75xNa3\nifZ4K/+GGYcqFxgdDb4TkOUtuBERVVyOvninj1nGfuXFg0rEmq9tIigEmY2pJj6/MbLyqMeVYFh8\n79hF8noUDkdXYn0+vgNQqkreN6UX+RoFvfwIzQ+ApmDyA6Apt5zZb9VyYo3ej8q66pOdvHi1+Xfk\ngpFN0eBkyEXX/2kZk9D8XDj7+9kQOf86ONlGvFea1lF6+80ndZ9cVKu8wJKkr
 t3IlThMRxsREVmz\n0oPeOs+Pb1FeK9NbiL3x65zwpPiy8ljWfTmcc4byqQCAvCevNb9VVyfap/6EI/fKGnhra+Zt+Uu9\n6Vu8lRS2P/9imX1v/nvevmr6xbDsVLYCw8DXGOpWqLmeNf5UA/vPFz1/RL4hwoU+q5Mj1RLreBFr\ncJtMZdXyV+zl5jgh22im/H/dWb5GsQZ5X116gBdnW/fL7T9nLH37NFCRvKjLxAe3iL7JBp6Kw1v4\nvu7831fEsVQinC1saH4ANCWvNb99Qz4PdnzJ86V/L/v0F/TLZ+nUxZ5I5LHqakV79G5OGjlbIn8/\n6w/y878xtrQ49yAwvRRdI7/SJvoqT3LKKKHxo9T2yjbnyO18veKDbP00f+uYOOa8w5oIy5anXmr5\nc19mhyJT2X3d8D2+j+wr0a3N+GsyqiVkej5XHX/OsQipwcG09wUNND8AmoLJD4CmGK4bVpKgdJyr\n7dGdDABNMRvPLek5D5ofAE2JdMHvA1/8rShPtyA//9v/Q0RE+1bvXGRkNHT1y+T++SZTvsnzwS98\nIceSMC8/+qho37/13hxKInn+7ZeWNR6aHwBNweQHQFMw+QHQFEx+ADQFkx8ATcHkB0BTMPkB0JTc\nB/Z4Pn8lP5OZUpdbvCG2bi0REU3c1hCYWHmFydUkzW1c5ml4m8xV6JfpcgqU4V7gSnyQL27ty0p4\n6BVZDCJszNJSIlLCjW0Zvmu1cmBU6sKlFZ8nUSNv4+vv887lsJNb5Ul5LH6T03OoBUOdHZzopOSl\nMtEXRN5Fo4TLbaUGvIy7KwhdtqqrVyxPJqD5AdCUnGv+4gEOf82k7c2dMtmB64WlWiOcaEEN4xXt\nd6vm97SGXcaJOwxFiTQ/wanLnGGZQMTXtFYDFwk99d/WiWPt/yEczT/+bzjPfPUB+fmul6/fMDl5\nhRqiPbybS1CVB6D5a/9Flvyq+v7in5cpx7NfeJWIiAKI
 gM6m8Y0YT7tstSiiAJofAE3JueZPrObn\nQqtaJne04/yMm/EXuIXHxyNK4JFPGK+wf3vlUVng0c6SFMO+xqXCzeSGcAUjovInuQDoUnVZsoK/\n3PIAzj25WSbsKGxkve56+fhnK7Lf4qVvXeWGkuos+cBdC4xeOlZlBcvhVUYyiuPyYJYCoLZiwYUN\nND8AmoLJD4Cm5Nzs97GLrbQ+w5G5P0peOcvjMmToTX1oV1rfuxk1c7BfeNPcsFb0OcW873d9N5ue\nHX8n88FFVc8gE+aOzaJdf4DN26Bz489UF875e7hd7oFO3sVZcO1Jedt3PMfboH49iKDx8/Wppr5R\nxd9L6pKSj39eUh1/+5qIyB0ZC0U2aH4ANCVvNH8m4l3HRdvOsi0Se4Gzwsb2bBd903XxhYa/q/Ct\ngOvvUTSX95M+vJOvmZmSWYfrenmByxkPv0CmsWsrnz/hlTMfUs7p5dM3quTGWxh1Fxq/eWBJ405/\nbaNo1x9i2eIj4dhJS8k2rY4xCth6MMtKA5UDmh8ATcHkB0BT8trsn7lP5o9zvT1/K8kLJ4XXpAlp\nn/IKdL76tnzzA3otAtY8djC9z0szN/Zv94i+qQ/yolt8/6uhy+Qe4+/DN57NuHwU8z06Jz9+t+gr\n/eHhwGVwPnC7aJtJliTWc130pQb4Maj9YXnu0c/I6xUGfsyDUSCnn5Pg65GpDJw761X0IJj9AIAA\nyGvNbxel/zY5hZ7nVrmMdIqf4leroixtPCCq+IdDon3jy1xMNBfLoZniNxI18jsOVq8xs2XKLe61\nkzVrlBHcjitRpdVPcztx79YQJFK2/xT84p3kvaqefv52btBA8wOgKZj8AGhKXpv92TDs9Mpf9qa1\nGUZqjJcEZOITMlCl/ggvlOZL3bTKCzOLDwqRiUaeAkXKQpu1vi1H0hAZ/l6+YvZbTRyq7o4G65sB\nzQ+Aptx6mt9TWUVKCKZffz5ZE87
 CyC2Hp/GtavaeG9kof+PLnz6bE5HmM/FJ3uIr++HRyM89XS3j\nSEpupnvxJdeuilIcIiIy4nzvZkqz5k4mQjknND8AmpLfml95MI0/u7BTSvIjd/JwM4D8S7cantUz\ns1c6NSVq+Wud8RJmrH30nDhmz+buGXv6wd2iXflSNxER2StIbLmkc1ZJLX/dO33dUXmfFD/D95W1\nuV30JUvSI0yDRGh5z8EoE7GWZtF2J6dCkQOaHwBNweQHQFPy2uwvHJ0V7VhbKxERpep5EStZK33U\nXEtDc9/HSwJR2CUXzkq8RBDuTd4ussfCSQaxGL4Pu1nLC2glL8nYC3tyMhIZqk7K7bHKJ95OO+6H\nHU81lkQwjbyaAAADhUlEQVQiDxGRPZSep8/34rNWsedqWKa+CjQ/AJpiuG507h7O1fZ88S0B4F2L\n2XhuSaYwND8AmoLJD4CmRLrg95G1uxcfFAE/6eG93X2rdy4yMhq6+l/PtQhAQ6D5AdAUTH4ANAWT\nHwBNweQHQFMw+QHQFEx+ADQl9779Xhhupnzly8UvaxQUVgUXVJze08Gv1fJy+eGydoF0prJm2YGx\n7qD03XbeOh2oTAAEBTQ/AJqSc83vJyccuatJ9MWHuMDkZKMsr1w0ypV6klX8ezW6Xv5utf7x0oox\nLhc/7qHnV/kybXxSRlrFBjztPiMjD/1IrML9xaIv8YFQRANgxUDzA6ApmPwAaErOzX67j/OYVao5\nyW3O61aQSC/v5BvUFXduDls0UcO+4/ff5L+VBBSpLO87P9Qi2qvpWiiyAbBSoPkB0JSca34fZyJ7\nWid3XtZZ4+Aboh3z0lbZvQPBC0ZzNb44p1fVxR2VKbJu/FonEREVPRuKGAAECjQ/AJqCyQ+ApuSN\n2Z8J++4tom142f/8eutqttrUxZ609wbt7Tefgb3sl+Baq0Xf6BZeqCzulUUfrGrOxqrWWwcgH4Dm\nB0BT8lrzF5
 y4oPzBohZUsr/9+K8rKcE893q/9FIUNHzvOBEROdNyO7Lee7U2rhN9p/+Y4wLaHz4c\nmWwALAVofgA0Ja81v6M6+fhViscmiIio5Eq/OOTu2hShVIyq8edjd18U7aLGsijEAWDZQPMDoCmY\n/ABoSl6b/dkwCqTo7mH2vTfjSvFO24lcJh+rqlL+8VZ5zuQAIBvQ/ABoyq2n+b20X85Uegljo1A6\n9rgZIgKjovt3pXPSxr/gNF52roQBYAGg+QHQFEx+ADQlv81+U2bGnZ/d1ywtle3aVUREZPdHlzjD\nqq0hIqKJ924QfSMb+HK27ZchwPbNochkAmA5QPMDoCl5rflNZcvs+kdZw1peTo/an0q//yg1vjjn\n4E0iIioaahV9LadZy9tnuiOXB4DlAs0PgKZg8gOgKYZfmCIKnKvt0Z0MAE0xG88Zi4+C5gdAWyLV\n/ACA/AGaHwBNweQHQFMw+QHQFEx+ADQFkx8ATcHkB0BTMPkB0BRMfgA0BZMfAE3B5AdAUzD5AdAU\nTH4ANAWTHwBNweQHQFMw+QHQFEx+ADQFkx8ATcHkB0BTMPkB0BRMfgA0BZMfAE3B5AdAUzD5AdCU\n/w99eh37iWlJHwAAAABJRU5ErkJggg==\n",
+      "text/plain": [
+       "<matplotlib.figure.Figure at 0x7f0747eafc90>"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    }
+   ],
+   "source": [
+    "r1 = tensor.to_numpy(r['relu1'])[0]\n",
+    "vis_square(r1)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 17,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [
+    {
+     "data": {
+      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAP8AAAD8CAYAAAC4nHJkAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAADrhJREFUeJzt3WlsHsUdx/F9DseOYydxnMN2Qsj1OIFwJA0JSYqIKkET\naKsWWrWiqOIqVUpVRBFVD6RW6iFatUK8KUUF2qJCqFAPoSKIK1SUCkJIyAV1aOwcDrEdxzmcA9ux\n/Ty7fVnN/jc8m83uPM+T//fzbkbjnSH2j9GMdmdSnuc5APRJl3oAAEqD8ANKEX5AKcIPKEX4AaUI\nP6AU4QeUIvyAUoQfUCprs7O19XeL1wlTVfaG8OqeTeZ4WpZY6ztIW+8uUXfLvJVG2R0dszUcp617\nu6grt3+jchuP45R2TEHjSTd1psL8LDM/oBThB5Qi/IBShB9QyuqGnzs4GMtzhr+wQtSNf3mbWVGh\nnyr7P7FO11SLNnt/u8gor1nYKdrkavuN8lufyYk2+cPdUYaIIjJXyH/rk8sajfLEA8OiTWrz7sTG\nFISZH1CK8ANKEX5AKatr/qj2/2qVUa4/KN9hGF+ha/xi3KEhUZe7a4dR7g34uV5nvFHe9/gs0WbB\nw/Gs+VNZ+WeUrq83yoWBAdHGXbPU/JlNOyP13/3XxUa5fdULos01W+8wyvUbJok2dS9tidS/X+qM\n3Nua9Lzcl/FL19Ya5aDffZyY+QGlCD+gFOEHlCL8gFIVseE3/7tvF23T8fRyo9x6/7bztIxfpqFB\n1AVtcNnU8ZT5IlTr+ng2s4J4+byoC/PfX9V31vyZiP3P+mK7UV7ryK/smp0PIj79wuV75BZsaqm5\nKbn3wRrRpvUe+VVlkpj5AaUIP6AU4QeUqog1f99Dq43y5bcdEG1a19hb4/ulJtTKygTX/KmqcUbZ\nGxsVbVrXb02s/zBO3mu+mDXl93LfptB50NZwhCMPr
 xZ1zY9vTqw/b6e5L9F6T2JdhcbMDyhF+AGl\nCD+gFOEHlKqIDb+mJ8yNmJEnZJtM4xSjXDhxMskhGfLdPaIuM2O6qCsc7Rd1UQRt8JWboA0+v66f\nmi8izXm0+M/EJcnNvVDSGVF18m7z3yPMv+FFDSHRpwMoW4QfUIrwA0pVxJo/jP7bFxrlxqftrR+D\nxLW+v1Rk58wWdUmt8YNOFup7wFxPt7x+XLQp7OlIZDyO4ziZyebJQYe+uVi0mfWY3X0IZn5AKcIP\nKEX4AaUIP6BUyn89VJLcvtyleb42UEbSTZ3ybPugdkkPBEB5IvyAUoQfUMrqSz5rW+SpqkEfOCSl\nrds8HTVwPBa19e4SdaUcU7mNx3HkmMptPI5Tfr+zsJj5AaUIP6AU4QeUIvyAUqX/qs+NeklTEalQ\n7zlUpEzrfKM81jxRtIl6131SMtOmibrCsWOxPDu95EqjfOTGyaLN1PfPGeWqk8Oijbs7uSu9spfN\nMspefcBx7wXXLO7dl9h4HIeZH1CL8ANKEX5AqZKv+Yduu94oNzx0SLS5fcYOo/ziopbExjP4JXM8\nE/7yTmJ9RVXo2G+U08kdQBOboPX9mvfMdfema8ZHera7a49RnhHivRe3eJNY5Q93W+6xOGZ+QCnC\nDyhF+AGlCD+gVMk3/Ope222UR/5+TrR50Sm+wZdtbjLK+b6jkcbj3+BLT5gg2riDg5GenZTj31gl\n6qb+rrRHl4fx7O7VRnmBk9yLSanqaqPsjYyINkO3Xy/qav9mb8P3xNfN32PjM1zXBSABhB9QivAD\nSpV8ze/61l7e6mtFm4Efmi+DDG6ZKtpc9jPfVUcxfdgTtL5P19QY5dn/ln11rZAfjiQlaH2fWmZe\nB+Vtb7c1nNBq34v2Uk8UYo0f8Pdhc30fJOk1vh8zP6AU4QeUIvyAUoQfUKrkG35+qbffE3VTPmve\n8jXFCfiMzeLJPe4580WkrhXn
 aVhCp66oN8qTtp+nYQkNXi1f6LIlMz3gZKGj/SUYSekw8wNKEX5A\nKcIPKEX4AaXKbsPP8byiTfxHVzuO4wwubDTKNa9si21I5S47U371OOn5LSUYycdYeY2oWvSdLqOc\n0CHugYI290bXLRd14zZeun9HzPyAUoQfUIrwA0qlvBBr7Li4fTl7nQFKpZs6Q73xxswPKEX4AaUI\nP6AU4QeUsvqSz9qWJTa7E9p6zUvcym08jlPaMZXbeByH31kxQeMJi5kfUIrwA0oRfkCp8vuwB4Z0\nba2oc4eGSjCS//NffTV8szxuva7dd13a6Jhok+/pjXVc5SyzeKFRLrTvFW1SWTOOXj6f6JiY+QGl\nCD+gFOEHlCL8gFIVueH30cZ5oq5u3YESjOT8hj8vz/Me//LWC39QJhNL/5H6Pg//vXc1r8hnJ7tV\n9fG8T8qXblJvmS/DdPxhmWjTeu+OgIdd+Ieo2XlzRF0+YINPdJXwBp8fMz+gFOEHlCL8gFIVseZ3\nbzDXcHXron/MYEv3bfIs2tzLF/4c9+xZUZeury/aJs41fqXxr+8dx3E++vJKo9x6T3KnG+cPdIm6\n7KyZRrnQPEW08ba9n9SQAjHzA0oRfkApwg8oRfgBpSpiwy/9prmB8+GPVos2s3+y2dZwBG+1/Kot\nd/f2xPrzb/BlcvKlp6GceX1Z9at2r51y1yw1yulNO63271f3UvENvlNfWyXqJv/p7Vj69wbNLzG9\nbT2xPPdiMPMDShF+QCnCDyhVEWt+v1Ku74OkNu8WdZkFc0VdYd/BRPovdMqPmkaXTTfK1aJFskq9\nxo8irvV9kMLAQNE2vY+Ye1ktv07275yZH1CK8ANKEX5AKcIPKFURG35jn77OKFf9890SjSS8Dx6Z\nJupa1yez4Ze+apGoq/9zcl+tVaLuH5ibabMes7xpnDZPZMrUTRBNkt7g82PmB5Qi/IBShB9QKuVF\nOJ00Krc
 vZ68zQKl0U2cqVLukBwKgPBF+QCnCDyhF+AGlrL7ks7ZFXqNkU1uveSJQuY3HcUo7pqDx\nrJt9naizea2U+J3NXCobWdy0Dvo3unXRjUbZHT5nazjOxkPRj2hn5geUIvyAUoQfUIrwA0qV/Ku+\ndE2NUT7xFbmhkx0xN3Rqj46KNplNvqO0XHlXXhQjtywXdbVdp8yKMbkBltSRXY7jOKnlVxvl0QZ5\nSFdcXz4mubmXmTzJKBdOnS7+QwGbe6nrrjKbvPufixrXhfIK5t9aespk2eajQaPsDg6KNrYx8wNK\nEX5AKcIPKFXyNb97znwhouE5eXyyf41r8x7z6tfkNVfx7CZE5//vr7Lcf8/3zVNxZv4i2gk0odb4\nIWR6jhtle68gBSsc7S/aJrN4ofy59r1JDOe8mPkBpQg/oBThB5Qi/IBSJd/wCyPMBt/pO1ca5Ukv\nJHd0dXbu5Ub51PJm0SbMffBxybTOF3XDcxuMcnX/sGjj7WyP1F/UDb6k5I/0GeXsrJmyTXePreGI\nF9ccR25s297cC8LMDyhF+AGlCD+gVEWs+bPz5hjl3nUtos30J+2tQwvdR4xy3cFD1voOUujYL+rG\ndZjl/b9cJdrM25nUiErL5vo+iH997zjyyrl//fEZ0WbVI+uN8sQNye4bMfMDShF+QCnCDyhF+AGl\nKmLDL3+gyyhPf7IrsJ0t3pg8Sajczfue/FpSs8y0aaKucOxYYv35T1YKOqJ9omPvxTDHYeYH1CL8\ngFKEH1CqItb8YWSubDXKhT0d52kJG/ofME/7sfkSVhhJru8rBTM/oBThB5Qi/IBShB9QKuVZvNvc\n7cvZ6wxQKt3UmQrVLumBAChPhB9QivADSll9ySfoYwab2np3GeVyG4/jlHZM5TYex6mM39mti240\nyoUzZ2wNJ3A8YTHzA0oRfkApwg8oRfgBpS6Zr/pgT+FTnzDKPWvk9VTnmseM8oINedEm
 val0Z4en\nli0Wdd72aNeXuSMjFzscx3EcJ9s0wyh7Q/KKtTg3E5n5AaUIP6AU4QeUumTW/IcfNU+Ouezn0U6O\nGbtpmVGe8mN5FdeejeapQUF9ZefMFnX5rg8jjancZN7YYZRnvxHtOf3f8p3285t4Tvs597kVom7C\nvgGjXAhY36eqxok6myc15/uOGuUDQVesxXgKMzM/oBThB5Qi/IBShB9Q6pLZ8Jvz5AdGuRDxOVWv\nbzfKZ1+XbS5zjhd9zqWyuRfVvidWGuXUdHln/fyvJnOcd80/too6/9/DsfVyM23aU+V1pVnumaOi\nLurfdRBmfkApwg8oRfgBpSpyzW/zBZqgvgavMD/AqH5tm2iTWn61qPO2vR/LmPzXSw/cPF+0mbjB\n7nXPfgseKt5/trnJKOeP9CU1HMdbfa1RDlrfZxYvFHWF9r2JjamY/35bXiOee/BAbM9n5geUIvyA\nUoQfUIrwA0pV5IZf0ObeifvMlzYan43nhY2gvqpDbC66VfL/q6HuUArBf7f8xA2Vedd8kht8fqnN\nu4u28aoyFkYSXu7BdxJ9PjM/oBThB5Qi/IBShB9QqiI3/ILEtcEXxpk7zC/W+m6Sx1K33veureGU\npXR9vVF2z54t0UiCDdwlv+preK68vupLGjM/oBThB5Qi/IBSKc/zrHXm9uXsdQYolW7qDPU+GTM/\noBThB5Qi/IBShB9QyupLPmtbltjsTmjr3WWUy208gE3M/IBShB9QivADSlXkhz3ZuZeLuvzBQyUY\nCVC5mPkBpQg/oBThB5Qi/IBSZbfhl66tFXXu0JBRTnJzz3+nW5gjn4OMrlsu6sZtlHf6AaXCzA8o\nRfgBpQg/oFTZrfn96/sgJ+6XJ682Ph3Pyav+Nb7/DnnHCXfNVO3uw/Lnog8LiB0zP6AU4QeUIvyA\nUoQfUKrsNvzCCNrcO32neYXWpBe2xNKXNzxctI17gzwRKP8mp/S
 gvDHzA0oRfkApwg8oVZFr/iBu\nNtQNRRescOp0Is8FSo2ZH1CK8ANKEX5AKcIPKFWRG36ZqY2iruG5eL7qiyId8EJP0BgLx0/YGA4Q\nCjM/oBThB5Qi/IBSKc/zrHXm9uXsdQYolW7qDPXGGzM/oBThB5Qi/IBShB9QyuqGH4DywcwPKEX4\nAaUIP6AU4QeUIvyAUoQfUIrwA0oRfkApwg8oRfgBpQg/oBThB5Qi/IBShB9QivADShF+QCnCDyhF\n+AGlCD+gFOEHlCL8gFKEH1CK8ANK/Q/zQX1soXEu2gAAAABJRU5ErkJggg==\n",
+      "text/plain": [
+       "<matplotlib.figure.Figure at 0x7f07356cc790>"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    }
+   ],
+   "source": [
+    "r2 = tensor.to_numpy(r['relu2'])[0]\n",
+    "vis_square(r2)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 18,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [
+    {
+     "name": "stdout",
+     "output_type": "stream",
+     "text": [
+      "(32, 288)\n"
+     ]
+    }
+   ],
+   "source": [
+    "p=net.param_values()[2]\n",
+    "print p.shape"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 20,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [
+    {
+     "data": {
+      "image/png": "iVBORw0KGgoAAAANSUhEUgAAAP8AAAD8CAYAAAC4nHJkAAAABHNCSVQICAgIfAhkiAAAAAlwSFlz\nAAALEgAACxIB0t1+/AAACwNJREFUeJzt3etv1vUdxvHvfZeWHjkptIUB0lJAJohyUvFQNVuGWZhh\nLjjnBupQqqDL3OKcPlh4sGxL5kaioImniUg2N6fZnBPnsFMHBhy2Ujy05ShQKgXKobS0ve/9B7u+\nD375ueR6vx5fuX8f7pur3yeffH+ZfD4fAPjJftEDAPhiUH7AFOUHTFF+wBTlB0xRfsAU5QdMUX7A\nFOUHTA1J82Hz/v6gXCfsbDtPfk5Fe4HMND2wVmamNC6VmeKSczIzrLhPZkII4Z2ZL8nMvYfmysxf\nd82QmWxnkcy03fK4zNS+eZvMzJ60X2a2fTJJZvYsfFJmpjzXIDNX1X8oM1sPTpSZEEJouXyDzJzO\n9crMxZv13CMbi2Vm++p1MpOtas3IUODkB2xRfsAU5QdMUX7AFOUHTFF+wBTlB0xRfsBUqks+Y8u7\nZaYz6CWffEJTD3TppYq+7lKZOTQiF/fAmTryyZx+mZmw6ZjM7OsZEzORVHBAf0dd1WUyUz6qJ4lx\nQm68XqhpX32hzGSXn0linBBCCPPfu0NmcqcLZebE1CSmicfJD5ii/IApyg+YovyAKcoPmKL8gCnK\nD5ii/ICpVJd89q+fLDMVpfoSkomLdycxTiipPi0z+SPDZWZ07edJjBNCCKHzlWkyc6p5hMxc+ESn\nftgiHXnm5sdkZvlTK2WmvyLinZDzdSRk9Od0LNOLQANt+ncNIUTNlPtQf1ZBmZ57WDL/raNx8gOm\nKD9givIDpig/YIryA6YoP2CK8gOmKD9gKpPPRyxfJCTXUZfewwBTvK4LwP9E+QFTlB8wRfkBU5Qf\nMEX5AVOUHzBF+QFTqd7kU/PHu2Qme36fzuwrkZlPl
 66Tmdo/rJCZ3HD9+qySCj1zCCHsuuJ5mdna\nOygz33vvdpnJ5/WeR2v9szLzVHeVzHzY8yWZ2XygTmaa5m2UmQ/69He9+G/3ykz57gKZCSGE5vvX\nykzNi/r/Ue1FB2XmcPcwmdl52QaZicXJD5ii/IApyg+YovyAKcoPmKL8gCnKD5ii/ICpVJd8Cs7q\nvzXFJedkZqC3NIlxwh3XbZaZjc9fLzM9M3NJjBNCCOH+n9wjMyMiXml2fHoS04Tw6GOLZebELP2b\nFVXoTIwfLr9bZqY/vF9mWkrGJTFOCCGErN4DC+27xspMQU/EWXxZxECROPkBU5QfMEX5AVOUHzBF\n+QFTlB8wRfkBU5QfMJXqks+Y9/XbuspeLpSZ3Tcls1Tzp8euk5meK8/KTMEQfftOrOI7D8nMod2V\nMlPZGHFTza06Url4n8ycaNcLLEXbyvXDrtKR41OHysxnWy6QmdKTUW+0CuFrEZlqfbvQhBd01Q5c\nHzlTQjj5AVOUHzBF+QFTlB8wRfkBU5QfMEX5AVOUHzBF+QFTmXxeb90lJddRl97DAFPZqtaoVUFO\nfsAU5QdMUX7AFOUHTFF+wBTlB0xRfsAU5QdMpXqN14IfrJCZY986IzPFjRUys+Ona2Vm0mvfl5kh\nR/W1YuX74q5f2vGwnmlrr74S7LsvrpSZohN6ppZVep5pTzbITCann9U7Xr+rb8/CJ2Vm0it3ykyM\nwhMR15yFED5duk5mLtl2s8zMq9bXoTW+donMfLxc/2axOPkBU5QfMEX5AVOUHzBF+QFTlB8wRfkB\nU5QfMJXqks/ha/RFPvnjxTLTf0VPEuOEuqcHZKb9m/orGihN7h1rK3+uF3gGZ+jv8Yobm5MYJwxv\n05nT3+iWmZIdwxOYJoRh1adkpq9ppMz01+h3MMY61TZCZjYdjXhXYaX+/5gkTn7AFOUHTFF+wBTl\nB0xRfsAU5QdMUX7AFOUHTKW6
 5FN4TP+tKbpIL3EMeUMvVYR6Hdlzj86Ub9Mzj7rhoP6gSFc2bJOZ\nv7w9R2a2/+5i/bCH35WRuav+IzOvNs+QmeF6DyjKmZ6hMjN63hGZOdo0JolxQgghZPsilrxK9AJP\n5vOSBKaJx8kPmKL8gCnKD5ii/IApyg+YovyAKcoPmKL8gKlMPq9vhUlKrqMuvYcBprJVrVFXS3Hy\nA6YoP2CK8gOmKD9givIDpig/YIryA6YoP2Aq1Zt8bt1bLzMrq96UmYZfr5KZHQ+tlZkpjUtlpr9b\n3xzzwNWvykwIIawYoW/8qXtrmczkD+lXmg3t0n/XW1bp7+iGT26QmQvKj8nM5r11MvPRgvUyE6N+\n540y07FlbNRnfbxcf0e37LlWZra2TZKZa6a2yswzE96WmVic/IApyg+YovyAKcoPmKL8gCnKD5ii\n/IApyg+YSnXJp2XDdJlZ/fsCmTn5UC6JcaKMu+CozPzqH1+P+qwVNz0hM8M361c2dc3Vr36qnbs3\nZiRp8MHzZeb1JRNkZlRzxOUyC3Tky4/eLTMDJfrCqNHzO/TDIm39YIrMZM/qf3/1zITeaRaJkx8w\nRfkBU5QfMEX5AVOUHzBF+QFTlB8wRfkBU6ku+Sy6s1FmNl41W39QQvsZZaV9+lHHhsnMy4vWRD5R\n38DTNXtQZgqP6p+t642JepyIsVu/o5eO8kV65vpV2/XDIvROPyszBQV6CexIU2XcA2fqyLxZ+gae\nptenycw7nbX6YZXNOhOJkx8wRfkBU5QfMEX5AVOUHzBF+QFTlB8wRfkBU5l8Xt96kpRcR116DwNM\nZataI65N4uQHbFF+wBTlB0xRfsAU5QdMUX7AFOUHTFF+wFSqN/l85du3yUzvqCKZ6Rmt/2a9/7N1\nMnPp6gaZORlxuUp2/BkdCiF8evVzMnPXZ5fLTNPRcTJzWeVemflttb5dZ/ILK2SmYo/+PfrLZSTs\nvG+tzNS
 8qOfJntM7LqWHo/ZgQvOP9EyT31omM9fW6tt+Wn4zQ2b+/cjjMhOLkx8wRfkBU5QfMEX5\nAVOUHzBF+QFTlB8wRfkBU6ku+bQtKdShQv2qpdnT2hOYJoRTNTpz0bzdMtPUEvFqrEi7fqHfD9V/\nvv6bvemr+tVgIWLJJzOol2FOj9cXNA2W6t81Rr5YvxqsamqXzAw0jUlinBBCCL+c85LMrNlzvcxM\nvm9XEuNE4+QHTFF+wBTlB0xRfsAU5QdMUX7AFOUHTFF+wFSqSz6ZsgGZKdupl1PGzDqdxDhhsFgv\nnnSf0/Nk+5L7G9p7+3GZWV7zrsw80qyXSmKU79NLPrmh+nN6qpL5jmpqj8hMWeE5mWmfkNxvtubH\nN8vMgYX6cw6WjdChif+KmCgOJz9givIDpig/YIryA6YoP2CK8gOmKD9givIDpig/YCqTz+srmJKS\n66hL72GAqWxVa9SLCDn5AVOUHzBF+QFTlB8wRfkBU5QfMEX5AVOUHzCV6jVeU59ukJlMxCvdsv16\nh2FXw1qZuXT7Epk5cbxMZoaW9stMCCF8tGB9VA5IAyc/YIryA6YoP2CK8gOmKD9givIDpig/YIry\nA6ZSXfIpmHZKZmZVH5SZ/adGJjFOGPzneTIz6oy+fOjkdYNJjAOkipMfMEX5AVOUHzBF+QFTlB8w\nRfkBU5QfMEX5AVOpLvkM+3O5zGypnywzY8d3JTFOOHlxn8yUfzRUZgY6S5IYB0gVJz9givIDpig/\nYIryA6YoP2CK8gOmKD9givIDpjL5vL6pJim5jrr0HgaYyla16vfZBU5+wBblB0xRfsAU5QdMUX7A\nFOUHTFF+wBTlB0yluuQD4P8HJz9givIDpig/YIryA6YoP2CK8gOmKD9givIDpig/YIryA6YoP2CK\n8gOmKD9givIDpig/YIryA6YoP2CK8gOmKD9givIDpig/YIryA6YoP2Dqv/x3CAzTt8/U
 AAAAAElF\nTkSuQmCC\n",
+      "text/plain": [
+       "<matplotlib.figure.Figure at 0x7f07355967d0>"
+      ]
+     },
+     "metadata": {},
+     "output_type": "display_data"
+    }
+   ],
+   "source": [
+    "vis_square(tensor.to_numpy(p)[0].reshape(32, 3,3))"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "collapsed": true
+   },
+   "outputs": [],
+   "source": []
+  }
+ ],
+ "metadata": {
+  "anaconda-cloud": {},
+  "kernelspec": {
+   "display_name": "Python [conda env:conda]",
+   "language": "python",
+   "name": "conda-env-conda-py"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 2
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython2",
+   "version": "2.7.13"
+  },
+  "widgets": {
+   "state": {
+    "0678ea185e8c48a9ab20bdb956e6b18a": {
+     "views": [
+      {
+       "cell_index": 13
+      }
+     ]
+    },
+    "1373fb7b9e754b639a3f27ebf0372d70": {
+     "views": [
+      {
+       "cell_index": 13
+      }
+     ]
+    },
+    "49561f2ab00b457f82357766d967ca8b": {
+     "views": [
+      {
+       "cell_index": 13
+      }
+     ]
+    }
+   },
+   "version": "1.2.0"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}

http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/5144bcf1/doc/en/docs/notebook/core.ipynb
----------------------------------------------------------------------
diff --git a/doc/en/docs/notebook/core.ipynb b/doc/en/docs/notebook/core.ipynb
new file mode 100644
index 0000000..c16cc1f
--- /dev/null
+++ b/doc/en/docs/notebook/core.ipynb
@@ -0,0 +1,1151 @@
+{
+ "cells": [
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "collapsed": true
+   },
+   "source": [
+    "# SINGA Core Classes\n",
+    "\n",
+    "<img src=\"http://singa.apache.org/en/_static/images/singav1-sw.png\" width=\"500px\"/>"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "collapsed": true
+   },
+   "source": [
+    "# Device\n",
+    "\n",
+    "A device instance represents a hardware device with multiple execution units, e.g.,\n",
+    "* A GPU which has multile cuda streams\n",
+    "* A CPU which has multiple threads\n",
+    "\n",
+    "All data structures (variables) are allocated on a device instance. Consequently, all operations are executed on the resident device.\n",
+    "\n",
+    "## Create a device instance"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 1,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "<singa.singa_wrap.Device; proxy of <Swig Object of type 'std::shared_ptr< singa::Device > *' at 0x7f69a05ff330> >"
+      ]
+     },
+     "execution_count": 1,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "from singa import device\n",
+    "default_dev = device.get_default_device()\n",
+    "gpu = device.create_cuda_gpu()  # the first gpu device\n",
+    "gpu"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "** NOTE: currently we can only call the creating function once due to the cnmem restriction.**"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": null,
+   "metadata": {
+    "collapsed": true
+   },
+   "outputs": [],
+   "source": [
+    "gpu = device.create_cuda_gpu_on(1)  # use the gpu device with the specified GPU ID\n",
+    "gpu_list1 = device.create_cuda_gpus(2)  # the first two gpu devices\n",
+    "gpu_list2 = device.create_cuda_gpus([0,2]) # create the gpu instances on the given GPU IDs\n",
+    "opencl_gpu = device.create_opencl_device()  # valid if SINGA is compiled with USE_OPENCL=ON"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 2,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "3"
+      ]
+     },
+     "execution_count": 2,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "device.get_num_gpus()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 3,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "(0, 1, 2)"
+      ]
+     },
+     "execution_count": 3,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "device.get_gpu_ids()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "# Tensor\n",
+    "\n",
+    "A tensor instance represents a multi-dimensional array allocated on a device instance.\n",
+    "It provides linear algbra operations, like +, -, *, /, dot, pow ,etc\n",
+    "\n",
+    "NOTE: class memeber functions are inplace; global functions are out-of-place."
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Create tensor instances"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 4,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "(2, 3)"
+      ]
+     },
+     "execution_count": 4,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "from singa import tensor\n",
+    "import numpy as np\n",
+    "a = tensor.Tensor((2, 3))\n",
+    "a.shape"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 5,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "<singa.singa_wrap.Device; proxy of <Swig Object of type 'std::shared_ptr< singa::Device > *' at 0x7f69a02f8a50> >"
+      ]
+     },
+     "execution_count": 5,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "a.device"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 6,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [],
+   "source": [
+    "gb = tensor.Tensor((2, 3), gpu)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 7,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "<singa.singa_wrap.Device; proxy of <Swig Object of type 'std::shared_ptr< singa::Device > *' at 0x7f69a05ff330> >"
+      ]
+     },
+     "execution_count": 7,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "gb.device"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Initialize tensor values"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 8,
+   "metadata": {
+    "collapsed": true
+   },
+   "outputs": [],
+   "source": [
+    "a.set_value(1.2)\n",
+    "gb.gaussian(0, 0.1)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### To and from numpy"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 9,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "array([[ 1.20000005,  1.20000005,  1.20000005],\n",
+       "       [ 1.20000005,  1.20000005,  1.20000005]], dtype=float32)"
+      ]
+     },
+     "execution_count": 9,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "tensor.to_numpy(a)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 10,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "array([[ 0.24042693, -0.21337385, -0.0969397 ],\n",
+       "       [-0.010797  , -0.07642138, -0.09220808]], dtype=float32)"
+      ]
+     },
+     "execution_count": 10,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "tensor.to_numpy(gb)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 11,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "(2,)"
+      ]
+     },
+     "execution_count": 11,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "c = tensor.from_numpy(np.array([1,2], dtype=np.float32))\n",
+    "c.shape"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 12,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "array([ 3.,  4.], dtype=float32)"
+      ]
+     },
+     "execution_count": 12,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "c.copy_from_numpy(np.array([3,4], dtype=np.float32))\n",
+    "tensor.to_numpy(c)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Move tensor between devices"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 13,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "<singa.singa_wrap.Device; proxy of <Swig Object of type 'std::shared_ptr< singa::Device > *' at 0x7f69a05ff330> >"
+      ]
+     },
+     "execution_count": 13,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "gc = c.clone()\n",
+    "gc.to_device(gpu)\n",
+    "gc.device"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 14,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "<singa.singa_wrap.Device; proxy of <Swig Object of type 'std::shared_ptr< singa::Device > *' at 0x7f69a02f8a50> >"
+      ]
+     },
+     "execution_count": 14,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "b = gb.clone()\n",
+    "b.to_host()  # the same as b.to_device(default_dev)\n",
+    "b.device"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "### Operations\n",
+    "\n",
+    "**NOTE: tensors should be initialized if the operation would read the tensor values**"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "#### Summary"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 15,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "0.12169448286294937"
+      ]
+     },
+     "execution_count": 15,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "gb.l1()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 16,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "0.4898979663848877"
+      ]
+     },
+     "execution_count": 16,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "a.l2()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 17,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "False"
+      ]
+     },
+     "execution_count": 17,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "e = tensor.Tensor((2, 3))\n",
+    "e.is_empty()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 18,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "6L"
+      ]
+     },
+     "execution_count": 18,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "gb.size()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 19,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "24L"
+      ]
+     },
+     "execution_count": 19,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "gb.memsize()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 20,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "False"
+      ]
+     },
+     "execution_count": 20,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "# note we can only support matrix multiplication for tranposed tensors; \n",
+    "# other operations on transposed tensor would result in errors\n",
+    "c.is_transpose()  "
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 21,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "True"
+      ]
+     },
+     "execution_count": 21,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "et=e.T()\n",
+    "et.is_transpose()"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 22,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "(3L, 2L)"
+      ]
+     },
+     "execution_count": 22,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "et.shape"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 23,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "2L"
+      ]
+     },
+     "execution_count": 23,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "et.ndim()"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "#### Member functions (in-place)\n",
+    "\n",
+    "These functions would change the content of the tensor"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 24,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "array([[ 1.44042695,  0.98662621,  1.10306036],\n",
+       "       [ 1.18920302,  1.12357867,  1.10779202]], dtype=float32)"
+      ]
+     },
+     "execution_count": 24,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "a += b\n",
+    "tensor.to_numpy(a)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 25,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "array([[ 1.20000005,  1.20000005,  1.20000005],\n",
+       "       [ 1.20000005,  1.20000005,  1.20000005]], dtype=float32)"
+      ]
+     },
+     "execution_count": 25,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "a -= b\n",
+    "tensor.to_numpy(a)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 26,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "array([[ 2.4000001,  2.4000001,  2.4000001],\n",
+       "       [ 2.4000001,  2.4000001,  2.4000001]], dtype=float32)"
+      ]
+     },
+     "execution_count": 26,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "a *= 2\n",
+    "tensor.to_numpy(a)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 27,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "array([[ 0.80000007,  0.80000007,  0.80000007],\n",
+       "       [ 0.80000007,  0.80000007,  0.80000007]], dtype=float32)"
+      ]
+     },
+     "execution_count": 27,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "a /= 3\n",
+    "tensor.to_numpy(a)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 28,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "array([ 0.62944734, -0.72904599,  0.81158388], dtype=float32)"
+      ]
+     },
+     "execution_count": 28,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "d = tensor.Tensor((3,))\n",
+    "d.uniform(-1,1)\n",
+    "tensor.to_numpy(d)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 29,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "array([[ 1.42944741,  0.07095408,  1.61158395],\n",
+       "       [ 1.42944741,  0.07095408,  1.61158395]], dtype=float32)"
+      ]
+     },
+     "execution_count": 29,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "a.add_row(d)\n",
+    "tensor.to_numpy(a)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "#### Global functions (out of place)\n",
+    "\n",
+    "These functions would not change the memory of the tensor, instead they return a new tensor\n",
+    "\n",
+    "**Unary functions**"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 30,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "array([ 1., -1.,  1.], dtype=float32)"
+      ]
+     },
+     "execution_count": 30,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "h = tensor.sign(d)\n",
+    "tensor.to_numpy(h)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 31,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "array([ 0.62944734, -0.72904599,  0.81158388], dtype=float32)"
+      ]
+     },
+     "execution_count": 31,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "tensor.to_numpy(d)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 32,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "array([ 0.62944734,  0.72904599,  0.81158388], dtype=float32)"
+      ]
+     },
+     "execution_count": 32,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "h = tensor.abs(d)\n",
+    "tensor.to_numpy(h)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 33,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "array([ 0.62944734,  0.        ,  0.81158388], dtype=float32)"
+      ]
+     },
+     "execution_count": 33,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "h = tensor.relu(d)\n",
+    "tensor.to_numpy(h)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 34,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "(3L,)"
+      ]
+     },
+     "execution_count": 34,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "g = tensor.sum(a, 0)\n",
+    "g.shape"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 35,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "(2L,)"
+      ]
+     },
+     "execution_count": 35,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "g = tensor.sum(a, 1)\n",
+    "g.shape"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 36,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "array([ 1.,  0.], dtype=float32)"
+      ]
+     },
+     "execution_count": 36,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "tensor.bernoulli(0.5, g)\n",
+    "tensor.to_numpy(g)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 37,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "array([-0.12226005, -0.05827543], dtype=float32)"
+      ]
+     },
+     "execution_count": 37,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "g.gaussian(0, 0.2)\n",
+    "tensor.gaussian(0, 0.2, g)\n",
+    "tensor.to_numpy(g)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "#### Binary functions"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 38,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "array([[ 1.66987431, -0.14241977,  1.51464427],\n",
+       "       [ 1.41865039, -0.0054673 ,  1.51937592]], dtype=float32)"
+      ]
+     },
+     "execution_count": 38,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "f = a + b\n",
+    "tensor.to_numpy(f)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 39,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "array([[ 0.,  0.,  0.],\n",
+       "       [ 0.,  0.,  0.]], dtype=float32)"
+      ]
+     },
+     "execution_count": 39,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "g = a < b\n",
+    "tensor.to_numpy(g)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 40,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "array([[ 7.66987419,  5.85758018,  7.51464415],\n",
+       "       [ 9.41865063,  7.99453259,  9.5193758 ]], dtype=float32)"
+      ]
+     },
+     "execution_count": 40,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "tensor.add_column(2, c, 1, f)   # f = 2 *c + 1* f\n",
+    "tensor.to_numpy(f)\n"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {},
+   "source": [
+    "#### BLAS\n",
+    "\n",
+    "BLAS function may change the memory of input tensor"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 41,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "array([[ 0.24042693, -0.21337385, -0.0969397 ],\n",
+       "       [-0.010797  , -0.07642138, -0.09220808]], dtype=float32)"
+      ]
+     },
+     "execution_count": 41,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "tensor.axpy(2, a, f)  # f = 2a + f\n",
+    "tensor.to_numpy(b)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 42,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "array([[ 0.17231143, -0.16945721],\n",
+       "       [ 0.17231143, -0.16945721]], dtype=float32)"
+      ]
+     },
+     "execution_count": 42,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "f = tensor.mult(a, b.T())\n",
+    "tensor.to_numpy(f)"
+   ]
+  },
+  {
+   "cell_type": "code",
+   "execution_count": 43,
+   "metadata": {
+    "collapsed": false
+   },
+   "outputs": [
+    {
+     "data": {
+      "text/plain": [
+       "array([[ 0.51693428, -0.50837165],\n",
+       "       [ 0.51693428, -0.50837165]], dtype=float32)"
+      ]
+     },
+     "execution_count": 43,
+     "metadata": {},
+     "output_type": "execute_result"
+    }
+   ],
+   "source": [
+    "tensor.mult(a, b.T(), f, 2, 1)  # f = 2a*b.T() + 1f\n",
+    "tensor.to_numpy(f)"
+   ]
+  },
+  {
+   "cell_type": "markdown",
+   "metadata": {
+    "collapsed": true
+   },
+   "source": [
+    "## Next: [SINGA model classes](./model.ipynb)"
+   ]
+  }
+ ],
+ "metadata": {
+  "anaconda-cloud": {},
+  "kernelspec": {
+   "display_name": "Python [conda env:pysinga]",
+   "language": "python",
+   "name": "conda-env-pysinga-py"
+  },
+  "language_info": {
+   "codemirror_mode": {
+    "name": "ipython",
+    "version": 2
+   },
+   "file_extension": ".py",
+   "mimetype": "text/x-python",
+   "name": "python",
+   "nbconvert_exporter": "python",
+   "pygments_lexer": "ipython2",
+   "version": "2.7.12"
+  }
+ },
+ "nbformat": 4,
+ "nbformat_minor": 2
+}


Mime
View raw message