singa-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From wan...@apache.org
Subject [05/13] incubator-singa git commit: SINGA-72 Minor updates to be consisten with documentation
Date Sun, 27 Sep 2015 14:34:28 GMT
SINGA-72 Minor updates to be consisten with documentation

Update install.sh to make OpenBLAS with ONLY_CBLAS=1, which removes the
dependency on gfortran. Thanks to ZhangHao for reporting this solution.

Corret the configuration for rnnlm example (replacing test with
validation).

Add the license of cpplint.py in LICENSE file.

Add the solution of openblas installation error into README.md.


Project: http://git-wip-us.apache.org/repos/asf/incubator-singa/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-singa/commit/ab984da8
Tree: http://git-wip-us.apache.org/repos/asf/incubator-singa/tree/ab984da8
Diff: http://git-wip-us.apache.org/repos/asf/incubator-singa/diff/ab984da8

Branch: refs/heads/master
Commit: ab984da88cb6809ced489e0a2377e50c1856827d
Parents: 243f210
Author: Wei Wang <wangwei@comp.nus.edu.sg>
Authored: Sat Sep 26 23:16:15 2015 +0800
Committer: Wei Wang <wangwei@comp.nus.edu.sg>
Committed: Sat Sep 26 23:27:46 2015 +0800

----------------------------------------------------------------------
 LICENSE                 |  13 ++--
 README.md               |  43 +++++++-----
 RELEASE_NOTES           |   3 +-
 examples/rnnlm/job.conf |   8 +--
 src/test/test_common.cc |  21 ++++++
 thirdparty/install.sh   | 156 +++++++++++++++++++++----------------------
 6 files changed, 139 insertions(+), 105 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/ab984da8/LICENSE
----------------------------------------------------------------------
diff --git a/LICENSE b/LICENSE
index 64f06dd..7e79eef 100644
--- a/LICENSE
+++ b/LICENSE
@@ -206,8 +206,8 @@ subcomponents is subject to the terms and conditions of the following
 licenses.
 
 ============================================================================
-SINGA bundles the following under BSD 2-clause license: include/utils/blob.h, src/utils/blob.cc,
-include/utils/common.h, src/utils/common.cc
+SINGA bundles the following under BSD 2-clause license: include/utils/blob.h,
+src/utils/blob.cc, include/utils/common.h, src/utils/common.cc
 
 Copyright (c) 2014, 2015, The Regents of the University of California (Regents)
 https://github.com/BVLC/caffe/blob/master/LICENSE
@@ -216,7 +216,7 @@ https://github.com/BVLC/caffe/blob/master/LICENSE
 SINGA bundles the following under BSD 2-clause license: include/utils/tinydir.h
 
 Copyright (c) 2013, Cong Xu
-https://github.com/cxong/tinydir/blob/master/COPYING 
+https://github.com/cxong/tinydir/blob/master/COPYING
 
 =====================================================================
 SINGA bundles the following under Apache v2.0 license: include/mshadow/*
@@ -225,10 +225,13 @@ Copyright (c) 2014 by Contributors
 https://github.com/dmlc/mshadow/blob/master/LICENSE
 
 =====================================================================
-SINGA bundles the following under New BSD license: include/mshadow/*
+SINGA bundles the following under New BSD license: include/gtest/*
 
 Copyright 2008, Google Inc.
 https://code.google.com/p/googletest/source/browse/trunk/LICENSE
-=====================================================================
 
+=====================================================================
+SINGA bundles the following under New BSD license: tool/cppling.py
 
+Copyright (c) 2009 Google Inc. All rights reserved.
+https://github.com/google/styleguide/tree/gh-pages/cpplint

http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/ab984da8/README.md
----------------------------------------------------------------------
diff --git a/README.md b/README.md
index af903a9..bf7874a 100644
--- a/README.md
+++ b/README.md
@@ -1,4 +1,3 @@
-
 #Apache SINGA
 
 Distributed deep learning system
@@ -22,13 +21,20 @@ The current code depends on the following external libraries:
   * czmq (Mozilla Public License Version 2.0)
   * zookeeper (Apache 2.0)
 
-To install openblas, you need a fortran compiler.
+You can install all dependencies into $PREFIX folder by
+
+    ./thirdparty/install.sh all $PREFIX
+
+If $PREFIX is not a system path (e.g., /usr/local/), you have to export some
+environment variables,
+
+    export LD_LIBRARY_PATH=$PREFIX/lib:$LD_LIBRARY_PATH
+    export CPLUS_INCLUDE_PATH=$PREFIX/include
 
 ##Documentation
 
 Full documentation is available online at [Official Documentation](https://singa.incubator.apache.org/docs/overview.html#).
 
-
 ##Building SINGA
 
     $ ./autogen.sh (optional)
@@ -38,8 +44,7 @@ Full documentation is available online at [Official Documentation](https://singa
 
 ##Running Examples
 
-Let us train the [CNN
-model](http://papers.nips.cc/paper/4824-imagenet-classification-with-deep-convolutional-neural-networks)
over the
+Let us train the [CNN model](http://papers.nips.cc/paper/4824-imagenet-classification-with-deep-convolutional-neural-networks)
over the
 [CIFAR-10](http://www.cs.toronto.edu/~kriz/cifar.html) dataset without parallelism as an
example. The hyper-parameters
 are set following [cuda-convnet](https://code.google.com/p/cuda-convnet/). More details about
this example are available
 at [CNN example](http://singa.incubator.apache.org/docs/cnn).
@@ -54,7 +59,7 @@ First, download the dataset and create data shards:
 If it reports errors due to libopenblas.so missing, then include the
 lib folder of OpenBLAS in LD_LIBRARY_PATH
 
-    $ export LD_LIBRARY_PATH= OPENBLAS_FOLDER/lib:$LD_LIBRARY_PATH
+    $ export LD_LIBRARY_PATH=$OPENBLAS_FOLDER/lib:$LD_LIBRARY_PATH
     # delete the newly created folders
     $ rm -rf cifar10_t*
     $ make create
@@ -69,7 +74,7 @@ Now we just need to wait until it is done!
 
 ##LICENSE
 
-Apache Singa is licensed under the [Apache License, Version 2.0](http://www.apache.org/licenses/LICENSE-2.0).
+Apache SINGA is licensed under the [Apache License, Version 2.0](http://www.apache.org/licenses/LICENSE-2.0).
 
 For additional information, see the LICENSE and NOTICE files.
 
@@ -80,20 +85,20 @@ run `install.sh OpenBLAS`.
 
   A1: `OpenBLAS` library is installed in `/opt` folder by default or
   other folders if you use `sudo apt-get install`.
-  You need to include the OpenBLAS library folder in the LDFLAGS.
+  You need to include the OpenBLAS library folder in the LDFLAGS, e.g.,
 
-      $ export LDFLAGS=-L/opt/OpenBLAS/lib
+      $ export LDFLAGS=-L/opt/OpenBLAS
 
   Alternatively, you can include the path in LIBRARY_PATH.
 
 
-* Q2: I get error `cblas.h not such file or directory exists`.
+* Q2: I get error `cblas.h no such file or directory exists`.
 
-  Q2: You need to include the folder of the cblas.h (e.g., /opt/OpenBLAS/include)
-  into CPLUS_INCLUDE_PATH
+  Q2: You need to include the folder of the cblas.h into CPLUS_INCLUDE_PATH,
+  e.g.,
 
       $ export CPLUS_INCLUDE_PATH=/opt/OpenBLAS/include:$CPLUS_INCLUDE_PATH
-      # reconfigure and make
+      # reconfigure and make SINGA
       $ ./configure
       $ make
 
@@ -147,12 +152,16 @@ google.protobuf.internal when I try to import .py files.
 
 * Q8: When I build OpenBLAS from source, I am told that I need a fortran compiler.
 
-  A8: Since OpenBLAS use fortran compiler to build the library, you need a compiler with
fortran support. As an alternative, you can build OpenBLAS from system tools. For example,
if you have APT, just run:
-	 
+  A8: You can compile OpenBLAS by
+
+      $ make ONLY_CBLAS=1
+
+  or install it using
+
 	  $ sudo apt-get install openblas
 
-  or you can also run the following command if you have yum:
+  or
 
 	  $ sudo yum install openblas-devel
 
-  It is worth noting that you need root access to run the aforementioned commands.
+  It is worth noting that you need root access to run the last two commands.

http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/ab984da8/RELEASE_NOTES
----------------------------------------------------------------------
diff --git a/RELEASE_NOTES b/RELEASE_NOTES
index 8c5eae0..b2dd00f 100644
--- a/RELEASE_NOTES
+++ b/RELEASE_NOTES
@@ -1,4 +1,4 @@
-Release Notes - SINGA - Version singa-incubating-0.1-rc1
+Release Notes - SINGA - Version singa-incubating-0.1.0-rc1
 -----------------------------------------
 
 SINGA is a general distributed deep learning platform for training big deep learning models
over large datasets. It is
@@ -19,6 +19,7 @@ This release includes following features:
     * [SINGA-39] - Avoid ssh in scripts for single node environment
     * [SINGA-43] - Remove Job-related output from workspace
     * [SINGA-56] - No automatic launching of zookeeper service
+    * [SINGA-73] - Refine the selection of available hosts from host list
 
   * Installation with GNU Auto tool
     * [SINGA-4] - Refine thirdparty-dependency installation

http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/ab984da8/examples/rnnlm/job.conf
----------------------------------------------------------------------
diff --git a/examples/rnnlm/job.conf b/examples/rnnlm/job.conf
index 5779ff6..db96e84 100644
--- a/examples/rnnlm/job.conf
+++ b/examples/rnnlm/job.conf
@@ -2,8 +2,8 @@ name: "rnnlm"
 #To scan the training file (81350) 10 times
 train_steps:81350
 #To scan the validation file (6828) once
-test_steps:683
-test_freq:8135
+valid_steps:683
+valid_freq:8135
 #disp_freq is specific to training
 disp_freq:8135
 train_one_batch {
@@ -36,14 +36,14 @@ layer {
     path: "examples/rnnlm/train_shard"
     max_window: 10
   }
-  exclude: kTest
+  exclude: kValidation
 }
 
 layer {
   name: "data"
   user_type: "kData"
   [data_conf] {
-    path: "examples/rnnlm/test_shard"
+    path: "examples/rnnlm/valid_shard"
     max_window: 10
   }
   exclude: kTrain

http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/ab984da8/src/test/test_common.cc
----------------------------------------------------------------------
diff --git a/src/test/test_common.cc b/src/test/test_common.cc
index e30c9cb..03a02fb 100644
--- a/src/test/test_common.cc
+++ b/src/test/test_common.cc
@@ -1,3 +1,24 @@
+/************************************************************
+*
+* Licensed to the Apache Software Foundation (ASF) under one
+* or more contributor license agreements.  See the NOTICE file
+* distributed with this work for additional information
+* regarding copyright ownership.  The ASF licenses this file
+* to you under the Apache License, Version 2.0 (the
+* "License"); you may not use this file except in compliance
+* with the License.  You may obtain a copy of the License at
+*
+*   http://www.apache.org/licenses/LICENSE-2.0
+*
+* Unless required by applicable law or agreed to in writing,
+* software distributed under the License is distributed on an
+* "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+* KIND, either express or implied.  See the License for the
+* specific language governing permissions and limitations
+* under the License.
+*
+*************************************************************/
+
 #include <string>
 #include <unordered_map>
 #include <vector>

http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/ab984da8/thirdparty/install.sh
----------------------------------------------------------------------
diff --git a/thirdparty/install.sh b/thirdparty/install.sh
index b9a3095..86211d6 100755
--- a/thirdparty/install.sh
+++ b/thirdparty/install.sh
@@ -64,7 +64,7 @@ function install_czmq()
 	fi
 	rm -rf czmq-3.0.0;
 	tar zxvf czmq-3.0.0-rc1.tar.gz && cd czmq-3.0.0;
-	
+
 	if [ $# == 2 ]
 	then
 		if [ $1 == "null" ]
@@ -109,7 +109,7 @@ function install_glog()
 	then
 		wget http://www.comp.nus.edu.sg/~dbsystem/singa/assets/file/thirdparty/glog-0.3.3.tar.gz;
 	fi
-	
+
 	rm -rf glog-0.3.3;
 	tar zxvf glog-0.3.3.tar.gz && cd glog-0.3.3;
 
@@ -178,7 +178,7 @@ function install_openblas()
 	rm -rf OpenBLAS-develop;
 	unzip OpenBLAS.zip && cd OpenBLAS-develop;
 
-	make;
+	make ONLY_CBLAS=1;
 	if [ $? -ne 0 ]
 	then
 		cd ..;
@@ -195,7 +195,7 @@ function install_openblas()
 			fi
 		elif [ $# == 0 ]
 		then
-			echo "install OpenBLAS in default path" 
+			echo "install OpenBLAS in default path"
 			sudo make install;
 			if [ $? -ne 0 ]
 			then
@@ -219,7 +219,7 @@ function install_opencv()
 
 	rm -rf opencv-2.4.10;
 	unzip opencv-2.4.10.zip && cd opencv-2.4.10;
-	
+
 	if [ $# == 1 ]
 		then
 			echo "install opencv in $1";
@@ -239,7 +239,7 @@ function install_opencv()
 		cd ..;
 		return -1;
 	fi
-	cd ..;	
+	cd ..;
 	return 0;
 }
 
@@ -307,7 +307,7 @@ function install_zeromq()
 		else
 			echo "wrong commands";
 	fi
-	
+
 	if [ $? -ne 0 ]
 	then
 		cd ..;
@@ -338,7 +338,7 @@ function install_zookeeper()
 			echo "install zookeeper in default path";
 			./configure;
 			make && sudo make install;
-		else 
+		else
 			echo "wrong commands";
 	fi
 
@@ -365,20 +365,20 @@ do
 #		echo "install cmake";
 #		if [[ $2 == */* ]];then
 #			install_cmake $2;
-#		    if [ $? -ne 0 ] 
+#		    if [ $? -ne 0 ]
 #		    then
 #		        echo "ERROR during cmake installation" ;
 #		        exit;
-#		    fi  
+#		    fi
 #			shift
 #			shift
 #		else
 #			install_cmake;
-#		    if [ $? -ne 0 ] 
+#		    if [ $? -ne 0 ]
 #		    then
 #		        echo "ERROR during cmake installation" ;
 #		        exit;
-#		    fi  
+#		    fi
 #			shift
 #		fi
 #		;;
@@ -392,7 +392,7 @@ do
 			else
 				install_czmq null $3;
 			fi
-			if [ $? -ne 0 ] 
+			if [ $? -ne 0 ]
 			then
 				echo "ERROR during czmq installation" ;
 				exit;
@@ -404,7 +404,7 @@ do
 		elif [ $3 == "-f" ]
 		then
 			install_czmq $2 $4;
-			if [ $? -ne 0 ] 
+			if [ $? -ne 0 ]
 			then
 				echo "ERROR during czmq installation" ;
 				exit;
@@ -416,7 +416,7 @@ do
 		elif [[ $2 == */* ]]
 		then
 			install_czmq $2;
-			if [ $? -ne 0 ] 
+			if [ $? -ne 0 ]
 			then
 				echo "ERROR during czmq installation" ;
 				exit;
@@ -425,11 +425,11 @@ do
 			shift
 		else
 			install_czmq null;
-			if [ $? -ne 0 ] 
+			if [ $? -ne 0 ]
 			then
 			    echo "ERROR during czmq installation" ;
 			    exit;
-			fi  
+			fi
 			shift
 		fi
 		;;
@@ -437,20 +437,20 @@ do
 		echo "install glog";
 		if [[ $2 == */* ]];then
 			install_glog $2;
-		    if [ $? -ne 0 ] 
+		    if [ $? -ne 0 ]
 		    then
 		        echo "ERROR during glog installation" ;
 		        exit;
-		    fi  
+		    fi
 			shift
 			shift
 		else
 			install_glog;
-		    if [ $? -ne 0 ] 
+		    if [ $? -ne 0 ]
 		    then
 		        echo "ERROR during glog installation" ;
 		        exit;
-		    fi  
+		    fi
 			shift
 		fi
 		;;
@@ -458,20 +458,20 @@ do
 		echo "install lmdb";
 		if [[ $2 == */* ]];then
 			install_lmdb $2;
-		    if [ $? -ne 0 ] 
+		    if [ $? -ne 0 ]
 		    then
 		        echo "ERROR during lmdb installation" ;
 		        exit;
-		    fi  
+		    fi
 			shift
 			shift
 		else
 			install_lmdb;
-		    if [ $? -ne 0 ] 
+		    if [ $? -ne 0 ]
 		    then
 		        echo "ERROR during lmdb installation" ;
 		        exit;
-		    fi  
+		    fi
 			shift
 		fi
 		;;
@@ -479,20 +479,20 @@ do
 		echo "install OpenBLAS";
 		if [[ $2 == */* ]];then
 			install_openblas $2;
-		    if [ $? -ne 0 ] 
+		    if [ $? -ne 0 ]
 		    then
 		        echo "ERROR during openblas installation" ;
 		        exit;
-		    fi  
+		    fi
 			shift
 			shift
 		else
 			install_openblas;
-		    if [ $? -ne 0 ] 
+		    if [ $? -ne 0 ]
 		    then
 		        echo "ERROR during openblas installation" ;
 		        exit;
-		    fi  
+		    fi
 			shift
 		fi
 		;;
@@ -500,20 +500,20 @@ do
 #		echo "install opencv";
 #		if [[ $2 == */* ]];then
 #			install_opencv $2;
-#		    if [ $? -ne 0 ] 
+#		    if [ $? -ne 0 ]
 #		    then
 #		        echo "ERROR during opencv installation" ;
 #		        exit;
-#		    fi  
+#		    fi
 #			shift
 #			shift
 #		else
 #			install_opencv;
-#		    if [ $? -ne 0 ] 
+#		    if [ $? -ne 0 ]
 #		    then
 #		        echo "ERROR during opencv installation" ;
 #		        exit;
-#		    fi  
+#		    fi
 #			shift
 #		fi
 #		;;
@@ -521,20 +521,20 @@ do
 		echo "install protobuf";
 		if [[ $2 == */* ]];then
 			install_protobuf $2;
-		    if [ $? -ne 0 ] 
+		    if [ $? -ne 0 ]
 		    then
 		        echo "ERROR during protobuf installation" ;
 		        exit;
-		    fi  
+		    fi
 			shift
 			shift
 		else
 			install_protobuf;
-		    if [ $? -ne 0 ] 
+		    if [ $? -ne 0 ]
 		    then
 		        echo "ERROR during protobuf installation" ;
 		        exit;
-		    fi  
+		    fi
 			shift
 		fi
 		;;
@@ -542,20 +542,20 @@ do
 		echo "install zeromq";
 		if [[ $2 == */* ]];then
 			install_zeromq $2;
-		    if [ $? -ne 0 ] 
+		    if [ $? -ne 0 ]
 		    then
 		        echo "ERROR during zeromq installation" ;
 		        exit;
-		    fi  
+		    fi
 			shift
 			shift
 		else
 			install_zeromq;
-		    if [ $? -ne 0 ] 
+		    if [ $? -ne 0 ]
 		    then
 		        echo "ERROR during zeromq installation" ;
 		        exit;
-		    fi  
+		    fi
 			shift
 		fi
 		;;
@@ -563,20 +563,20 @@ do
 		echo "install zookeeper";
 		if [[ $2 == */* ]];then
 			install_zookeeper $2;
-		    if [ $? -ne 0 ] 
+		    if [ $? -ne 0 ]
 		    then
 		        echo "ERROR during zookeeper installation" ;
 		        exit;
-		    fi  
+		    fi
 			shift
 			shift
 		else
 			install_zookeeper;
-		    if [ $? -ne 0 ] 
+		    if [ $? -ne 0 ]
 		    then
 		        echo "ERROR during zookeeper installation" ;
 		        exit;
-		    fi  
+		    fi
 			shift
 		fi
 		;;
@@ -584,53 +584,53 @@ do
 		echo "install all dependencies";
 		if [[ $2 == */* ]];then
 #			install_cmake $2;
-#		    if [ $? -ne 0 ] 
+#		    if [ $? -ne 0 ]
 #		    then
 #		        echo "ERROR during cmake installation" ;
 #		        exit;
-#		    fi  
+#		    fi
 			install_zeromq $2;
-		    if [ $? -ne 0 ] 
+		    if [ $? -ne 0 ]
 		    then
 		        echo "ERROR during zeromq installation" ;
 		        exit;
-		    fi  
+		    fi
 			install_czmq $2 $2;
-		    if [ $? -ne 0 ] 
+		    if [ $? -ne 0 ]
 		    then
 		        echo "ERROR during czmq installation" ;
 		        exit;
-		    fi  
+		    fi
 			install_glog $2;
-		    if [ $? -ne 0 ] 
+		    if [ $? -ne 0 ]
 		    then
 		        echo "ERROR during glog installation" ;
 		        exit;
-		    fi  
+		    fi
 			install_lmdb $2;
-		    if [ $? -ne 0 ] 
+		    if [ $? -ne 0 ]
 		    then
 		        echo "ERROR during lmdb installation" ;
 		        exit;
-		    fi  
+		    fi
 			install_openblas $2;
-		    if [ $? -ne 0 ] 
+		    if [ $? -ne 0 ]
 		    then
 		        echo "ERROR during openblas installation" ;
 		        exit;
-		    fi  
+		    fi
 #			install_opencv $2;
-#		    if [ $? -ne 0 ] 
+#		    if [ $? -ne 0 ]
 #		    then
 #		        echo "ERROR during opencv installation" ;
 #		        exit;
-#		    fi  
+#		    fi
 			install_protobuf $2;
-		    if [ $? -ne 0 ] 
+		    if [ $? -ne 0 ]
 		    then
 		        echo "ERROR during protobuf installation" ;
 		        exit;
-		    fi  
+		    fi
 			install_zookeeper $2;
 			if [ $? -ne 0 ]
 			then
@@ -641,53 +641,53 @@ do
 			shift
 		else
 #			install_cmake;
-#		    if [ $? -ne 0 ] 
+#		    if [ $? -ne 0 ]
 #		    then
 #		        echo "ERROR during cmake installation" ;
 #		        exit;
-#		    fi  
+#		    fi
 			install_zeromq;
-		    if [ $? -ne 0 ] 
+		    if [ $? -ne 0 ]
 		    then
 		        echo "ERROR during zeromq installation" ;
 		        exit;
-		    fi  
+		    fi
 			install_czmq null;
-		    if [ $? -ne 0 ] 
+		    if [ $? -ne 0 ]
 		    then
 		        echo "ERROR during czmq installation" ;
 		        exit;
-		    fi  
+		    fi
 			install_glog;
-		    if [ $? -ne 0 ] 
+		    if [ $? -ne 0 ]
 		    then
 		        echo "ERROR during glog installation" ;
 		        exit;
-		    fi  
+		    fi
 			install_lmdb;
-		    if [ $? -ne 0 ] 
+		    if [ $? -ne 0 ]
 		    then
 		        echo "ERROR during lmdb installation" ;
 		        exit;
-		    fi  
+		    fi
 			install_openblas;
-		    if [ $? -ne 0 ] 
+		    if [ $? -ne 0 ]
 		    then
 		        echo "ERROR during openblas installation" ;
 		        exit;
-		    fi  
+		    fi
 #			install_opencv;
-#		    if [ $? -ne 0 ] 
+#		    if [ $? -ne 0 ]
 #		    then
 #		        echo "ERROR during opencv installation" ;
 #		        exit;
-#		    fi  
+#		    fi
 			install_protobuf;
-		    if [ $? -ne 0 ] 
+		    if [ $? -ne 0 ]
 		    then
 		        echo "ERROR during protobuf installation" ;
 		        exit;
-		    fi  
+		    fi
 			install_zookeeper;
 			if [ $? -ne 0 ]
 			then
@@ -712,5 +712,5 @@ do
 		echo " To install all dependencies, you can run:	"
 		echo "	./install.sh all"
 		exit;
-	esac	
+	esac
 done


Mime
View raw message