singa-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From wan...@apache.org
Subject [4/5] incubator-singa git commit: SINGA-303 Create conda packages
Date Sat, 25 Feb 2017 05:27:45 GMT
SINGA-303 Create conda packages

Fix bugs:
_singa_wrap must be ended with .so (cannot use dylib) on osx
must link with libs from conda, there are two approaches
1. export CMAKE_PREFIX_PATH=$PREFIX
2. pass each dep lib path to cmake.
Otherwise the system lib (e.g., python) path would be linked into _singa_wrap.so.

@rpath/libpython2.7.dylib is replaced by a path RPATH list that has libpython2.7.dylib.
conda would store an entry @load_path/../../.. in RPATH for _singa_wrap.so on osx.
@load_path is the path of _singa_wrap.so.

update readme


Project: http://git-wip-us.apache.org/repos/asf/incubator-singa/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-singa/commit/ca0be884
Tree: http://git-wip-us.apache.org/repos/asf/incubator-singa/tree/ca0be884
Diff: http://git-wip-us.apache.org/repos/asf/incubator-singa/diff/ca0be884

Branch: refs/heads/master
Commit: ca0be884c5cedc83fa10aab1547e9453c006a264
Parents: 493fc65
Author: xiezl <xiezhongle@comp.nus.edu.sg>
Authored: Fri Feb 24 10:22:56 2017 +0800
Committer: wangwei <wangwei@comp.nus.edu.sg>
Committed: Fri Feb 24 23:56:25 2017 +0800

----------------------------------------------------------------------
 python/CMakeLists.txt  |  8 ++++++++
 tool/conda/build.sh    |  4 ++++
 tool/jenkins/README.md | 29 +++++++++++++++--------------
 3 files changed, 27 insertions(+), 14 deletions(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/ca0be884/python/CMakeLists.txt
----------------------------------------------------------------------
diff --git a/python/CMakeLists.txt b/python/CMakeLists.txt
index 7445a57..177326e 100644
--- a/python/CMakeLists.txt
+++ b/python/CMakeLists.txt
@@ -129,3 +129,11 @@ CONFIGURE_FILE(${SETUP_PY_IN} ${SETUP_PY})
 
 #create python/singa/proto/__init__.py
 FILE(WRITE ${CMAKE_BINARY_DIR}/python/singa/proto/__init__.py "")
+IF(APPLE)
+  ADD_CUSTOM_TARGET(
+    change_suffix ALL
+    COMMAND ${CMAKE_COMMAND} -E rename "${CMAKE_BINARY_DIR}/python/singa/_singa_wrap.dylib"
"${CMAKE_BINARY_DIR}/python/singa/_singa_wrap.so"
+    COMMENT "change .dylib to .so in mac system"
+  )
+  ADD_DEPENDENCIES(change_suffix _singa_wrap)
+ENDIF(APPLE)

http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/ca0be884/tool/conda/build.sh
----------------------------------------------------------------------
diff --git a/tool/conda/build.sh b/tool/conda/build.sh
index f3a84d9..926da71 100644
--- a/tool/conda/build.sh
+++ b/tool/conda/build.sh
@@ -15,8 +15,12 @@
 # limitations under the License.
 #
 
+# to compile swig api files which depdend on numpy.i
 export export CPLUS_INCLUDE_PATH=`python -c "import numpy; print numpy.get_include()"`:$CPLUS_INCLUDE_PATH
 
+# to let cmake use the dependent libs installed by conda, including python
+export CMAKE_PREFIX_PATH=$PREFIX
+
 mkdir build
 cd build
 USE_CUDA=ON

http://git-wip-us.apache.org/repos/asf/incubator-singa/blob/ca0be884/tool/jenkins/README.md
----------------------------------------------------------------------
diff --git a/tool/jenkins/README.md b/tool/jenkins/README.md
index 6a1cb1e..9a76ee8 100644
--- a/tool/jenkins/README.md
+++ b/tool/jenkins/README.md
@@ -1,12 +1,11 @@
 # Jenkins CI Support
 
 ## Introduction
-This documentation is to guide SINGA developers to setup Jenkins service.
+This documentation is to guide SINGA developers to setup Jenkins service to support continuous
integration on GPU systems. After each commit,
+1. SINGA should be compiled and tested automatically under different settings (e.g., OS and
hardware).
+2. Convenient binaries should be generated automatically and archived.
 
-We use Jenkins to support continuous integration.
-After each commit, we want to automatically compile and test SINGA
-under different OS and settings.
-Those built binaries need to be archived for users to download.
+Continuous integration for CPU systems is enabled via [Travis](../travis).
 
 ## Install Jenkins
 [Jenkins Official Wiki](https://wiki.jenkins-ci.org/display/JENKINS/Installing+Jenkins)
@@ -56,18 +55,21 @@ Each node should configure the following environment variable
   * create conda package
     Execute shell - command -
 
-        git push --mirror <https://username:auth_token@github.com/mirror_repo>
-        /root/miniconda/conda tool/conda/
+        git push https://username:token@github.com/nusdbsystem/incubator-singa.git -f
+        bash -ex tool/jenkins/jenkins_test.sh $lang
         export CONDA_BLD_PATH=/root/conda-bld-$BUILD_NUMBER
         mkdir $CONDA_BLD_PATH
-        /root/miniconda/anaconda -t $ANACONDA_UPLOAD_TOKEN upload -u $USER -l main $CONDA_BLD_PATH/$OS/singa-*.tar.bz2
--force
+        /root/miniconda/bin/conda-build tool/conda
+        /root/miniconda/bin/anaconda -t ANACONDA_UPLOAD_TOKEN upload -u nusdbsystem -l main
$CONDA_BLD_PATH/linux-64/singa-*.tar.bz2 --force
+
 
     It first pushes to a mirror site to invoke travis-ci for CPU package creation;
-    Then it creates the conda package for GPU and uploads the package.
+    Then it compiles and runs unit tests;
+    Finally it creates the conda package for GPU and upload it.
 
 ### Post-build Actions
   * Publish JUnit test result report - Test report XMLs - ``**/gtest.xml, **/unittest.xml``
-  * (optional) Archive the artifacts - ``build/python/dist/**.whl, build/debian/**.deb``
+  * (optional) Archive the artifacts - ``build/debian/**.deb``
   * Send build artifacts (Debian package) over SSH for wheel
     * ../debian/build.sh packages the .deb file into $BUILD_ID.tar.gz. Inside the tar file,
       the folder layout is `build_id/commit_hash/os_lang/*.deb`, where `os_lang` is the combination
of os version, device programming language (cuda/cpp/opencl) and cudnn version.
@@ -81,12 +83,11 @@ Each node should configure the following environment variable
 
     * The file links on the remote public server would be like
 
-            debian/32/84d56b7/ubuntu16.04-cpp/singa-1.0.1.deb
-            debian/32/84d56b7/ubuntu16.04-cuda8.0-cudnn5/singa-1.0.1.deb
-
+            debian/32/84d56b7/ubuntu14.04-cpp/singa-1.0.1.deb
+            debian/32/84d56b7/ubuntu14.04-cuda8.0-cudnn5/singa-1.0.1.deb
 
 ### Docker Images
-We provide in a number of singa docker [images](../docker) for Jenkins to use as slaves.
+We provide in a number of singa docker [images](./docker) for Jenkins to use as slaves.
 To run the docker images,
 
     nvidia-docker run --name <jenkins-slaveXX> -d <Image ID>


Mime
View raw message