spark-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
Subject spark git commit: SPARK-5665 [DOCS] Update netlib-java documentation
Date Mon, 09 Feb 2015 00:34:39 GMT
Repository: spark
Updated Branches:
  refs/heads/branch-1.3 9e4d58fe2 -> c515634ef

SPARK-5665 [DOCS] Update netlib-java documentation

I am the author of netlib-java and I found this documentation to be out of date. Some main

1. Breeze has not depended on jBLAS for some time
2. netlib-java provides a pure JVM implementation as the fallback (the original docs did not
appear to be aware of this, claiming that gfortran was necessary)
3. The licensing issue is not just about LGPL: optimised natives have proprietary licenses.
Building with the LGPL flag turned on really doesn't help you get past this.
4. I really think it's best to direct people to my detailed setup guide instead of trying
to compress it into one sentence. It is different for each architecture, each OS, and for
each backend.

I hope this helps to clear things up :smile:

Author: Sam Halliday <>
Author: Sam Halliday <>

Closes #4448 from fommil/patch-1 and squashes the following commits:

18cda11 [Sam Halliday] remove link to skillsmatters at request of @mengxr
a35e4a9 [Sam Halliday] reword netlib-java/breeze docs

(cherry picked from commit 56aff4bd6c7c9d18f4f962025708f20a4a82dcf0)
Signed-off-by: Xiangrui Meng <>


Branch: refs/heads/branch-1.3
Commit: c515634ef178b49cd4f8ce2c5d08a77054be3a55
Parents: 9e4d58f
Author: Sam Halliday <>
Authored: Sun Feb 8 16:34:26 2015 -0800
Committer: Xiangrui Meng <>
Committed: Sun Feb 8 16:34:34 2015 -0800

 docs/ | 41 ++++++++++++++++++++++++-----------------
 1 file changed, 24 insertions(+), 17 deletions(-)
diff --git a/docs/ b/docs/
index 7779fbc..3d32d03 100644
--- a/docs/
+++ b/docs/
@@ -56,25 +56,32 @@ See the **[ programming guide](ml-guide.html)** for more information
 # Dependencies
-MLlib uses the linear algebra package [Breeze](,
-which depends on [netlib-java](,
-and [jblas]( 
-`netlib-java` and `jblas` depend on native Fortran routines.
-You need to install the
+MLlib uses the linear algebra package
+[Breeze](, which depends on
+[netlib-java]( for optimised
+numerical processing. If natives are not available at runtime, you
+will see a warning message and a pure JVM implementation will be used
+To learn more about the benefits and background of system optimised
+natives, you may wish to watch Sam Halliday's ScalaX talk on
+[High Performance Linear Algebra in Scala](
+Due to licensing issues with runtime proprietary binaries, we do not
+include `netlib-java`'s native proxies by default. To configure
+`netlib-java` / Breeze to use system optimised binaries, include
+`com.github.fommil.netlib:all:1.1.2` (or build Spark with
+`-Pnetlib-lgpl`) as a dependency of your project and read the
+[netlib-java]( documentation for
+your platform's additional installation instructions.
+MLlib also uses [jblas]( which
+will require you to install the
 [gfortran runtime library](
 if it is not already present on your nodes.
-MLlib will throw a linking error if it cannot detect these libraries automatically.
-Due to license issues, we do not include `netlib-java`'s native libraries in MLlib's
-dependency set under default settings.
-If no native library is available at runtime, you will see a warning message.
-To use native libraries from `netlib-java`, please build Spark with `-Pnetlib-lgpl` or
-include `com.github.fommil.netlib:all:1.1.2` as a dependency of your project.
-If you want to use optimized BLAS/LAPACK libraries such as
-[OpenBLAS](, please link its shared libraries to
-`/usr/lib/` and `/usr/lib/`, respectively.
-BLAS/LAPACK libraries on worker nodes should be built without multithreading.
-To use MLlib in Python, you will need [NumPy]( version 1.4 or newer.
+To use MLlib in Python, you will need [NumPy](
+version 1.4 or newer.

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message