mxnet-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From YiZhi Liu <liuyi...@apache.org>
Subject Apache MXNet Scala nightly-build now available on Apache Nexus
Date Mon, 29 Oct 2018 17:39:27 GMT
Hi,



I am happy to announce that the availability of the experimental
nightly-build Scala package on Maven in Nexus. The nightly-builds,
currently published as 1.3.1-SNAPSHOT version, include the latest changes
from the master branch from apache/incubator-mxnet GitHub repo and refresh
every day. Furthermore, we completely overhauled the MXNet library shipped
inside and adopted the binary build logic used for Python pip, so now Scala
users can also enjoy the portable MXNet with fully loaded features.
Currently, the supported artifacts are:

- mxnet-full_2.11-linux-x86_64-cpu (Linux-CPU)

- mxnet-full_2.11-linux-x86_64-gpu (Linux-GPU, CUDA 9.0)

- mxnet-full_2.11-osx-x86_64-cpu (OSX-CPU)



FAQ:

- What's new in the nightly Scala packages compared to the previous Scala
Maven artifacts?

Most notably, better usability, more supported features, and better
platform compatibility. The new packages now support Ubuntu 14.04 and
Amazon Linux and support the same broad set of features as the existing pip
packages. Besides, users are no longer required to install all the
dependencies to use the nightly Scala packages as the included MXNet
libraries are fully loaded. For the list of supported platforms and
features, see [1].



- What's different in the build approach compared to what's in the previous
Scala Maven packages?

The previous Scala Maven packages (1.2, 1.2.1, 1.3) were built with the
same build commands that regular users would use when building from source
[2]. This naive approach limits the compatibility to only similar
environments as where the packages were built (which was Ubuntu 16.04 for
Linux, and OSX 10.13 for OSX). As a result, users cannot use it on more
CentOS or Amazon Linux and may suffer the problem of mismatching of
dependency versions due to the difference in their environment.

On the other hand, Python users have long been enjoying portable and fully
loaded MXNet packages from pip. The MXNet libraries in pip packages are
built in more controlled environments with compatibility in mind.
Dependencies for extended features are statically linked and masked so that
no additional steps are needed.

Now, in the new Scala nightly packages, the same approach as pip is used
for building the MXNet library so that the Scala users can benefit from the
same great features.



- How do I get the new nightly packages?

You can find the packages on Apache (Snapshots) repository [3] including
pom files and jars, which you can download and add to your projects.
Alternatively, you can configure your pom.xml to use the Maven repository.
Remember to add repository config:

<repositories>

  <repository>

    <id>Apache Snapshot</id>

    <url>https://repository.apache.org/content/groups/snapshots</url>

  </repository>

</repositories>



- What's the support level of the Scala nightly packages?

This package is currently experimental, so use at your own risk. While the
first nightly packages are already manually verified on several platforms,
and the automated publish pipeline runs unit tests and integration tests on
OSX and Ubuntu 14.04 before publishing, the testing is currently limited.



- How can I help?

We call on all MXNet Scala community to test out the new nightly packages.



- I found a problem in using the package. What should I do?

Please report this in the Github issue in [1].



- What's next?

There are several more things to be done:

1. We are adding more comprehensive support for GPU versions for different
CUDA versions.

2. We are adding more tests on more platforms. We are currently examining
the MXNet's Jenkins CI solution for supporting more platforms.

3. As we recognize that the build logic benefits the community, we plan to
open source the build scripts in the mxnet main repo. See progress at [5].



Qing Lan, Zach Kimberg, Sheng Zha, and I worked together to make this
happen. Thanks to Sheng Zha for sharing the pip build logic, and for
building the automated publishing pipelines. Special thanks to Qing and
Zach for sharing their expertise in Scala with Sheng and guiding and
supporting him locally for this milestone.



[1] https://github.com/apache/incubator-mxnet/issues/8671

[2]
https://github.com/apache/incubator-mxnet/blob/master/scala-package/dev/compile-mxnet-backend.sh#L59-L105

[3]
https://repository.apache.org/#nexus-search;gav~org.apache.mxnet~~1.3.1-SNAPSHOT~~

[4] https://maven.apache.org/repository-management.html

[5] https://github.com/apache/incubator-mxnet/projects/13

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message