incubator-mesos-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Matei Zaharia <ma...@eecs.berkeley.edu>
Subject Spark release 0.6.0 is out!
Date Tue, 16 Oct 2012 01:18:59 GMT
I'm happy to announce that the next major release of Spark, 0.6.0, is now available. This is
the biggest Spark release to date in terms of features (other than perhaps the first one),
as well as the biggest in terms of contributors, with over a dozen new contributors from Berkeley
and outside. Apart from the visible features, such as a standalone deploy mode and Java API,
it includes a significant rearchitecting of Spark under the hood that provides up to 2x faster
network performance and support for even lower-latency jobs.

The major focus points in this release have been accessibility (making Spark easier to deploy
and use) and performance. The full release notes are posted online at http://www.spark-project.org/release-0.6.0.html,
but here are some highlights:

* Simpler deployment: Spark now has a pure-Java standalone deploy mode that lets it run without
an external cluster manager, as well as experimental support for running on YARN (Hadoop NextGen).

* Java API: exposes all of Spark's features to Java developers in a clean manner.

* Expanded documentation: a new documentation site, http://spark-project.org/docs/0.6.0/,
contains significantly expanded docs, such as a quick start guide, tuning guide, configuration
guide, and detailed Scaladoc help.

* Engine enhancements: a new, custom communication layer and storage manager based on Java
NIO provide improved performance for network-heavy operations.

* Debugging enhancements: Spark now prints which line of your code each operation in its logs
corresponds to.

As mentioned above, this release is also the work of an unprecedentedly large set of developers.
Here are some of the people who contributed to Spark 0.6:

- Tathagata Das contributed the new communication layer, and parts of the storage layer.
- Haoyuan Li contributed the new storage manager.
- Denny Britz contributed the YARN deploy mode, key aspects of the standalone one, and several
other features.
- Andy Konwinski contributed the revamped documentation site, Maven publishing, and several
API docs.
- Josh Rosen contributed the Java API, as well as several bug fixes.
- Patrick Wendell contributed the enhanced debugging feature and helped with testing and documentation.
- Reynold Xin contributed numerous bug and performance fixes.
- Imran Rashid contributed the new Accumulable class.
- Harvey Feng contributed improvements to shuffle operations.
- Shivaram Venkataraman improved Spark's memory estimation and wrote a memory tuning guide.
- Ravi Pandya contributed Spark run scripts for Windows.
- Mosharaf Chowdhury provided several fixes to broadcast.
- Henry Milner pointed out several bugs in sampling algorithms.
- Ray Racine provided improvements to the EC2 scripts.
- Paul Ruan and Bill Zhao helped with testing.

We're very proud of this release, and hope that you enjoy it. You can grab the code at http://www.spark-project.org/release-0.6.0.html.


Matei
Mime
View raw message