spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Patrick Wendell <>
Subject Re: Scala 2.10 Merge
Date Thu, 12 Dec 2013 09:12:21 GMT
Hey Raymond,

This won't work because AFAIK akka 2.3-M1 is not binary compatible with
akka 2.2.3 (right?). For all of the non-yarn 2.2 versions we need to still
use the older protobuf library, so we'd need to support both.

I'd also be concerned about having a reference to a non-released version of
akka. Akka is the source of our hardest-to-find bugs and simultaneously
trying to support 2.2.3 and 2.3-M1 is a bit daunting. Of course, if you are
building off of master you can maintain a fork that uses this.

- Patrick

On Thu, Dec 12, 2013 at 12:42 AM, Liu, Raymond <>wrote:

> Hi Patrick
>         What does that means for drop YARN 2.2? seems codes are still
> there. You mean if build upon 2.2 it will break, and won't and work right?
> Since the home made akka build on scala 2.10 are not there. While, if for
> this case, can we just use akka 2.3-M1 which run on protobuf 2.5 for
> replacement?
> Best Regards,
> Raymond Liu
> -----Original Message-----
> From: Patrick Wendell []
> Sent: Thursday, December 12, 2013 4:21 PM
> To:
> Subject: Scala 2.10 Merge
> Hi Developers,
> In the next few days we are planning to merge Scala 2.10 support into
> Spark. For those that haven't been following this, Prashant Sharma has been
> maintaining the scala-2.10 branch of Spark for several months. This branch
> is current with master and has been reviewed for merging:
> Scala 2.10 support is one of the most requested features for Spark - it
> will be great to get this into Spark 0.9! Please note that *Scala 2.10 is
> not binary compatible with Scala 2.9*. With that in mind, I wanted to give
> a few heads-up/requests to developers:
> If you are developing applications on top of Spark's master branch, those
> will need to migrate to Scala 2.10. You may want to download and test the
> current scala-2.10 branch in order to make sure you will be okay as Spark
> developments move forward. Of course, you can always stick with the current
> master commit and be fine (I'll cut a tag when we do the merge in order to
> delineate where the version changes). Please open new threads on the dev
> list to report and discuss any issues.
> This merge will temporarily drop support for YARN 2.2 on the master branch.
> This is because the workaround we used was only compiled for Scala 2.9. We
> are going to come up with a more robust solution to YARN 2.2 support before
> releasing 0.9.
> Going forward, we will continue to make maintenance releases on branch-0.8
> which will remain compatible with Scala 2.9.
> For those interested, the primary code changes in this merge are upgrading
> the akka version, changing the use of Scala 2.9's ClassManifest construct
> to Scala 2.10's ClassTag, and updating the spark shell to work with Scala
> 2.10's repl.
> - Patrick

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message