spark-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Patrick Wendell <>
Subject Re: Scala 2.10 Merge
Date Thu, 12 Dec 2013 09:13:16 GMT
Also - the code is still there because of a recent merge that took in some
newer changes... we'll be removing it for the final merge.

On Thu, Dec 12, 2013 at 1:12 AM, Patrick Wendell <> wrote:

> Hey Raymond,
> This won't work because AFAIK akka 2.3-M1 is not binary compatible with
> akka 2.2.3 (right?). For all of the non-yarn 2.2 versions we need to still
> use the older protobuf library, so we'd need to support both.
> I'd also be concerned about having a reference to a non-released version
> of akka. Akka is the source of our hardest-to-find bugs and simultaneously
> trying to support 2.2.3 and 2.3-M1 is a bit daunting. Of course, if you are
> building off of master you can maintain a fork that uses this.
> - Patrick
> On Thu, Dec 12, 2013 at 12:42 AM, Liu, Raymond <>wrote:
>> Hi Patrick
>>         What does that means for drop YARN 2.2? seems codes are still
>> there. You mean if build upon 2.2 it will break, and won't and work right?
>> Since the home made akka build on scala 2.10 are not there. While, if for
>> this case, can we just use akka 2.3-M1 which run on protobuf 2.5 for
>> replacement?
>> Best Regards,
>> Raymond Liu
>> -----Original Message-----
>> From: Patrick Wendell []
>> Sent: Thursday, December 12, 2013 4:21 PM
>> To:
>> Subject: Scala 2.10 Merge
>> Hi Developers,
>> In the next few days we are planning to merge Scala 2.10 support into
>> Spark. For those that haven't been following this, Prashant Sharma has been
>> maintaining the scala-2.10 branch of Spark for several months. This branch
>> is current with master and has been reviewed for merging:
>> Scala 2.10 support is one of the most requested features for Spark - it
>> will be great to get this into Spark 0.9! Please note that *Scala 2.10 is
>> not binary compatible with Scala 2.9*. With that in mind, I wanted to give
>> a few heads-up/requests to developers:
>> If you are developing applications on top of Spark's master branch, those
>> will need to migrate to Scala 2.10. You may want to download and test the
>> current scala-2.10 branch in order to make sure you will be okay as Spark
>> developments move forward. Of course, you can always stick with the current
>> master commit and be fine (I'll cut a tag when we do the merge in order to
>> delineate where the version changes). Please open new threads on the dev
>> list to report and discuss any issues.
>> This merge will temporarily drop support for YARN 2.2 on the master
>> branch.
>> This is because the workaround we used was only compiled for Scala 2.9.
>> We are going to come up with a more robust solution to YARN 2.2 support
>> before releasing 0.9.
>> Going forward, we will continue to make maintenance releases on
>> branch-0.8 which will remain compatible with Scala 2.9.
>> For those interested, the primary code changes in this merge are
>> upgrading the akka version, changing the use of Scala 2.9's ClassManifest
>> construct to Scala 2.10's ClassTag, and updating the spark shell to work
>> with Scala 2.10's repl.
>> - Patrick

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message