crunch-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Josh Wills <jwi...@cloudera.com>
Subject Re: spark/scala version changes for 0.10.0/0.8.3
Date Tue, 29 Apr 2014 16:50:45 GMT
Thanks guys; patch is up here:
https://issues.apache.org/jira/browse/CRUNCH-384

Micah, do you want to take a pass through this one?

J


On Mon, Apr 28, 2014 at 10:30 PM, Som Satpathy <somsatpathy@gmail.com>wrote:

> +1 for option 2.
>
> Thanks,
> Som
>
>
> On Fri, Apr 25, 2014 at 1:45 PM, Josh Wills <jwills@cloudera.com> wrote:
>
>> Hey all,
>>
>> I've been working on doing a new release candidate, and I think that
>> we're going to need to upgrade to Spark 0.9.1 in order for Crunch-on-Spark
>> to work in Spark standalone/Spark-on-YARN on top of Hadoop, as opposed to
>> the local mode that works now. There are a couple of implications to this,
>> but the big one is that Spark 0.9.1 is only developed against Scala 2.10,
>> which would mean that we would need to switch over to 2.10 for at least
>> crunch-spark, and it would seem much easier to me for us to upgrade Scrunch
>> to 2.10 as well.
>>
>> Couple of options here off the top of my head:
>> 1) Do the upgrade to Scala 2.10/Spark 0.9.1 in 0.10.0, but not 0.8.3.
>> 2) Do the upgrade in both 0.8.3 and 0.10.0.
>> 3) Only upgrade Spark to 2.10 (in either 0.8.3 or 0.10.0), but still have
>> Scrunch build against 2.9.3.
>>
>> I'm in favor of #2 myself, but am open to suggestion here, and would
>> especially like to know if anyone feels strongly about staying on 2.9.3 for
>> any reason.
>>
>> Thanks!
>> Josh
>>
>> Director of Data Science
>> Cloudera <http://www.cloudera.com>
>> Twitter: @josh_wills <http://twitter.com/josh_wills>
>>
>
>


-- 
Director of Data Science
Cloudera <http://www.cloudera.com>
Twitter: @josh_wills <http://twitter.com/josh_wills>

Mime
View raw message