crunch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Tom White (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CRUNCH-618) Run on Spark 2
Date Mon, 05 Dec 2016 12:34:59 GMT

    [ https://issues.apache.org/jira/browse/CRUNCH-618?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15722146#comment-15722146
] 

Tom White commented on CRUNCH-618:
----------------------------------

HBase 2 isn't available yet, so gating 2.x on both Spark 2 and HBase 2, means Crunch support
for Spark 2 won't be available for a while. I'm not sure how much of a problem that is. There's
also Hadoop 3 to think about, although I would hope that the HDFS and MR APIs will be compatible.

Also, 0.x releases are generally treated as major versions from a compatibility point of view.
So it would be OK to have 0.14.x use Spark 1 and 0.15.x use Spark 2. And then 0.16.x use HBase
2 perhaps. 

> Run on Spark 2
> --------------
>
>                 Key: CRUNCH-618
>                 URL: https://issues.apache.org/jira/browse/CRUNCH-618
>             Project: Crunch
>          Issue Type: Improvement
>          Components: Spark
>    Affects Versions: 0.14.0
>            Reporter: Tom White
>            Assignee: Tom White
>         Attachments: CRUNCH-618.patch
>
>
> Spark 2 is GA now so it would be good to be able to run Crunch on it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message