crunch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Josh Wills (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (CRUNCH-384) Upgrade Spark to 0.9.1
Date Tue, 29 Apr 2014 16:50:15 GMT

     [ https://issues.apache.org/jira/browse/CRUNCH-384?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Josh Wills updated CRUNCH-384:
------------------------------

    Attachment: CRUNCH-384.patch

This patch upgrades Crunch-on-Spark to use Spark 0.9.1 and fixes a number of limitations in
the old version: this version works w/Spark in standalone mode on top of HDFS and also makes
using Counters transparently map to using Spark's accumulators (even storing them in the same
place in the PipelineResult object.)

[~mkwhitacre] want to take a look at this one?

> Upgrade Spark to 0.9.1
> ----------------------
>
>                 Key: CRUNCH-384
>                 URL: https://issues.apache.org/jira/browse/CRUNCH-384
>             Project: Crunch
>          Issue Type: Improvement
>          Components: Core
>            Reporter: Josh Wills
>            Assignee: Josh Wills
>         Attachments: CRUNCH-384.patch
>
>
> Spark 0.9.x has better compatibility with Hadoop 2.x along with a bunch of other fixes.
It also requires Scala 2.10, which we should upgrade to anyway.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message