crunch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Josh Wills (JIRA)" <>
Subject [jira] [Updated] (CRUNCH-384) Upgrade Spark to 0.9.1
Date Tue, 29 Apr 2014 16:50:15 GMT


Josh Wills updated CRUNCH-384:

    Attachment: CRUNCH-384.patch

This patch upgrades Crunch-on-Spark to use Spark 0.9.1 and fixes a number of limitations in
the old version: this version works w/Spark in standalone mode on top of HDFS and also makes
using Counters transparently map to using Spark's accumulators (even storing them in the same
place in the PipelineResult object.)

[~mkwhitacre] want to take a look at this one?

> Upgrade Spark to 0.9.1
> ----------------------
>                 Key: CRUNCH-384
>                 URL:
>             Project: Crunch
>          Issue Type: Improvement
>          Components: Core
>            Reporter: Josh Wills
>            Assignee: Josh Wills
>         Attachments: CRUNCH-384.patch
> Spark 0.9.x has better compatibility with Hadoop 2.x along with a bunch of other fixes.
It also requires Scala 2.10, which we should upgrade to anyway.

This message was sent by Atlassian JIRA

View raw message