crunch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Josh Wills (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (CRUNCH-388) Fix issues running Spark/memory impls against Oryx
Date Mon, 05 May 2014 05:33:16 GMT

     [ https://issues.apache.org/jira/browse/CRUNCH-388?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Josh Wills updated CRUNCH-388:
------------------------------

    Attachment: CRUNCH-388.patch

The fixes:

1) Added getNumReduceTasks() support to the fake TaskInputOutputContext we use in the MemoryCollection.
2) Be more careful about setting the JARs on the JavaSparkContext and logging any errors we
encounter when we're running.

> Fix issues running Spark/memory impls against Oryx
> --------------------------------------------------
>
>                 Key: CRUNCH-388
>                 URL: https://issues.apache.org/jira/browse/CRUNCH-388
>             Project: Crunch
>          Issue Type: Bug
>    Affects Versions: 0.9.0
>            Reporter: Josh Wills
>             Fix For: 0.10.0
>
>         Attachments: CRUNCH-388.patch
>
>
> I found a couple of small issues with the in-memory implementation and the Spark implementation
when I was testing them out against Oryx, a machine learning project I work on that uses Crunch's
MR pipeline implementation.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message