crunch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Micah Whitacre (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CRUNCH-474) Reduce dependencies on MapReduce library in standard MemPipeline
Date Mon, 06 Oct 2014 21:18:34 GMT

    [ https://issues.apache.org/jira/browse/CRUNCH-474?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14160969#comment-14160969
] 

Micah Whitacre commented on CRUNCH-474:
---------------------------------------

{quote}
Currently MemPipeline relies on setting up tasks to match Hadoop libraries without using any
of their functionality, beyond the counters. Crunch may be useful in areas where data is processed
without Hadoop.
{quote}

[~Downchuck], what runtime are you specifically looking at targeting?  Typically the MemPipeline
is used for testing or prototyping but I was curious if you had some specific production use
cases you were thinking about using it in.

> Reduce dependencies on MapReduce library in standard MemPipeline
> ----------------------------------------------------------------
>
>                 Key: CRUNCH-474
>                 URL: https://issues.apache.org/jira/browse/CRUNCH-474
>             Project: Crunch
>          Issue Type: Bug
>            Reporter: Charles Pritchard
>
> There are currently dependencies on the MapReduce library that could be removed or otherwise
re-wired in the MemPipeline method.
> Currently MemPipeline relies on setting up tasks to match Hadoop libraries without using
any of their functionality, beyond the counters. Crunch may be useful in areas where data
is processed without Hadoop.
> As an aside, the Avro writables have completely unused references to Hadoop in their
import statements.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message