crunch-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Rafal Wojdyla (JIRA)" <j...@apache.org>
Subject [jira] [Created] (CRUNCH-470) Add hdfs/yarn minicluster crunch pipeline
Date Thu, 11 Sep 2014 10:27:33 GMT
Rafal Wojdyla created CRUNCH-470:
------------------------------------

             Summary: Add hdfs/yarn minicluster crunch pipeline
                 Key: CRUNCH-470
                 URL: https://issues.apache.org/jira/browse/CRUNCH-470
             Project: Crunch
          Issue Type: Improvement
          Components: Core
    Affects Versions: 0.8.3
            Reporter: Rafal Wojdyla
            Assignee: Josh Wills
            Priority: Minor


Crunch currently has two pipelines:
* MemPipeline
* MRPipeline

MemPipeline is in-memory pipelines based on local in-memory mapreduce mode.
MRPipeline is distributed pipeline based on distributed MapReduce.

Using HDFS/YARN Minicluster it's possible to better emulate Hadoop cluster, and it could be
a 'final test' before running on the cluster.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message