hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Allen Wittenauer (JIRA)" <j...@apache.org>
Subject [jira] Commented: (MAPREDUCE-987) Exposing MiniDFS and MiniMR clusters as a single process command-line
Date Wed, 24 Feb 2010 22:55:28 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-987?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12838055#action_12838055

Allen Wittenauer commented on MAPREDUCE-987:

There is a patch for this in the Cloudera distribution but it doesn't appear to have been
committed upstream.  What is the status?

> Exposing MiniDFS and MiniMR clusters as a single process command-line
> ---------------------------------------------------------------------
>                 Key: MAPREDUCE-987
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-987
>             Project: Hadoop Map/Reduce
>          Issue Type: New Feature
>          Components: build, test
>            Reporter: Philip Zeyliger
>            Assignee: Philip Zeyliger
>            Priority: Minor
>         Attachments: HDFS-621-0.20-patch, HDFS-621.patch, MAPREDUCE-987.patch
> It's hard to test non-Java programs that rely on significant mapreduce functionality.
 The patch I'm proposing shortly will let you just type "bin/hadoop jar hadoop-hdfs-hdfswithmr-test.jar
minicluster" to start a cluster (internally, it's using Mini{MR,HDFS}Cluster) with a specified
number of daemons, etc.  A test that checks how some external process interacts with Hadoop
might start minicluster as a subprocess, run through its thing, and then simply kill the java
> I've been using just such a system for a couple of weeks, and I like it.  It's significantly
easier than developing a lot of scripts to start a pseudo-distributed cluster, and then clean
up after it.  I figure others might find it useful as well.
> I'm at a bit of a loss as to where to put it in 0.21.  hdfs-with-mr tests have all the
required libraries, so I've put it there.  I could conceivably split this into "minimr" and
"minihdfs", but it's specifically the fact that they're configured to talk to each other that
I like about having them together.  And one JVM is better than two for my test programs.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message