hadoop-hdfs-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Philip Zeyliger (JIRA)" <j...@apache.org>
Subject [jira] Created: (HDFS-621) Exposing MiniDFS and MiniMR clusters as a single process command-line
Date Tue, 15 Sep 2009 21:31:57 GMT
Exposing MiniDFS and MiniMR clusters as a single process command-line

                 Key: HDFS-621
                 URL: https://issues.apache.org/jira/browse/HDFS-621
             Project: Hadoop HDFS
          Issue Type: New Feature
          Components: test, tools
            Reporter: Philip Zeyliger
            Priority: Minor

It's hard to test non-Java programs that rely on significant mapreduce functionality.  The
patch I'm proposing shortly will let you just type "bin/hadoop jar hadoop-hdfs-hdfswithmr-test.jar
minicluster" to start a cluster (internally, it's using Mini{MR,HDFS}Cluster) with a specified
number of daemons, etc.  A test that checks how some external process interacts with Hadoop
might start minicluster as a subprocess, run through its thing, and then simply kill the java

I've been using just such a system for a couple of weeks, and I like it.  It's significantly
easier than developing a lot of scripts to start a pseudo-distributed cluster, and then clean
up after it.  I figure others might find it useful as well.

I'm at a bit of a loss as to where to put it in 0.21.  hdfs-with-mr tests have all the required
libraries, so I've put it there.  I could conceivably split this into "minimr" and "minihdfs",
but it's specifically the fact that they're configured to talk to each other that I like about
having them together.  And one JVM is better than two for my test programs.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message