hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Milind Bhandarkar (JIRA)" <j...@apache.org>
Subject [jira] Updated: (HADOOP-123) mini map/reduce cluster for junit tests
Date Fri, 09 Jun 2006 19:32:31 GMT
     [ http://issues.apache.org/jira/browse/HADOOP-123?page=all ]

Milind Bhandarkar updated HADOOP-123:

    Attachment: minimr.patch

I have added a patch that now includes a mini-map-reduce cluster in unit tests. There are
three unit tests added that use the mini-map-reduce cluster. Main changes (other than test
additions) are: Now jobtracker, and tasktrackers can be cleanly shut down (earlier ctrl-c
was the only way). Class path changes in test.classpath (job-tracker and task-tracker need
webapps, and a temporary overriding mechanism to overriding src/test/hadoop-site.xml). Multiple
task-trackers can be spawned on the same node with their respective mapred.local.dir, therefore
unless explicitly overridden in hadoop-site.xml, the task uses task-tracker's mapred.local.dir.

The three new unit tests take 15 seconds, 115 seconds, and 119 seconds respectively, on my
linux box. They test mini MR cluster's bringup-shutdown, a map-reduce job with local file
system, and a map-reduce job with mini-DFS cluster.

By the way, Hadoop now satisfies an important requirement of being a parallel programming
infrastructure. I have included a map-reduce job that estimates the value of PI using monte-carlo
method :-)

> mini map/reduce cluster for junit tests
> ---------------------------------------
>          Key: HADOOP-123
>          URL: http://issues.apache.org/jira/browse/HADOOP-123
>      Project: Hadoop
>         Type: New Feature

>   Components: mapred
>     Versions: 0.3.0
>     Reporter: Owen O'Malley
>     Assignee: Milind Bhandarkar
>      Fix For: 0.4.0
>  Attachments: ClusterTestWordCount.patch, minimr.patch
> I would like a single process map-reduce pseduo-distributed cluster that can be used
for unit tests. We should add a unit test with a word count map/reduce.

This message is automatically generated by JIRA.
If you think it was sent incorrectly contact one of the administrators:
For more information on JIRA, see:

View raw message