hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "alan wootton (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-123) mini map/reduce cluster for junit tests
Date Tue, 06 Jun 2006 20:57:31 GMT
    [ http://issues.apache.org/jira/browse/HADOOP-123?page=comments#action_12415047 ] 

alan wootton commented on HADOOP-123:

I put my changes into the latest trunk, and built a patch. It doesn't work yet, but the outline
is there.


There is an added conf to override the hostname for nodes (DataNode and TaskTracker)

There is a DNS hack so that java resolves all names in the form localhostN to (MyNameService)
This requires the system property 'sun.net.spi.nameservice.provider.1' to be set in the command
or in the ant script.

dfs sim = RunDfsSim.java
mapred sim = RunMapRedSim.java

The dfs sim works pretty good. The MapRed sim is broken. It's always config issues. 
Generally you want for the TaskTrackers to use their local config's and avoid loading the
default configs (which are all for localmode).

I changed the build xml so that the 'test-cluster' target runs all tests that start with 'ClusterTest'.
The ClusterTestDFS.java test is working but needs to be refactored to use RunDfsSim.java

'new JobConf(conf)' used to simply drop the conf on the floor, so there is a fix to Configuration
for that.

I had to add code to close the editlog in FSNamesystem or ClusterTestDFS.java fails on the
second pass.

Oops, the multiple editlogs patch, in FSDirectory, is also in this patch, but the default
case changes nothing.

There is a small change to the test-cluster target of build.xml to support the sims.

Tell me what you think.

> mini map/reduce cluster for junit tests
> ---------------------------------------
>          Key: HADOOP-123
>          URL: http://issues.apache.org/jira/browse/HADOOP-123
>      Project: Hadoop
>         Type: New Feature

>   Components: mapred
>     Versions: 0.3.0
>     Reporter: Owen O'Malley
>     Assignee: Milind Bhandarkar
>      Fix For: 0.4.0
>  Attachments: ClusterTestWordCount.patch
> I would like a single process map-reduce pseduo-distributed cluster that can be used
for unit tests. We should add a unit test with a word count map/reduce.

This message is automatically generated by JIRA.
If you think it was sent incorrectly contact one of the administrators:
For more information on JIRA, see:

View raw message