hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Uros Gruber (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-307) Many small jobs benchmark for MapReduce
Date Fri, 04 Aug 2006 15:18:14 GMT
    [ http://issues.apache.org/jira/browse/HADOOP-307?page=comments#action_12425771 ] 
            
Uros Gruber commented on HADOOP-307:
------------------------------------

I want to try this on Windows with cygwin and found some problems.

1. $JAVA_HOME in run.sh needs to be "$JAVA_HOME" because on windows java is installed in Program
Files and there is problem with spaces
2. also there is some /export/crawlspace/kryptonite/java/jdk/lib/tools.jar in classpath which
I believe is not needed.

also with this I could not get script working with error

java.lang.NoClassDefFoundError: org/apache/hadoop/benchmarks/mapred/MultiJobRunner

If I have only ./classes in my classpath there is problem with loger. I don't know If there
is any problem with classpath definition in windows.


> Many small jobs benchmark for MapReduce
> ---------------------------------------
>
>                 Key: HADOOP-307
>                 URL: http://issues.apache.org/jira/browse/HADOOP-307
>             Project: Hadoop
>          Issue Type: Task
>          Components: mapred
>            Reporter: Sanjay Dahiya
>            Priority: Minor
>             Fix For: 0.5.0
>
>         Attachments: patch.txt, patch.txt
>
>
> A benchmark that runs many small MapReduce tasks in sequence. A single map reduce implementation
is used, it is invoked multiple times with input as the output from previous run. The input
to first Map is a TextInputFormat ( a text file with few hundred KBs). Input records are passed
to output without much processing. The idea is to benchmark the time taken by initialization
of Mapper and Reducer. An initial prototyping on a single machine with 20 MR tasks in sequence
took ~47 seconds per task. Looking for suggestions on what else can be included in the benchmark.


-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message