hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sanjay Dahiya (JIRA)" <j...@apache.org>
Subject [jira] Updated: (HADOOP-307) Many small jobs benchmark for MapReduce
Date Wed, 12 Jul 2006 12:38:30 GMT
     [ http://issues.apache.org/jira/browse/HADOOP-307?page=all ]

Sanjay Dahiya updated HADOOP-307:

    Attachment: patch.txt

The only reason to keep it separate is we dont want these jar files already in classpath on
all nodes. Part of the benchmark's goal is to estimate the overhead in transfering the jar
file through HDFS. Also there is bin dir in this for scripts to run the benchmark. If this
doesnt conflict with existing examples the we can put it there as well. 

Updating the patch, it now generates excel friendly CSV output to plot graphs etc.  

> Many small jobs benchmark for MapReduce
> ---------------------------------------
>          Key: HADOOP-307
>          URL: http://issues.apache.org/jira/browse/HADOOP-307
>      Project: Hadoop
>         Type: Task

>   Components: mapred
>     Reporter: Sanjay Dahiya
>     Priority: Minor
>  Attachments: patch.txt
> A benchmark that runs many small MapReduce tasks in sequence. A single map reduce implementation
is used, it is invoked multiple times with input as the output from previous run. The input
to first Map is a TextInputFormat ( a text file with few hundred KBs). Input records are passed
to output without much processing. The idea is to benchmark the time taken by initialization
of Mapper and Reducer. An initial prototyping on a single machine with 20 MR tasks in sequence
took ~47 seconds per task. Looking for suggestions on what else can be included in the benchmark.

This message is automatically generated by JIRA.
If you think it was sent incorrectly contact one of the administrators:
For more information on JIRA, see:

View raw message