Return-Path: Delivered-To: apmail-lucene-hadoop-dev-archive@locus.apache.org Received: (qmail 44604 invoked from network); 27 Jun 2006 03:22:07 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (209.237.227.199) by minotaur.apache.org with SMTP; 27 Jun 2006 03:22:07 -0000 Received: (qmail 82785 invoked by uid 500); 27 Jun 2006 03:22:07 -0000 Delivered-To: apmail-lucene-hadoop-dev-archive@lucene.apache.org Received: (qmail 82636 invoked by uid 500); 27 Jun 2006 03:22:07 -0000 Mailing-List: contact hadoop-dev-help@lucene.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: hadoop-dev@lucene.apache.org Delivered-To: mailing list hadoop-dev@lucene.apache.org Received: (qmail 82626 invoked by uid 99); 27 Jun 2006 03:22:06 -0000 Received: from asf.osuosl.org (HELO asf.osuosl.org) (140.211.166.49) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 26 Jun 2006 20:22:06 -0700 X-ASF-Spam-Status: No, hits=0.0 required=10.0 tests= X-Spam-Check-By: apache.org Received: from [209.237.227.198] (HELO brutus.apache.org) (209.237.227.198) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 26 Jun 2006 20:22:06 -0700 Received: from brutus (localhost [127.0.0.1]) by brutus.apache.org (Postfix) with ESMTP id C9408410123 for ; Tue, 27 Jun 2006 03:20:32 +0000 (GMT) Message-ID: <11619877.1151378432821.JavaMail.jira@brutus> Date: Tue, 27 Jun 2006 03:20:32 +0000 (GMT+00:00) From: "eric baldeschwieler (JIRA)" To: hadoop-dev@lucene.apache.org Subject: [jira] Commented: (HADOOP-307) Many small jobs benchmark for MapReduce In-Reply-To: <13009062.1150636529837.JavaMail.jira@brutus> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-Virus-Checked: Checked by ClamAV on apache.org X-Spam-Rating: minotaur.apache.org 1.6.2 0/1000/N [ http://issues.apache.org/jira/browse/HADOOP-307?page=comments#action_12417951 ] eric baldeschwieler commented on HADOOP-307: -------------------------------------------- What configuration options does it have? Can you set an arbitrary number of map and reduce tasks? Do you validate that the right bytes get sent through the system? Set the number of steps? > Many small jobs benchmark for MapReduce > --------------------------------------- > > Key: HADOOP-307 > URL: http://issues.apache.org/jira/browse/HADOOP-307 > Project: Hadoop > Type: Task > Components: mapred > Reporter: Sanjay Dahiya > Priority: Minor > > A benchmark that runs many small MapReduce tasks in sequence. A single map reduce implementation is used, it is invoked multiple times with input as the output from previous run. The input to first Map is a TextInputFormat ( a text file with few hundred KBs). Input records are passed to output without much processing. The idea is to benchmark the time taken by initialization of Mapper and Reducer. An initial prototyping on a single machine with 20 MR tasks in sequence took ~47 seconds per task. Looking for suggestions on what else can be included in the benchmark. -- This message is automatically generated by JIRA. - If you think it was sent incorrectly contact one of the administrators: http://issues.apache.org/jira/secure/Administrators.jspa - For more information on JIRA, see: http://www.atlassian.com/software/jira