Return-Path: 
 <hadoop-dev-return-153-apmail-lucene-hadoop-dev-archive=lucene.apache.org@lucene.apache.org>
Delivered-To: apmail-lucene-hadoop-dev-archive@locus.apache.org
Received: (qmail 56023 invoked from network); 14 Feb 2006 20:49:33 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (209.237.227.199)
  by minotaur.apache.org with SMTP; 14 Feb 2006 20:49:33 -0000
Received: (qmail 87043 invoked by uid 500); 14 Feb 2006 20:49:33 -0000
Delivered-To: apmail-lucene-hadoop-dev-archive@lucene.apache.org
Received: (qmail 87019 invoked by uid 500); 14 Feb 2006 20:49:32 -0000
Mailing-List: contact hadoop-dev-help@lucene.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:hadoop-dev-help@lucene.apache.org>
List-Unsubscribe: <mailto:hadoop-dev-unsubscribe@lucene.apache.org>
List-Post: <mailto:hadoop-dev@lucene.apache.org>
List-Id: <hadoop-dev.lucene.apache.org>
Reply-To: hadoop-dev@lucene.apache.org
Delivered-To: mailing list hadoop-dev@lucene.apache.org
Received: (qmail 87002 invoked by uid 99); 14 Feb 2006 20:49:32 -0000
X-ASF-Spam-Status: No, hits=0.0 required=10.0
	tests=
X-Spam-Check-By: apache.org
Received: from [192.87.106.226] (HELO ajax.apache.org) (192.87.106.226)
    by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 14 Feb 2006 12:49:30 -0800
Received: from ajax.apache.org (ajax.apache.org [127.0.0.1])
	by ajax.apache.org (Postfix) with ESMTP id D7922DC
	for <hadoop-dev@lucene.apache.org>; Tue, 14 Feb 2006 21:49:08 +0100 (CET)
Message-ID: <104031761.1139950148861.JavaMail.jira@ajax.apache.org>
Date: Tue, 14 Feb 2006 21:49:08 +0100 (CET)
From: "Bryan Pendleton (JIRA)" <jira@apache.org>
To: hadoop-dev@lucene.apache.org
Subject: [jira] Commented: (HADOOP-38) default splitter should incorporate
 fs block size
In-Reply-To: <1089400937.1139945203207.JavaMail.jira@ajax.apache.org>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org
X-Spam-Rating: minotaur.apache.org 1.6.2 0/1000/N

    [ http://issues.apache.org/jira/browse/HADOOP-38?page=comments#action_12366381 ] 

Bryan Pendleton commented on HADOOP-38:
---------------------------------------

The idea sounds sound, but is blocksize the best unit? There's a certain overhead for each additional task added to a job - for jobs with really large input, this could cause really large task lists. Is there going to be any code for pre-replicating blocks? Maybe sequences, so there'd be a natural "first choice" node for many chunkings of larger than one block? Obviously, as datanodes come and go this might not always work ideally, but it could help in the 80% case.

> default splitter should incorporate fs block size
> -------------------------------------------------
>
>          Key: HADOOP-38
>          URL: http://issues.apache.org/jira/browse/HADOOP-38
>      Project: Hadoop
>         Type: Improvement
>   Components: mapred
>     Reporter: Doug Cutting

>
> By default, the file splitting code should operate as follows.
>   inputs are <file>*, numMapTasks, minSplitSize, fsBlockSize
>   output is <file,start,length>*
>   totalSize = sum of all file sizes;
>   desiredSplitSize = totalSize / numMapTasks;
>   if (desiredSplitSize > fsBlockSize)             /* new */
>     desiredSplitSize = fsBlockSize;
>   if (desiredSplitSize < minSplitSize)
>     desiredSplitSize = minSplitSize;
>   chop input files into desiredSplitSize chunks & return them
> In other words, the numMapTasks is a desired minimum.  We'll try to chop input into at least numMapTasks chunks, each ideally a single fs block.
> If there's not enough input data to create numMapTasks tasks, each with an entire block, then we'll permit tasks whose input is smaller than a filesystem block, down to a minimum split size.
> This handles cases where:
>   - each input record takes a lot of time to process.  In this case we want to make sure we use all of the cluster.  Thus it is important to permit splits smaller than the fs block size.
>   - input i/o dominates.  In this case we want to permit the placement of tasks on hosts where their data is local.  This is only possible if splits are fs block size or smaller.
> Are there other common cases that this algorithm does not handle well?
> The part marked 'new' above is not currently implemented, but I'd like to add it.
> Does this sound reasonble?

-- 
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators:
   http://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see:
   http://www.atlassian.com/software/jira