hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Doug Cutting (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-38) default splitter should incorporate fs block size
Date Tue, 14 Feb 2006 21:20:11 GMT
    [ http://issues.apache.org/jira/browse/HADOOP-38?page=comments#action_12366385 ] 

Doug Cutting commented on HADOOP-38:

The surest way to get larger chunks is to increase the block size.

The default DFS blocksize is currently 32MB, which gives 31k tasks for terabyte inputs, which
is reasonable.  I think we should design things to be able to handle perhaps a million tasks,
which, with the current block size, would get us to 32 terabyte inputs.

Perhaps the default should be 1GB/block.  With a million tasks, would get us to maximum of
a petabyte per job.  On a 10k node cluster, a petabyte takes hours to read (100GB/node @ 10MB/second
= 10k seconds).

We'll also need to revise the web UI to better handle a million tasks...

> default splitter should incorporate fs block size
> -------------------------------------------------
>          Key: HADOOP-38
>          URL: http://issues.apache.org/jira/browse/HADOOP-38
>      Project: Hadoop
>         Type: Improvement
>   Components: mapred
>     Reporter: Doug Cutting

> By default, the file splitting code should operate as follows.
>   inputs are <file>*, numMapTasks, minSplitSize, fsBlockSize
>   output is <file,start,length>*
>   totalSize = sum of all file sizes;
>   desiredSplitSize = totalSize / numMapTasks;
>   if (desiredSplitSize > fsBlockSize)             /* new */
>     desiredSplitSize = fsBlockSize;
>   if (desiredSplitSize < minSplitSize)
>     desiredSplitSize = minSplitSize;
>   chop input files into desiredSplitSize chunks & return them
> In other words, the numMapTasks is a desired minimum.  We'll try to chop input into at
least numMapTasks chunks, each ideally a single fs block.
> If there's not enough input data to create numMapTasks tasks, each with an entire block,
then we'll permit tasks whose input is smaller than a filesystem block, down to a minimum
split size.
> This handles cases where:
>   - each input record takes a lot of time to process.  In this case we want to make sure
we use all of the cluster.  Thus it is important to permit splits smaller than the fs block
>   - input i/o dominates.  In this case we want to permit the placement of tasks on hosts
where their data is local.  This is only possible if splits are fs block size or smaller.
> Are there other common cases that this algorithm does not handle well?
> The part marked 'new' above is not currently implemented, but I'd like to add it.
> Does this sound reasonble?

This message is automatically generated by JIRA.
If you think it was sent incorrectly contact one of the administrators:
For more information on JIRA, see:

View raw message