hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Doug Cutting (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-72) hadoop doesn't take advatage of distributed compiting in TestDFSIO
Date Thu, 09 Mar 2006 23:01:21 GMT
    [ http://issues.apache.org/jira/browse/HADOOP-72?page=comments#action_12369754 ] 

Doug Cutting commented on HADOOP-72:

Did you look at the web ui to see how many map tasks were used to execute this?  My suspicion
is that only a single map task is used.  A SequenceFile cannot be split into chunks smaller
than 2k.  With less than 64 files, your single input file is probably less than 2k.  You could
instead use a text input file, which can be split into smaller chunks, or you could use a
custom getSplits() implementation that actually parses the input file, or you could use a
much larger number of files.

> hadoop doesn't take advatage of distributed compiting in TestDFSIO
> ------------------------------------------------------------------
>          Key: HADOOP-72
>          URL: http://issues.apache.org/jira/browse/HADOOP-72
>      Project: Hadoop
>         Type: Test
>   Components: dfs, fs, mapred
>  Environment: 200 node cluster
>     Reporter: Konstantin Shvachko
>  Attachments: TestDFSIO.java, TestDFSIO_results_200_node_cluster.log, TestDFSIO_results_sequential.log
> TestDFSIO runs N map jobs, each either writing to or reading from a separate file of
the same size, 
> and collects statistical information on its performance. 
> The reducer further calculates the overall statistics for all maps. 
> It outputs the following data:
> - read or write test
> - date and time the test finished   
> - number of files
> - total number of bytes processed
> - overall throughput in mb/sec
> - average IO rate in mb/sec per file
> __Results__
> I run 7 iterations of the test one after another on a cluster of ~200 nodes. 
> The file size is the same in all cases 320Mb. 
> The number of files tried is 1,2,4,8,16,32,64.
> The log file with statistics is attached.
> It looks like we don't have any distributed computing here at all.
> The total execution time increases proportionally to the total size of data both for
writes and reads.
> Another thing is that the io ratio for read is higher than the write rate just gradually.
> For comparison I attach time measuring for the same ios performed on the same cluster
but sequentially in a simple loop.
> This is the summary:
> Files	map/red time	sequential time
>  1		49			  34 
>  2		86			  69
>  4		158			131
>  8		299			266
> 16		569			532
> 32		1131
> 64		2218
> This doesn't look good, unless there is something wrong with my test (attached) or the
cluster settings.

This message is automatically generated by JIRA.
If you think it was sent incorrectly contact one of the administrators:
For more information on JIRA, see:

View raw message