hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From pob <peterob...@gmail.com>
Subject Re: TestDFSIO results
Date Tue, 03 May 2011 15:42:03 GMT
Hello,
these are results from 3 and 6 nodes cluster.

3modes
Throughput mb/sec: 16.32167574644889

6nodes
 Throughput mb/sec: 23.875848571907156

The insertion time of data is ~triple with 3nodes, but throughput s are near
same.


Whats the meaning of that throughput is?


Thanks


3nodes
11/05/03 17:34:23 INFO mapred.FileInputFormat: ----- TestDFSIO ----- : write
11/05/03 17:34:23 INFO mapred.FileInputFormat:            Date & time: Tue
May 03 17:34:23 CEST 2011
11/05/03 17:34:23 INFO mapred.FileInputFormat:        Number of files: 1000
11/05/03 17:34:23 INFO mapred.FileInputFormat: Total MBytes processed:
128000
11/05/03 17:34:23 INFO mapred.FileInputFormat:      Throughput mb/sec:
16.32167574644889
11/05/03 17:34:23 INFO mapred.FileInputFormat: Average IO rate mb/sec:
21.17328643798828
11/05/03 17:34:23 INFO mapred.FileInputFormat:  IO rate std deviation:
5.105609326092506
11/05/03 17:34:23 INFO mapred.FileInputFormat:     Test exec time sec:
1912.383



6 nodes

11/05/03 16:21:46 INFO mapred.FileInputFormat: ----- TestDFSIO ----- : write
11/05/03 16:21:46 INFO mapred.FileInputFormat:            Date & time: Tue
May 03 16:21:46 CEST 2011
11/05/03 16:21:46 INFO mapred.FileInputFormat:        Number of files: 1000
11/05/03 16:21:46 INFO mapred.FileInputFormat: Total MBytes processed:
128000
11/05/03 16:21:46 INFO mapred.FileInputFormat:      Throughput mb/sec:
23.875848571907156
11/05/03 16:21:46 INFO mapred.FileInputFormat: Average IO rate mb/sec:
25.087316513061523
11/05/03 16:21:46 INFO mapred.FileInputFormat:  IO rate std deviation:
5.885708523356224
11/05/03 16:21:46 INFO mapred.FileInputFormat:     Test exec time sec:
734.372
11/05/03 16:21:46 INFO mapred.FileInputFormat:



2011/5/3 pob <peterob333@gmail.com>

> Hello,
>
> I run hadoop jar hadoop-0.20.2-test.jar TestDFSIO -write -nrFiles 1000
> -fileSize 128, because wanted to know what is the througput of my cluster
> (how many data per sec its able to write).
>
> Here are results, that doesnt have any "meaning"  for my point of view.
>
> 11/05/03 16:21:46 INFO mapred.FileInputFormat:      Throughput mb/sec:
> 23.875848571907156
> 11/05/03 16:21:46 INFO mapred.FileInputFormat: Average IO rate mb/sec:
> 25.087316513061523
>
> The througput mb/s suppose to be:  total MB written / total time.
> Apparently not? Or its for one node or what does it mean?
>
> During the test i was monitoring all nodes disk and there was  stale write
> traffic of 120MB/s.
>
>
> Thanks
>
>
> Best
> P
>

Mime
View raw message