hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From William Kinney <william.kin...@gmail.com>
Subject Re: Hadoop read/write performance tests problem
Date Tue, 15 Dec 2009 01:13:03 GMT
I had a similar issue and found the profiling information to be helpful:
http://hadoop.apache.org/common/docs/current/mapred_tutorial.html#Profiling

It should tell you where it's spending most of the time.

On Wed, Dec 9, 2009 at 8:51 AM, Dmitriy Lyfar <dlyfar@gmail.com> wrote:

> Hi,
>
> We are using hadoop installation over 6 nodes, that looks like:
>
> 172.19.0.10 (master)
> 172.19.0.11
> 172.19.0.12
> 172.19.0.13
> 172.19.0.14
> 172.19.0.15
>
> Each node has (4x1.5 TB) hard disk (hdparm -t shows ~125 MB/sec for each
> other) and 12 GB memory. Connected to
> 100 Mb switch.
> I get following results with standard io tests, running as:
> bin/hadoop jar hadoop-*-test.jar TestDFSIO -read -nrFiles 15 -fileSize 512
> -bufferSize 1000000
>
> Write test:
>
>  ----- TestDFSIO ----- : write
> Number of files: 15
> Total MBytes processed: 7680
> Throughput mb/sec: 123.82503264917852
> IO rate mb/sec: 129.23692321777344
> IO rate std deviation: 22.25608461080549
> Test exec time sec: 28.656
>
> Read test:
>  ----- TestDFSIO ----- : read
> Date & time: Wed Dec 09 13:42:52 CET 2009
> Number of files: 15
> Total MBytes processed: 7680
> Throughput mb/sec: 4.833249737412311
> Average IO rate mb/sec: 37.22291946411133
> IO rate std deviation: 65.57958375615678
> Test exec time sec: 300.542
>
> As you can see read performance is very poor. Also we noticed following:
>
> 1) Run ping on all nodes to master. When write test is working ping's times
> always stable and less than 1 ms, but when read test is working
> we see increased ping's times:
>
> 64 bytes from 172.19.0.10: icmp_seq=4444 ttl=64 time=13.3 ms
> 64 bytes from 172.19.0.10: icmp_seq=4445 ttl=64 time=18.1 ms
> 64 bytes from 172.19.0.10: icmp_seq=4446 ttl=64 time=18.3 ms
> 64 bytes from 172.19.0.10: icmp_seq=4447 ttl=64 time=14.3 ms
>
> 2) When running write test iostat shows that there is activity on all 4
> disks on node, but when running read test, there is almost no activity.
>
> Could you explain where problem could be? Thank you.
>
> --
> Regards, Dmitriy
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message