hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From KrzyCube <yuxh...@gmail.com>
Subject About read/write performance
Date Wed, 04 Jul 2007 10:06:46 GMT

I test upload and download files(bunch of files , from 1M~ N*100M) with the
FSShell . and There are two problems:

1. write is much faster than read.
    with single DataNode and single client ,  average UPLOAD speed is 6m/s ,
this is the speed that my network has( i test the real network performance
with ftp protocol ,aslo 6m/s ). but average DOWNLOAD speed is only 3~4 m/s.
    Is this normal ? Why this happened ?  as i know , the write and read
process is almost same , connect to the Master , get info , then communicate
with DataNode.  i think CRC won't take so much time .
    
2. Cluster is slower than single
    with 3 DataNode , set 3 replication , upload and download again .  (Is
replication upload to other datanodes immediately , or just let
SecondNameNode to do this  ?)
    this time , in same network , the average UPLOAD speed is 1~2 m/s  , and
DOWNLOAD speed is 700~900 KB/s . 
so , both single and cluster ,  Download ~= Upload * 0.5 .

Isn't that crazy ?  the cluster is eating performance.

any body could told me about these problems  , this real confused me .

And , a question :
      what's those in hadoop may affect performance . 
"blocks seek , CRC , Network latency" ?
-- 
View this message in context: http://www.nabble.com/About-read-write-performance-tf4023552.html#a11428257
Sent from the Hadoop Users mailing list archive at Nabble.com.


Mime
View raw message