hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Owen O'Malley" <owen.omal...@gmail.com>
Subject Re: HDFS read/write speeds, and read optimization
Date Fri, 10 Apr 2009 16:03:22 GMT
On Thu, Apr 9, 2009 at 9:30 PM, Brian Bockelman <bbockelm@cse.unl.edu> wrote:
>
> On Apr 9, 2009, at 5:45 PM, Stas Oskin wrote:
>
>> Hi.
>>
>> I have 2 questions about HDFS performance:
>>
>> 1) How fast are the read and write operations over network, in Mbps per
>> second?
>>
>
> Depends.  What hardware?  How much hardware?  Is the cluster under load?
>  What does your I/O load look like?  As a rule of thumb, you'll probably
> expect very close to hardware speed.

For comparison, on a 1400 node cluster, I can checksum 100 TB in
around 10 minutes, which means I'm seeing read averages of roughly 166
GB/sec. For writes with replication of 3, I see roughly 40-50 minutes
to write 100TB, so roughly 33 GB/sec average. Of course the peaks are
much higher. Each node has 4 SATA disks, dual quad core, and 8 GB of
ram.

-- Owen

Mime
View raw message