hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jason Venner <jason.had...@gmail.com>
Subject Re: Hadoop DFS IO Performance measurement
Date Thu, 01 Apr 2010 01:31:58 GMT
Unless you are getting all local IO, and or you have better than GigE
nic interfaces
100MB/sec is your cap.

For local IO the bound is going to be your storage subsystem.
Decent drives in a raid 0 interface are going to cap out on those
machines about 400MB/sec, which is the buffer cache bandwidth on those
processors/memory.
realistically you are going to see a quite a bit less, but 200 should be doable.


On Wed, Mar 31, 2010 at 2:55 PM, Sagar Naik <snaik@attributor.com> wrote:
> Hi Edson,
>
> usual commodity machine :
> 8GB Ram, 2.5GHZ Intel Xeon,
>
> 6 Disks : 2 drives RAID 1 , 4 RAID 0 using PERC 6
>
> Centos , ext4
>
> Datanodes configured to use 4 RAID 0 drives
> OS and hadoop installation on RAID 1
>
> -Sagar
> On Mar 30, 2010, at 4:21 PM, Edson Ramiro wrote:
>
>> Hi Sagar,
>>
>> What hardware did you run it on ?
>>
>> Edson Ramiro
>>
>>
>> On 30 March 2010 19:41, sagar naik <snaik@attributor.com> wrote:
>>
>>> Hi All,
>>>
>>> I am trying to get DFS IO performance.
>>> I used TestDFSIO from hadoop jars.
>>> The results were abt 100Mbps read and write .
>>> I think it should be more than this
>>>
>>> Pl share some stats to compare
>>>
>>> Either I am missing something like  config params or something else
>>>
>>>
>>> -Sagar
>>>
>
>



-- 
Pro Hadoop, a book to guide you from beginner to hadoop mastery,
http://www.amazon.com/dp/1430219424?tag=jewlerymall
www.prohadoopbook.com a community for Hadoop Professionals

Mime
View raw message