hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Da Zheng <zhengda1...@gmail.com>
Subject Re: the performance of HDFS
Date Thu, 27 Jan 2011 04:18:07 GMT

Tools like iostat and iotop isn't installed.
>From the specification, there is a RAID controller, but I'm not sure if it's used.
zhengda@occalit28:~$ cat /proc/scsi/scsi
Attached devices:
Host: scsi0 Channel: 00 Id: 00 Lun: 00
  Vendor: ATA      Model: WDC WD800AAJS-18 Rev: 01.0
  Type:   Direct-Access                    ANSI SCSI revision: 05
Host: scsi1 Channel: 00 Id: 00 Lun: 00
  Vendor: ATA      Model: Hitachi HUA72101 Rev: GKAO
  Type:   Direct-Access                    ANSI SCSI revision: 05


On 1/26/11 12:14 PM, Esteban Gutierrez Moguel wrote:
> Da,
> Do you have permissions to run iostat? That could help you to find
> persistent IO activity from other processes.
> Another possibility is that your system has a RAID controller without a
> cache memory card, from my experience that makes writes awfully slow.
> esteban.
> On Tue, Jan 25, 2011 at 20:29, Da Zheng <zhengda1936@gmail.com> wrote:
>> No, each node in the cluster is powerful server. I was told the nodes are
>> Dell
>> Poweredge SC1435, but I cannot figure out the configuration of hard drives.
>> Dell
>> provides several possible hard drives for this model.
>> On 1/25/11 7:59 PM, Ted Dunning wrote:
>>> This is a really slow drive or controller.
>>> Consumer grade 3.5 inch 2TB drives typically can handle 100MB/s.  I would
>>> suspect in the absence of real information that your controller is more
>>> likely to be deficient than your drive.  If this is on a laptop or
>>> something, then I withdraw my thought.
>>> On Tue, Jan 25, 2011 at 4:50 PM, Da Zheng <zhengda1936@gmail.com> wrote:
>>>> The aggregate write-rate can get much higher if you use more drives, but
>> a
>>>>> single stream throughput is limited to the speed of one disk spindle.
>>>>>  You are right. I measure the performance of the hard drive. It seems
>> the
>>>> bottleneck is the hard drive, but the hard drive is a little too slow.
>> The
>>>> average writing rate is 50MB/s.

View raw message