hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Da Zheng <zhen...@cs.jhu.edu>
Subject Re: Hadoop use direct I/O in Linux?
Date Thu, 06 Jan 2011 20:08:41 GMT
On 01/05/2011 12:44 AM, Christopher Smith wrote:
>> Yes, my C program can reach 100MB/s or even 110MB/s when writing data to
>> the
>> disk sequentially, but with direct I/O enabled, the maximal throughput is
>> about
>> 140MB/s. But the biggest difference is CPU usage.
>> Without direct I/O, operating system uses a lot of CPU time (the data below
>> is
>> got with top, and this is a dual-core processor with hyperthread enabled).
>> Cpu(s):  3.4%us, 32.8%sy,  0.0%ni, 50.0%id, 12.1%wa,  0.0%hi,  1.6%si,
>>   0.0%st
>> But with direct I/O, the system time can be as little as 3%.
>>
> I'm surprised that system time is really that high. We did Atom experiments
> where it wasn't even close to that. Are you using a memory mapped file? If
> not are you buffering your writes? Is there perhaps
> something dysfunctional about the drive controller/driver you are using?
Which Atom processor did you use? Could you tell me how you did your 
experiment?

Best,
Da

Mime
View raw message