hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Raghu Angadi <rang...@yahoo-inc.com>
Subject Re: Faster alternative to FSDataInputStream
Date Wed, 19 Aug 2009 20:35:17 GMT
Ananth T. Sarathy wrote:
> Also, I just want to clear... the delay seems to at the intial
> 
> (read = in.read(buf))

It the file on HDFS (over S3) or S3?

Does it always happen?

Raghu.

> after the first time into the loop it flies...
> 
> Ananth T Sarathy
> 
> 
> On Wed, Aug 19, 2009 at 1:58 PM, Raghu Angadi <rangadi@yahoo-inc.com> wrote:
> 
>> Edward Capriolo wrote:
>>
>>> On Wed, Aug 19, 2009 at 11:11 AM, Edward Capriolo <edlinuxguru@gmail.com
>>>>> wrote:
>>>>  It would be as fast as underlying filesystem goes.
>>>>>> I would not agree with that statement. There is overhead.
>> You might be misinterpreting my comment. There is of course some over head
>> (at the least the procedure calls).. depending on you underlying filesystem,
>> there could be extra buffer copies and CRC overhead. But none of that
>> explains transfer as slow as 1 MBps (if my interpretation of of results is
>> correct).
>>
>> Raghu.
>>
>>

Mime
View raw message