hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Brian Bockelman <bbock...@cse.unl.edu>
Subject Re: FUSE HDFS significantly slower
Date Tue, 26 Oct 2010 12:43:13 GMT
In general, unless you run newer kernels and versions of FUSE as that ticket suggests, it is
significantly slower in raw throughput.

However, we generally don't have a day go by at my site where we don't push FUSE over 30Gbps,
as the bandwidth is spread throughout nodes.  Additionally, as we are limited by the latency
of spinning disk and random reads, we don't particularly hurt by going "only" 60MB/s on our
nodes.  If we wanted to go faster, we use the native clients.

Of course, if anyone wants to donate a lowly university 1.5PB of SSDs, I'm all ears :)

Brian

On Oct 26, 2010, at 12:40 AM, Ted Yu wrote:

> https://issues.apache.org/jira/browse/HADOOP-3805 tried to mitigate this
> problem.
> 
> On Mon, Oct 25, 2010 at 10:17 PM, aniket ray <aniket.ray@gmail.com> wrote:
> 
>> Hi,
>> 
>> I'm seeing in my experiments that Fuse-HDFS is significantly slower (around
>> 3x slower) than using the Java hdfs API directly.
>> Wanted to ask if this slowness the norm? Or is there something wrong with
>> my
>> configuration.
>> Also is this purely JNI slowness or is there something deeper to it?
>> 
>> 
>> My experiment is basically opening a file in write mode and calling writes
>> multiple times  (close to 5GB data) to write to that file.
>> 
>> Thanks for the help,
>> aniket ray
>> 


Mime
View raw message