hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Pete Wyckoff (JIRA)" <j...@apache.org>
Subject [jira] Updated: (HADOOP-4397) fuse-dfs causes corruptions on multi-threaded access
Date Wed, 15 Oct 2008 21:10:44 GMT

     [ https://issues.apache.org/jira/browse/HADOOP-4397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Pete Wyckoff updated HADOOP-4397:

    Attachment: HADOOP-4397.2.txt

I suggest we use this patch which fixes both bugs and is the version i intend to submit to
trunk - it needs to be compiled with the extra flag -Dlibhdfs.noperms=1  so that 0.19 new
permission functions in libhdfs are not used.  I actually use this version of the code with
0.17 in production.

of course the unit tests fail for chown and chmod (since they are not implemented) with this
but all else passes.

and i hope it fixes all the concurrency problems including system calls that return static
structs and so need to be protected.

> fuse-dfs causes corruptions on multi-threaded access
> ----------------------------------------------------
>                 Key: HADOOP-4397
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4397
>             Project: Hadoop Core
>          Issue Type: Bug
>          Components: contrib/fuse-dfs
>    Affects Versions: 0.18.1
>            Reporter: Brian Bockelman
>             Fix For: 0.18.2
>         Attachments: HADOOP-4397.2.txt, hadoop-4397.patch
> If multiple threads in the same process perform file system reads, then fuse-dfs causes
various problems due to the per-context buffer.  I've seen this reflected in segmentation
violations and corruptions.
> I'll attach a proposed patch which takes the "easy way" out - I surround all calls to
dfs_read with a mutex.  You will obviously get performance degradations through thrashing
if the threads are reading different parts of the file (but for our application, the multi-threaded
reads are very, very infrequent.
> If we want to have fuse-dfs writes/reads in 0.19 or 0.20, we'll probably need to do the
same thing with writes.
> This patch could be easily integrated as stands, or a more elaborate approach could be
taken - per-thread buffers maybe?
> Thanks as always for looking into this,
> Brian

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message