hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Pete Wyckoff (JIRA)" <j...@apache.org>
Subject [jira] Updated: (HADOOP-4397) fuse-dfs causes corruptions on multi-threaded access
Date Thu, 16 Oct 2008 22:54:44 GMT

     [ https://issues.apache.org/jira/browse/HADOOP-4397?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Pete Wyckoff updated HADOOP-4397:
---------------------------------

    Attachment: hadoop-4397.out

     [exec] +1 overall.  

     [exec]     +1 @author.  The patch does not contain any @author tags.

     [exec]     +1 tests included.  The patch appears to include 21 new or modified tests.

     [exec]     +1 javadoc.  The javadoc tool did not generate any warning messages.

     [exec]     +1 javac.  The applied patch does not increase the total number of javac compiler
warnings.

     [exec]     +1 findbugs.  The patch does not introduce any new Findbugs warnings.



patch is good.  this is c code that does not affect anything else so not attaching ant test.


> fuse-dfs causes corruptions on multi-threaded access
> ----------------------------------------------------
>
>                 Key: HADOOP-4397
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4397
>             Project: Hadoop Core
>          Issue Type: Bug
>          Components: contrib/fuse-dfs
>    Affects Versions: 0.18.1
>            Reporter: Brian Bockelman
>             Fix For: 0.18.2
>
>         Attachments: HADOOP-4397.2.txt, HADOOP-4397.3.txt, HADOOP-4397.4.txt, hadoop-4397.out,
hadoop-4397.patch
>
>
> If multiple threads in the same process perform file system reads, then fuse-dfs causes
various problems due to the per-context buffer.  I've seen this reflected in segmentation
violations and corruptions.
> I'll attach a proposed patch which takes the "easy way" out - I surround all calls to
dfs_read with a mutex.  You will obviously get performance degradations through thrashing
if the threads are reading different parts of the file (but for our application, the multi-threaded
reads are very, very infrequent.
> If we want to have fuse-dfs writes/reads in 0.19 or 0.20, we'll probably need to do the
same thing with writes.
> This patch could be easily integrated as stands, or a more elaborate approach could be
taken - per-thread buffers maybe?
> Thanks as always for looking into this,
> Brian

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message