hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Brian Bockelman (JIRA)" <j...@apache.org>
Subject [jira] Created: (HADOOP-4397) fuse-dfs causes corruptions on multi-threaded access
Date Sat, 11 Oct 2008 17:05:44 GMT
fuse-dfs causes corruptions on multi-threaded access

                 Key: HADOOP-4397
                 URL: https://issues.apache.org/jira/browse/HADOOP-4397
             Project: Hadoop Core
          Issue Type: Bug
          Components: contrib/fuse-dfs
    Affects Versions: 0.18.1
            Reporter: Brian Bockelman
             Fix For: 0.18.2

If multiple threads in the same process perform file system reads, then fuse-dfs causes various
problems due to the per-context buffer.  I've seen this reflected in segmentation violations
and corruptions.

I'll attach a proposed patch which takes the "easy way" out - I surround all calls to dfs_read
with a mutex.  You will obviously get performance degradations through thrashing if the threads
are reading different parts of the file (but for our application, the multi-threaded reads
are very, very infrequent.

If we want to have fuse-dfs writes/reads in 0.19 or 0.20, we'll probably need to do the same
thing with writes.

This patch could be easily integrated as stands, or a more elaborate approach could be taken
- per-thread buffers maybe?

Thanks as always for looking into this,


This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message