Mailing-List: contact core-dev-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: core-dev@hadoop.apache.org
Message-ID: <546295372.1223858204236.JavaMail.jira@brutus>
Date: Sun, 12 Oct 2008 17:36:44 -0700 (PDT)
From: "Pete Wyckoff (JIRA)" <jira@apache.org>
To: core-dev@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-4397) fuse-dfs causes corruptions on
 multi-threaded access
In-Reply-To: <1844616353.1223744744198.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/HADOOP-4397?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12638927#action_12638927 ] 

Pete Wyckoff commented on HADOOP-4397:
--------------------------------------

thanks Brian.

Let's get this patch into 0.18.2 and something like this into 0.19 (if possible as this should be a blocker).

And then, right, work on something better.

One question though, it doesn't look like you protected all writes to the dfs context in other operations. Also, do you have a test case for this? I know it's nearly impossible :) .

We should probably work on 0.19 patch with the highest priority since it may be released soon (or maybe has ?).

pete


> fuse-dfs causes corruptions on multi-threaded access
> ----------------------------------------------------
>
>                 Key: HADOOP-4397
>                 URL: https://issues.apache.org/jira/browse/HADOOP-4397
>             Project: Hadoop Core
>          Issue Type: Bug
>          Components: contrib/fuse-dfs
>    Affects Versions: 0.18.1
>            Reporter: Brian Bockelman
>             Fix For: 0.18.2
>
>         Attachments: hadoop-4397.patch
>
>
> If multiple threads in the same process perform file system reads, then fuse-dfs causes various problems due to the per-context buffer.  I've seen this reflected in segmentation violations and corruptions.
> I'll attach a proposed patch which takes the "easy way" out - I surround all calls to dfs_read with a mutex.  You will obviously get performance degradations through thrashing if the threads are reading different parts of the file (but for our application, the multi-threaded reads are very, very infrequent.
> If we want to have fuse-dfs writes/reads in 0.19 or 0.20, we'll probably need to do the same thing with writes.
> This patch could be easily integrated as stands, or a more elaborate approach could be taken - per-thread buffers maybe?
> Thanks as always for looking into this,
> Brian

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.