Mailing-List: contact core-dev-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: core-dev@hadoop.apache.org
Message-ID: <955748534.1234350062768.JavaMail.jira@brutus>
Date: Wed, 11 Feb 2009 03:01:02 -0800 (PST)
From: "Luca Telloli (JIRA)" <jira@apache.org>
To: core-dev@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-5188) Modifications to enable multiple
 types of logging
In-Reply-To: <1079209870.1233922679612.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/HADOOP-5188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12672588#action_12672588 ] 

Luca Telloli commented on HADOOP-5188:
--------------------------------------

Raghu, in reply to your comment https://issues.apache.org/jira/browse/HADOOP-5189?focusedCommentId=12672454#action_12672454 it's hard for me to tell if the current interface is "inadequate", but I can definitely tell was insufficient for our needs, and this brings back to the logSync() issue, that I'll detail here for the sake of completeness.

In the current implementation, during sync/flush a process among others gets elected to be the "flusher" while the others wait for it to complete; the fact that the NN writes multiple entries for each flush due to the interleaving of threads is a very well written piece of code, but it is not good for our purposes.

What we do with BookKeeper is to allow multiple parallel ordered asynchronous writes to be pipelined on the BookKeeper side rather than the NN side, and to achieve this we need to hack the logSync() method to avoid unnecessary synchronizations. While the NN uses multiple synchronized threads, BookKeeper uses queues, which we believe should help the overall performance. 

Shortly: Bk does implement the two interfaces + hacks the logSync(), which is not part of the interfaces. So I went the "abstract class" way, do you think that's reasonable? 


> Modifications to enable multiple types of logging 
> --------------------------------------------------
>
>                 Key: HADOOP-5188
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5188
>             Project: Hadoop Core
>          Issue Type: Improvement
>          Components: dfs
>    Affects Versions: 0.19.0
>            Reporter: Luca Telloli
>         Attachments: HADOOP-5188.patch
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.