hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Luca Telloli (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-5188) Modifications to enable multiple types of logging
Date Wed, 11 Feb 2009 11:01:02 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-5188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12672588#action_12672588

Luca Telloli commented on HADOOP-5188:

Raghu, in reply to your comment https://issues.apache.org/jira/browse/HADOOP-5189?focusedCommentId=12672454#action_12672454
it's hard for me to tell if the current interface is "inadequate", but I can definitely tell
was insufficient for our needs, and this brings back to the logSync() issue, that I'll detail
here for the sake of completeness.

In the current implementation, during sync/flush a process among others gets elected to be
the "flusher" while the others wait for it to complete; the fact that the NN writes multiple
entries for each flush due to the interleaving of threads is a very well written piece of
code, but it is not good for our purposes.

What we do with BookKeeper is to allow multiple parallel ordered asynchronous writes to be
pipelined on the BookKeeper side rather than the NN side, and to achieve this we need to hack
the logSync() method to avoid unnecessary synchronizations. While the NN uses multiple synchronized
threads, BookKeeper uses queues, which we believe should help the overall performance. 

Shortly: Bk does implement the two interfaces + hacks the logSync(), which is not part of
the interfaces. So I went the "abstract class" way, do you think that's reasonable? 

> Modifications to enable multiple types of logging 
> --------------------------------------------------
>                 Key: HADOOP-5188
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5188
>             Project: Hadoop Core
>          Issue Type: Improvement
>          Components: dfs
>    Affects Versions: 0.19.0
>            Reporter: Luca Telloli
>         Attachments: HADOOP-5188.patch

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message