hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Flavio Paiva Junqueira (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-5188) Modifications to enable multiple types of logging
Date Wed, 25 Feb 2009 14:47:02 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-5188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12676669#action_12676669
] 

Flavio Paiva Junqueira commented on HADOOP-5188:
------------------------------------------------

Konstantin says that if we don't synchronize on logSync, then edits can be written multiple
times and the edit log would be corrupted. Although this statement sounds correct for the
current implementation of FSEditLog, it is not mandatory for every possible implementation
of an EditLog, and that's why Luca is proposing to change it. One example is the implementation
in the patch. With this implementation, there is no shared buffer, and each thread has its
own buffer and writes that buffer to a BookKeeper ledger, which obviates the need for synchronization
on logSync.

I understand that there might be a concern about the semantics of the file system, but in
the way I see there is no such a problem. Having a buffer for each thread and writing it to
BookKeeper does not violate the file system semantics because BookKeeper respects the order
of calls to add edits to a ledger, and calls back to the Namenode using the same order. That
is, the namenode will receive callbacks in the same order that have been submitted, so if
the namenode processes return values using the same order of callbacks, then the execution
order will be respected. Note that there is no change with respect to the order of callback
processing, and if the Namenode processes them correctly today, then I don't see why it wouldn't
with the proposed solution. 

Back to the logSync issue, in my interpretation, Konstantin is trying to enforce the semantics
defined for the file system by not allowing logSync to be overridden. This is a valid argument,
but the lack of flexibility precludes solutions that do not require logSync implemented in
the way it is currently, so I'd rather have a more flexible implementation.  

> Modifications to enable multiple types of logging 
> --------------------------------------------------
>
>                 Key: HADOOP-5188
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5188
>             Project: Hadoop Core
>          Issue Type: Improvement
>          Components: dfs
>    Affects Versions: 0.19.0
>            Reporter: Luca Telloli
>         Attachments: HADOOP-5188.patch
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message