hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Konstantin Shvachko (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-5188) Modifications to enable multiple types of logging
Date Thu, 19 Feb 2009 21:44:02 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-5188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12675143#action_12675143

Konstantin Shvachko commented on HADOOP-5188:

Sorry it took me a little more than a couple of days.

_Luca> will your patch support multiple logging system at a time?_

Yes HADOOP-4539 supports multiple logging systems. Namely, it supports traditional file streams
and (what I call) backup streams. In my view you should be able to extend EditLogOutputStream
and it will work with BookKeeper.

_Luca> 0.19 didn't allow me to modify the logSync() method. Will the patch you'll post
fix the issue?_

No I don't think logSync() method should be modified at least not the way you did in your
patch HADOOP-5189.
Synchronization (locking) in the logSync() method prevents different handler threads to write
the same information (edits) multiple times into the same edits file. Suppose you have 1 file
stream (one edits file) and 2 threads call logSync() at the same time. If you remove locking
as you did in your patch then both threads may write the same edits transactions into the
same file twice. This will simply corrupt the edits file.

logSync() may be modified to write in parallel into _different streams_. If you have 2 file
streams then it makes sense to write into 2 files in parallel. But the information should
be written into each file only once and in the right order. And your patch breaks that, afaiu.

My main objection to your approach though, is abstracting the whole EditsLog class rather
than just the {{EditLog*Streams}}. This leads to replication of a lot of code. Say, both classes
{{EditLog}} and {{BKEditLogThreadBuf}} have the {{loadFSEdits()}} methods, which are literally
identical, right? We should avoid that at all cost.

> Modifications to enable multiple types of logging 
> --------------------------------------------------
>                 Key: HADOOP-5188
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5188
>             Project: Hadoop Core
>          Issue Type: Improvement
>          Components: dfs
>    Affects Versions: 0.19.0
>            Reporter: Luca Telloli
>         Attachments: HADOOP-5188.patch

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message