hadoop-common-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Luca Telloli (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-5188) Modifications to enable multiple types of logging
Date Sat, 30 May 2009 15:04:07 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-5188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12714728#action_12714728

Luca Telloli commented on HADOOP-5188:

Konstantin, to follow up on Flavio's comment, let me use BookKeeper as an example. 

To use bookkeeper as a log device I need to open a client. If I open it in EditLogOutputStream
that client should be destroyed upon a close() call and would be recreated each time a new
EditLogOutputStream is called: I don't think this is the right way to proceed. LogDevice would
create an instance of such a client and keep it persistent thourough the life of a server
process (namenode, etc). This motivates the insufficiency of the current EditStreams. 

So far HDFS has been working only with files for logging. BackupNode is a special type of
process that, although it registers a stream, at the end uses files again. I don't think that
every other log device should spawn a new process similar to BackupNode for logging, although
I assume that in production users will choose only one device. 

Finally, it would be hard to integrate Bookkeeper with the current streams because BookKeeper
ledgers do not support append. Files do, bookkeeper doesn't (instead of appending, we create
another "ledger" - the bookkeeper log file). It's part of the diversity of the two "devices",
it's another argument in favor of the need of an abstraction.

LogDevice has been passing so far all unit tests successfully and it guarantees compatibility
with the current implementation, so it seems a safe absstraction. I believe it's also a good
one but I'd be delighted to hear a concrete proposal to make it better or replace it. 

Other points: 
1. HADOOP-5188 does introduce LogDevice and I agree it's not fully uses yet thorough the code.
But if we want to keep it, we'll do the modifications eventually. 

2. Method EditLogFileOutputStream.switchEditStreams() should eventually contain part of the
rollEditLog() code. In the case of the BackupNode you probably know better what should be
there, and if you wanna contribute to it, you're very welcome. 

3. The use of instanceof has been reduced and it will disappear in my next version, since
I introduced a getType() method. Reflection is used at the end only to instantiate the correct
log device implementation 

> Modifications to enable multiple types of logging 
> --------------------------------------------------
>                 Key: HADOOP-5188
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5188
>             Project: Hadoop Core
>          Issue Type: Improvement
>          Components: dfs
>    Affects Versions: 0.20.0
>            Reporter: Luca Telloli
>             Fix For: 0.21.0
>         Attachments: HADOOP-5188.patch, HADOOP-5188.patch, HADOOP-5188.patch, HADOOP-5188.patch,
HADOOP-5188.patch, HADOOP-5188.patch, HADOOP-5188.pdf

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message