Mailing-List: contact core-dev-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: core-dev@hadoop.apache.org
Message-ID: <919460025.1242989685570.JavaMail.jira@brutus>
Date: Fri, 22 May 2009 03:54:45 -0700 (PDT)
From: "Luca Telloli (JIRA)" <jira@apache.org>
To: core-dev@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-5188) Modifications to enable multiple
 types of logging
In-Reply-To: <1079209870.1233922679612.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/HADOOP-5188?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12712022#action_12712022 ] 

Luca Telloli commented on HADOOP-5188:
--------------------------------------

Konstantin, thanks for your comments! Let me address each of your point separately: 

1. I opened HADOOP-5832 by mistake using the subtask option, without knowing that it would have generated a new patch. My bad. But I think that there's no clear cut between the two patches, because with URI processing I'm introducing logging devices which is something related to logging abstraction. Without introducing logging devices, HADOOP-5832 would be trivial, as you'd read a string, treat is as a URI and retrieve the path fragment, at least until other types of logging are developed, unless I'm forgetting something. So, unless you think it's necessary to have separate patches, I'd rather link them. 

2. I think edit streams are opened and closed many times during the life of a server process (namenode/secondaryNN/backupnode) while the LoggingDevices are ideally created only once. While this might not make a difference for files, which can be opened/closed many times, it might fit better other types of logging which might want to store persistent information inside their "LoggingDevice" implementation. For instance, when thinking about Bookkeeper, I want to create a Bookkeeper client only once through the life of a NameNode and use it for Input/Output streams when needed. Does this sound reasonable? 

3. From what I see in FSImage.loadFSEdits(StorageDirectory sd) EditLogFileInputStream needs to access a StorageDirectory object. Assuming that we keep the LoggingDevice abstraction and both edit streams (input and output) need to access it, it sounds better to leave it inside FileLoggingDevice and retrieve it with the getNewInputStreamInstance() method. 

4. In the current patch you can specify values in dfs.name.dir as URI, as you can see in the FSNamesystem.getStorageDirs(...) method, where arguments of type URI are processed as well as path strings, both adding the correct value to the dirNames ArrayList. In general, values of this property will be mapped directly to storage directories, since StorageDirectory is the unit of storage for file system images. The mapping is different for the other property: dfs.name.edits.dir, where the mapping is not one-to-one with StorageDirectories, since other types of device can be used. 

> Modifications to enable multiple types of logging 
> --------------------------------------------------
>
>                 Key: HADOOP-5188
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5188
>             Project: Hadoop Core
>          Issue Type: Improvement
>          Components: dfs
>    Affects Versions: 0.20.0
>            Reporter: Luca Telloli
>             Fix For: 0.21.0
>
>         Attachments: HADOOP-5188.patch, HADOOP-5188.patch, HADOOP-5188.patch, HADOOP-5188.patch, HADOOP-5188.pdf
>
>


-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.