hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Amir Langer (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HDFS-5732) Separate memory space between BM and NN
Date Thu, 30 Jan 2014 13:30:09 GMT

     [ https://issues.apache.org/jira/browse/HDFS-5732?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Amir Langer updated HDFS-5732:
------------------------------

    Attachment: 0002-Separation-of-BM-from-NN-Step-2-Separate-memory-spac.patch

Separation of memory space between BM and NN. This is an incremental patch on top of the patch
for the first sub task (HDFS-5731)

NN=NameNode
BM=BlockManager
DN=DataNode

This patch mainly separates the block instances between BM and NN.
This means that Inodes contain different BlockInfo instances from the BlocksMap. 
The replications information is only maintained by the instances in the BlocksMap - and those
no longer contain a reference to the Inode but the id instead. (The new object is called BlockReplicationInfo).

Improved API between BM and NN to pass more ids instead of references between the two (Although
not complete, still work to invest there).
Also - Separation of memory space meant there are a few "upcalls" - calls where the BM calls
the NN.
Some of these are normal (when the call originated from the DN and changed the state of the
block).
But some should be eliminated as they create a circular call NN->BM->NN.
Currently not a problem (and was always there) - but will become problematic when we try and
separate BM to its own process.

The next sub-task should deal with those.



> Separate memory space between BM and NN
> ---------------------------------------
>
>                 Key: HDFS-5732
>                 URL: https://issues.apache.org/jira/browse/HDFS-5732
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: namenode
>            Reporter: Amir Langer
>         Attachments: 0002-Separation-of-BM-from-NN-Step-2-Separate-memory-spac.patch
>
>
> Change created APIs to not rely on the same instance being shared in both BM and NN.
Use immutable objects / keep state in sync.
> BM and NN will still exist in the same VM work on a new BM service as an independent
process is deferred to later tasks.
> Also, a one to one relation between BM and NN is assumed. 
> This task should maintain backward compatibility.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)

Mime
View raw message