hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Todd Lipcon (Updated) (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HDFS-2563) Some cleanup in BPOfferService
Date Fri, 18 Nov 2011 09:24:51 GMT

     [ https://issues.apache.org/jira/browse/HDFS-2563?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Todd Lipcon updated HDFS-2563:

    Attachment: hdfs-2563.txt

Here is a prelim patch (haven't run through all the unit tests locally yet)... we'll see what
Hudson has to say.

Summary of the changes:

- Add javadoc for bpRegistration, bpNSInfo fields to clarify when they get set
- Remove blockPoolId member and replace with an accessor, since this is already stored as
part of bpNSInfo
- Don't assign bpRegistration until the block pool is actually determined during the handshake
process. Change use of {{bpRegistration}} in log messages to use {{this.toString()}} which
now has a sensible output. We used to see log messages like:
2011-11-18 01:13:24,244 INFO  datanode.DataNode (DataNode.java:run(1195)) - DatanodeRegistration(,
storageID=, infoPort=39362, ipcPort=38231, storageInfo=lv=0;cid=;nsid=0;c=0)In BPOfferService.run,
data = null;bp=null
which looks like an error to the user. Now it will say something like:
2011-11-18 01:19:08,209 INFO  datanode.DataNode (DataNode.java:run(1238)) - Block pool <registering>
(storage id unknown) connecting to todd-w510/ starting to offer service
and later
2011-11-18 01:19:08,395 INFO  datanode.DataNode (DataNode.java:register(1197)) - Block pool
BP-428670582- (storage id DS-329594984-
registered with todd-w510/ beginning handshake with NN at todd-w510/
when the blockPoolId and storageId become known

- Split the handshake method into two clearer parts: (1) {{retrieveNamespaceInfo}} which calls
{{versionRequest}} to determine the NN's version info and namespace/cluster ID, and (2) {{register()}}
like before, which actually registers the DN.
- Refactor the code to check layout versions (used to be duplicated in two places) into a
{{checkNNVersion}} class.
- Some other trivial improvements of log messages, etc, to be more user readable.
- Removed unused test function {{DataNodeTestUtils.setBPNamenodeByIndex}}. Fixed {{TestDatanodeRegister}}
to correspond to above changes.

> Some cleanup in BPOfferService
> ------------------------------
>                 Key: HDFS-2563
>                 URL: https://issues.apache.org/jira/browse/HDFS-2563
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: data-node
>    Affects Versions: 0.24.0, 0.23.1
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>         Attachments: hdfs-2563.txt
> BPOfferService is currently rather difficult to follow and not really commented. This
JIRA is to clean up the code a bit, add javadocs/comments where necessary, and improve the
formatting of the log messages.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message