hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Colin Patrick McCabe (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-4045) SecondaryNameNode cannot read from QuorumJournal URI
Date Tue, 04 Dec 2012 19:06:58 GMT

    [ https://issues.apache.org/jira/browse/HDFS-4045?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13509948#comment-13509948
] 

Colin Patrick McCabe commented on HDFS-4045:
--------------------------------------------

Here's a proposal that might make this a little nicer:
* augment the {{StreamLimiter}} interface with a {{getRemainingBeforeLimit}} call.  This call
will return how many bytes are left before the limit hits.
* add a field, {{decodedSize}}, to {{FSEDitLogOp.}}  At the end of {{FSEditLogOp#decodeOp}},
set {{this.decodedSize = limiter.getRemainingBeforeLimit()}}.
* add a {{FSEditLogOp#decodeRawOp}} function which returns a byte array.  It does so by calling
readOp in a loop, and noticing the decoded opcode sizes.  Then it just pulls the bytes from
the stream itself.

I think this is nicer because it doesn't burden the main opcode loading path with the need
to re-read every byte twice or create an array with all the opcode bytes each time.  Calculating
the remaining bytes before the limit is really cheap, just a subtraction.  {{decodeRawOp}}
still does re-read everything twice, but this only affects that one code path, not all of
them.  For extra efficiency, decodeRawOp could just fill an array you pass, rather than creating
a lot of garbage all the time.  Allocating a single 1.5 MB array is not a big deal.

Thoughts?
                
> SecondaryNameNode cannot read from QuorumJournal URI
> ----------------------------------------------------
>
>                 Key: HDFS-4045
>                 URL: https://issues.apache.org/jira/browse/HDFS-4045
>             Project: Hadoop HDFS
>          Issue Type: Bug
>          Components: namenode
>    Affects Versions: 3.0.0
>            Reporter: Vinithra Varadharajan
>            Assignee: Andy Isaacson
>         Attachments: hdfs-4045-2.txt, hdfs4045-3.txt, hdfs4045-4.txt, hdfs-4045.txt
>
>
> If HDFS is set up in basic mode (non-HA) with QuorumJournal, and the dfs.namenode.edits.dir
is set to only the QuorumJournal URI and no local dir, the SecondaryNameNode is unable to
do a checkpoint.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message