hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Konstantin Shvachko (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HDFS-1073) Simpler model for Namenode's fs Image and edit Logs
Date Tue, 09 Nov 2010 09:14:19 GMT

    [ https://issues.apache.org/jira/browse/HDFS-1073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12930023#action_12930023

Konstantin Shvachko commented on HDFS-1073:

Rob, I seems we cannot have the "I'm quitting!" record just because there is no "quit" or
"shutdown" command. I agree the "rolled" transaction can be useful for a sanity check for
the edits files that are not in-progress.

Todd, based on the design doc (I should have read first thing) I don't see much difference
between the current and your new implementation ascept that you don't need a side file to
write the edits while spooling. Currently BN.startCheckpoint() causes NN.rollEdits(), which
in turn sends back to BN the SPOOL_START record. This is when BN starts spooling. You seem
to be trying to call the process of spooling (writing into edits file but not applying to
memory) by journal. That is how the state is called in your design, right? Which may be confusing
as BN continues journaling (writing to edits file) whether it is in synchronized or in spooling
Also I don't see how you can get by with only 2 states for BN you need 3. While spooling there
are 2 active threads: one (writer) is writing edits from NN directly to the edits_K, another
(reader) is reading formerly written records from edits_K. At the end we need to switch the
writer thread from writing to applying the records to in memory state and shut down the second
thread. This is where you need the third state, currently called WAIT. When the reader thread
reaches the end of file it sets the WAIT state. The writer may still be writing before it
sees the WAIT state. After seeing WAIT it blocks and waits until spooling is OFF. The reader
read the remaining records and turns the spooling OFF.
Let me summarize the current journal spool states meaning
- JSpoolState.OFF - spooling is off, apply edits to memory state and write into journal (edits
- JSpoolState.INPROGRESS - spooling in progress, do not apply to memory, just journal
- JSpoolState.WAIT - stop, do nothing wait until spooling is OFF

Does that make sense?

> Simpler model for Namenode's fs Image and edit Logs 
> ----------------------------------------------------
>                 Key: HDFS-1073
>                 URL: https://issues.apache.org/jira/browse/HDFS-1073
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>            Reporter: Sanjay Radia
>            Assignee: Todd Lipcon
>         Attachments: hdfs-1073.txt, hdfs1073.pdf
> The naming and handling of  NN's fsImage and edit logs can be significantly improved
resulting simpler and more robust code.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message