hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ivan Kelly (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-2018) 1073: Move all journal stream management code into one place
Date Fri, 12 Aug 2011 10:38:27 GMT

    [ https://issues.apache.org/jira/browse/HDFS-2018?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13084034#comment-13084034
] 

Ivan Kelly commented on HDFS-2018:
----------------------------------


{quote}
No, fencing is to keep two NNs from writing at the same time to a storage directory. This
is about one NN reading while the other writes – in your design, the reader ends up finalizing
log segments without explicitly saying "do recovery". Recovery is an operation that should
only be done by the active (writer). So making it an explicit API makes more sense.
{quote}
What is recovery but a write operation? I disagree that an explicit API makes more sense.
The JournalManager must know whether it is a writer or not to verify whether it can create
OutputStreams or not. Since we already have this information, we can use it to inform our
decision whether to finalized recovered in progress logs or not.

The new approach also retains getInProgressInputStream() which is only used in a special case
in BackupNode. We shouldn't have methods in the JournalStream API only for corner cases like
this. My patch gets rid of this. It should be possible to do the same with the other approach.

I agree with Jitendra about the way EditLogReference is used. Streams should be the unit of
currency used in FSImage at least. FSEditLog should return a list of streams from which the
data can be read. getNumberOfTransactions could be replaced with a getTransactionRanges call.
The editlog could then select the list of transaction ranges it wants to load. It then calls
getInputStream() on the JournalManagers to get a stream starting with the firstTxId of the
range. I had considered an approach vaguely similar before, but didn't implement it. 

Also, the name EditLogReference is meaningless. It should be called TransactionRange or similar.
Removing the openForEdit() call makes it equivalent to RemoteEditLog, so RemoteEditLog should
be removed. 

{code}
public interface JournalManager {
    List<TransactionRange> getTransactionRanges(long fromTxId);
    EditLogInputStream getInputStream(long fromTxId);
}

public class FSEditLog {
    List<EditLogInputStream> selectInputStreams(long fromTxId) {
       List<EditLogInputStream> streams = Lists.newArrayList();
       MultiMap<TransactionRange,JournalManager> rangeMap = new MultiMap();
       MultiMap<Long,TransactionRange> rangesByStartTx = new MultiMap();

       for (JournalManager j : journals) {
          List<TransactionRange> ranges = j.getTransactionRanges(fromTxId);
	    for (TransactionRange r : ranges) {
	       rangeMap.add(r, j);
             rangesByStartTx.add(r.getFirstTxId(), r);
          }
       }

       while (true) {
          ImmutableList<TransactionRange> ranges = rangesByStartTxId.get(fromTxId);
	    if (ranges.isEmpty()) {
            break;
          }
	    TransactionRange bestRange = Collections.max(range, TransactionRange.COMPARATOR);
	    assert bestRange.getLastTxId() >= curStartTxId;
	    streams.add(rangeMap.get(bestRange).getInputStream(bestRange.getFirstTxId()));
	    fromTxId = bestLog.getLastTxId() + 1;
	 }
	 return streams;
    }
}
{code}

Im on holiday until the 29th from today, which is why I was eager to get this resolved quickly,
as it has been holding up Jitendra. I won't be able to work on anything until I get back,
so if one of you guys do continue forward, and if you choose the different approach, I'd like
to see the following changes.

- Recovery should not be explicit in the API
- get rid of getInProgressInputStream
- Rename EditLogReference to TransactionRange, remove openForEdit & merge with RemoteEditLog.


> 1073: Move all journal stream management code into one place
> ------------------------------------------------------------
>
>                 Key: HDFS-2018
>                 URL: https://issues.apache.org/jira/browse/HDFS-2018
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>            Reporter: Ivan Kelly
>            Assignee: Ivan Kelly
>             Fix For: 0.23.0
>
>         Attachments: HDFS-2018.diff, HDFS-2018.diff, HDFS-2018.diff, HDFS-2018.diff,
HDFS-2018.diff, HDFS-2018.diff, HDFS-2018.diff, HDFS-2018.diff, HDFS-2018.diff, HDFS-2018.diff,
HDFS-2018.diff, HDFS-2018.diff, HDFS-2018.diff, HDFS-2018.diff, HDFS-2018.diff, HDFS-2018.diff,
hdfs-2018-otherapi.txt, hdfs-2018.txt
>
>
> Currently in the HDFS-1073 branch, the code for creating output streams is in FileJournalManager
and the code for input streams is in the inspectors. This change does a number of things.
>   - Input and Output streams are now created by the JournalManager.
>   - FSImageStorageInspectors now deals with URIs when referring to edit logs
>   - Recovery of inprogress logs is performed by counting the number of transactions instead
of looking at the length of the file.
> The patch for this applies on top of the HDFS-1073 branch + HDFS-2003 patch.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

       

Mime
View raw message