hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mukul Kumar Singh (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-11694) Block Storage: Add Support for 2 BlockIDBuffers and also for periodic flush of BlockIDBuffer
Date Sun, 14 May 2017 17:59:04 GMT

    [ https://issues.apache.org/jira/browse/HDFS-11694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16009807#comment-16009807

Mukul Kumar Singh commented on HDFS-11694:

Hi [~vagarychen], Thanks for the review.

A few clarification notes:
1. could you please elaborate a little bit about the reason for as there are maximum of 2
buffers which can be flushed at the same time.? Basically, what is the reasoning of 2 BlockIDBuffers?
--> done, added a comment 
2 block id buffers are maintained so that the writes are not blocked on the DirtyLog flush.
Once a buffer is full, it is scheduled for DirtyLog write
while the other buffer accepts the write.

2. I think currentBuffer = acquireQueue.take(); is a blocking call and seems there are three
places calling triggerBlockBufferFlush(). Will any of the places block infinitely? I was mainly
concerning about the shutdown() which seems possible to block forever. I might be wrong but
could you please verify these calls?
--> verified this, none of the functions will block infinitely. the blocked thread will
resume as soon as DirtyLog flush completes.

Also, this change is actually fairly complex to follow so it would be great if we can have
more comments explaining the workflow a little bit (especially BlockBufferManager.java), otherwise
it may become very difficult to maintain/modify in the future  .
--> done, added more comments. :)

> Block Storage: Add Support for 2 BlockIDBuffers and also for periodic flush of BlockIDBuffer
> --------------------------------------------------------------------------------------------
>                 Key: HDFS-11694
>                 URL: https://issues.apache.org/jira/browse/HDFS-11694
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: ozone
>            Reporter: Mukul Kumar Singh
>            Assignee: Mukul Kumar Singh
>         Attachments: HDFS-11694-HDFS-7240.001.patch, HDFS-11694-HDFS-7240.002.patch
> This bug tracks 2 issues
> 1) The change supports periodic flushing of BlockIdBuffer, earlier this buffer was only
flushed when either the buffer was full or shutdown was triggered
> 2) This change also adds 2 BlockIdBuffer(s), this will ensure that incoming writes are
not blocked on BlockIdBuffer flush.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org

View raw message