hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Colin Patrick McCabe (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-7923) The DataNodes should rate-limit their full block reports by asking the NN on heartbeat messages
Date Tue, 16 Jun 2015 20:22:01 GMT

    [ https://issues.apache.org/jira/browse/HDFS-7923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14588686#comment-14588686

Colin Patrick McCabe commented on HDFS-7923:

BTW it would be relatively easy to enforce "fairness" on the NN side in this patch.  In fact
I had an earlier version that did do that.  It would give out FBR leases to datanodes based
on a round-robin order.  But I decided against it since it was extra complexity without a
good reason.  In particular, I was worried that I'd create starvation if the set of DNs the
NN selected to report in didn't want to report in (perhaps because they were using initialDelay...).
 Since full block report periods can be very long this is a real concern and would necessitate
a hack like a timeout.  It was much easier to keep block report scheduling as a datanode-side

We can revisit this if it ever seems like a good idea, but I bet that you will not be able
to create starvation in any real cluster without creating conditions that would make the cluster
unusable no matter what.

> The DataNodes should rate-limit their full block reports by asking the NN on heartbeat
> -----------------------------------------------------------------------------------------------
>                 Key: HDFS-7923
>                 URL: https://issues.apache.org/jira/browse/HDFS-7923
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>    Affects Versions: 2.8.0
>            Reporter: Colin Patrick McCabe
>            Assignee: Colin Patrick McCabe
>             Fix For: 2.8.0
>         Attachments: HDFS-7923.000.patch, HDFS-7923.001.patch, HDFS-7923.002.patch, HDFS-7923.003.patch,
HDFS-7923.004.patch, HDFS-7923.006.patch, HDFS-7923.007.patch
> The DataNodes should rate-limit their full block reports.  They can do this by first
sending a heartbeat message to the NN with an optional boolean set which requests permission
to send a full block report.  If the NN responds with another optional boolean set, the DN
will send an FBR... if not, it will wait until later.  This can be done compatibly with optional

This message was sent by Atlassian JIRA

View raw message