hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Colin Patrick McCabe (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-7923) The DataNodes should rate-limit their full block reports by asking the NN on heartbeat messages
Date Fri, 05 Jun 2015 19:21:01 GMT

    [ https://issues.apache.org/jira/browse/HDFS-7923?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14575058#comment-14575058

Colin Patrick McCabe commented on HDFS-7923:

bq. Should the checkLease logs be done to the blockLog? We log the startup error log there
in processReport

I like having the ability to turn on and off TRACE logging for various subsystems.  Putting
everything in the blockLog would make that harder, right?

bq. Update javadoc in BlockReportContext with what leaseID is for.


bq. Add something to the log message about overwriting the old leaseID in offerService. Agree
that this shouldn't really trigger, but good defensive coding practice 


bq. DatanodeManager, there's still a register/unregister in registerDatanode I think we could
skip. This is the node restart case where it's registered previously.

Good catch.  The calls in {{removeDatanode}} and {{addDatanode}} should take care of this,
so there's no need to have it here.

bq. BRLManager requestLease, we auto-register the node on requestLease. This shouldn't happen
since DNs need to register before doing anything else. We can keep this here

I added a warn message since this shouldn't happen.

bq. Still need documentation of new config keys in hdfs-default.xml


bq. Extra import in TestBPSAScheduler and BPSA


bq. If you want to pursue \[the block report timing\] logic change more, let's split it out
into a follow-on JIRA. The rest LGTM, +1 pending above comments.

OK, I will restore the old behavior for now, and we can do this in a follow-on change.

> The DataNodes should rate-limit their full block reports by asking the NN on heartbeat
> -----------------------------------------------------------------------------------------------
>                 Key: HDFS-7923
>                 URL: https://issues.apache.org/jira/browse/HDFS-7923
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>    Affects Versions: 2.8.0
>            Reporter: Colin Patrick McCabe
>            Assignee: Colin Patrick McCabe
>         Attachments: HDFS-7923.000.patch, HDFS-7923.001.patch, HDFS-7923.002.patch, HDFS-7923.003.patch,
> The DataNodes should rate-limit their full block reports.  They can do this by first
sending a heartbeat message to the NN with an optional boolean set which requests permission
to send a full block report.  If the NN responds with another optional boolean set, the DN
will send an FBR... if not, it will wait until later.  This can be done compatibly with optional

This message was sent by Atlassian JIRA

View raw message