hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Andrew Wang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-7979) Initialize block report IDs with a random number
Date Tue, 24 Mar 2015 20:20:53 GMT

    [ https://issues.apache.org/jira/browse/HDFS-7979?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14378510#comment-14378510

Andrew Wang commented on HDFS-7979:

Sorry if I wasn't clear about this, but "safety" here is about not repeating. The monotonic
time is the time since DN startup, so if the machine goes down and comes back up, it'll be
in the same range of monotonic times. Agree that it doesn't reset if just the DN process is

bq. When using the monotonic time, two block reports cannot get the same ID since the monotonic
time is always increasing. We don't have the same guarantee here. Admittedly, the chances
of a repeat are extremely low. But previously they were effectively 0, and now they're nonzero.

This seems less likely to repeat compared to monotonic time, because of machine reboots.

bq. If the datanode is taken down and restarted, the monotonic time will still be higher than
before. And so the current behavior makes it easy to see from the logs that block report N+1
came after block report N, even if there was a datanode restart in between. We don't have
this behavior with a random number generated on datanode start.

This is true, but not sure how useful this is. If there's logging on both the DN and NN, that
would also be a way of determining ordering.

> Initialize block report IDs with a random number
> ------------------------------------------------
>                 Key: HDFS-7979
>                 URL: https://issues.apache.org/jira/browse/HDFS-7979
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: datanode
>    Affects Versions: 2.7.0
>            Reporter: Andrew Wang
>            Assignee: Andrew Wang
>            Priority: Minor
>         Attachments: HDFS-7979.001.patch
> Right now block report IDs use system nanotime. This isn't that random, so let's start
it at a random number for some more safety.

This message was sent by Atlassian JIRA

View raw message