hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Arpit Agarwal (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-6931) Move lazily persisted replicas to finalized directory on DN startup
Date Thu, 28 Aug 2014 04:37:57 GMT

    [ https://issues.apache.org/jira/browse/HDFS-6931?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14113329#comment-14113329

Arpit Agarwal commented on HDFS-6931:

On restart each volume will be scanned and replicas under {{lazyPersist/}} will be moved to
their corresponding locations under {{finalized/}}.

We may end up with two replicas of the same block on different volumes, so we use the following
scheme to decide which replica to keep. See {{BlockPoolSlice.resolveDuplicateReplicas}}.

   * Given two replicas, decide which one to keep. The preference is as
   * follows:
   *   1. Prefer the replica with the higher generation stamp.
   *   2. If generation stamps are equal, prefer the replica with the
   *      larger on-disk length.
   *   3. If on-disk length is the same, prefer the replica on persistent
   *      storage volume.
   *   4. All other factors being equal, keep replica1.
   * The other replica is removed from the volumeMap and is deleted from
   * its storage volume.

Also added test cases.

> Move lazily persisted replicas to finalized directory on DN startup
> -------------------------------------------------------------------
>                 Key: HDFS-6931
>                 URL: https://issues.apache.org/jira/browse/HDFS-6931
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: datanode
>            Reporter: Arpit Agarwal
>            Assignee: Arpit Agarwal
>             Fix For: HDFS-6581
>         Attachments: HDFS-6931.01.patch
> On restart the DN should move replicas from the {{lazyPersist}} directory to {{finalized}},
and evict/unlink replicas of the same block from RAM disk.

This message was sent by Atlassian JIRA

View raw message