hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Arpit Agarwal (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HDFS-6978) Directory scanner should correctly reconcile blocks on RAM disk
Date Fri, 29 Aug 2014 21:45:53 GMT

     [ https://issues.apache.org/jira/browse/HDFS-6978?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Arpit Agarwal updated HDFS-6978:
--------------------------------

    Description: 
It used to be very unlikely that the directory scanner encountered two replicas of the same
block on different volumes.

With memory storage, it is very likely to hit this with the following sequence of events:
# Block is written to RAM disk
# Lazy writer saves a copy on persistent volume
# DN attempts to evict the original replica from RAM disk, file deletion fails as the replica
is in use.
# Directory scanner finds a replica on both RAM disk and persistent storage.

The directory scanner should never delete the block on persistent storage.

  was:
Currently it would be a very unlikely situation for the directory scanner to encounter two
replicas of the same block on different volumes. Hence the scanner does not handle this situation
predictable.

With memory storage, it is common to save blocks to persistent storage and eviction of the
original RAM disk block could fail since the block is in use. In this situation, the directory
scanner should never delete the block on persistent storage.


> Directory scanner should correctly reconcile blocks on RAM disk
> ---------------------------------------------------------------
>
>                 Key: HDFS-6978
>                 URL: https://issues.apache.org/jira/browse/HDFS-6978
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: datanode
>    Affects Versions: HDFS-6581
>            Reporter: Arpit Agarwal
>            Assignee: Arpit Agarwal
>
> It used to be very unlikely that the directory scanner encountered two replicas of the
same block on different volumes.
> With memory storage, it is very likely to hit this with the following sequence of events:
> # Block is written to RAM disk
> # Lazy writer saves a copy on persistent volume
> # DN attempts to evict the original replica from RAM disk, file deletion fails as the
replica is in use.
> # Directory scanner finds a replica on both RAM disk and persistent storage.
> The directory scanner should never delete the block on persistent storage.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message