hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Anu Engineer (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-11154) Block Storage : store server state to persistent storage
Date Mon, 21 Nov 2016 22:01:59 GMT

    [ https://issues.apache.org/jira/browse/HDFS-11154?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15684862#comment-15684862

Anu Engineer commented on HDFS-11154:

bq. The operations calling into StorageManager are already syncd on StorageManager's own lock.
The next patch will include locks for those operations on the levelDB instance.

Perhaps I am mistaken, but let me walk you through an extreme scenario.  This is more of an
exercise to make sure that my understanding is correct.

Client Thread 1: Calls into DeleteVolume , executes storageManager.deleteVolume ; thread gets

Client Thread 2: Calls into createVolume, with the same username and volume name -- Since
thread1 has executed storageManager.deleteVolume, storageManager.createVolume is successful.
Thread 2 goes to execute writeToPresistentStore, we return success to the caller.

Thread 1 wakes up, continues the deleteCall by removeFromPersistentStore, but in this case
the Database Key is from createVolume call from the Thread2 and not from the createVolume
call that you had when you called deleteVolume. 

So the client gets two success calls, but in some strange cases the volume will not exist.
In other words, there seems to be a race condition.

> Block Storage : store server state to persistent storage
> --------------------------------------------------------
>                 Key: HDFS-11154
>                 URL: https://issues.apache.org/jira/browse/HDFS-11154
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: hdfs
>            Reporter: Chen Liang
>            Assignee: Chen Liang
>         Attachments: HDFS-11154-HDFS-7240.001.patch, HDFS-11154-HDFS-7240.002.patch,
> Currently, all the storage state are kept in server memory. If server crashes, we would
lose all the volume information. This JIRA stores server internal state into its local disk.
Such that on server failure, we can simply restart server and restore volume information from
> More specifically, the internal state written to disk is mainly the mapping from volume
to its underlying containers, plus some meta information such as volume size, block size,
> Note that this is only a simple, minimum set mechanism for persistence. It is more like
a counterpart of fsimage in HDFS, but without edit logs.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org

View raw message