hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mukul Kumar Singh (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDDS-638) enable ratis snapshots for HDDS datanodes
Date Fri, 19 Oct 2018 16:32:00 GMT

    [ https://issues.apache.org/jira/browse/HDDS-638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16657038#comment-16657038
] 

Mukul Kumar Singh commented on HDDS-638:
----------------------------------------

Thanks for taking a look at the patch [~jnp].

With this patch, the snapshot file generated for TermIndex (t:1, i:4) is of the name sm/snapshot.1_4.
So the snapshot file is unique each time, Also as part of Raft code, ServerState#initStatemachine,
the last applied index is read from the term index, which is inturn formed from the file name.

The following log snippet explain the file names and recovery on restart.
{code}
2018-10-17 21:39:19,532 INFO  ratis.ContainerStateMachine (ContainerStateMachine.java:takeSnapshot(195))
- Taking snapshot at termIndex:(t:1, i:4)
2018-10-17 21:39:19,534 INFO  ratis.ContainerStateMachine (ContainerStateMachine.java:takeSnapshot(198))
- Taking a snapshot to file /Users/msingh/code/apache/ozone/ozone_review/hadoop-ozone/tools/target/test-dir/MiniOzoneClusterImpl-65253a3a-c5ca-4e5e-b7b5-4ed4bcc92c99/datanode-0/data/ratis/a85420d0-71d2-497b-8890-b0689bf55a6e/sm/snapshot.1_4
2018-10-17 21:39:19,534 INFO  impl.RaftServerImpl (ServerState.java:close(381)) - 84aca8dc-41f8-4a21-a957-8d2cdd51d75e
closes. The last applied log index is 4
{code}

After restart, the 4 is the loaded index.
{code}
2018-10-17 21:39:26,535 INFO  ratis.ContainerStateMachine (ContainerStateMachine.java:addRequest(452))
- Setting the last applied index to 4
{code}

> enable ratis snapshots for HDDS datanodes
> -----------------------------------------
>
>                 Key: HDDS-638
>                 URL: https://issues.apache.org/jira/browse/HDDS-638
>             Project: Hadoop Distributed Data Store
>          Issue Type: Bug
>          Components: Ozone Datanode
>    Affects Versions: 0.3.0
>            Reporter: Mukul Kumar Singh
>            Assignee: Mukul Kumar Singh
>            Priority: Blocker
>         Attachments: HDDS-638.001.patch
>
>
> Currently on a restart, a hdds datanode, starts applying log entries from the start of
the log.
> This should can be avoided by taking a ratis snapshot to persist the last stable state
and on restart the datanodes start applying log from that index.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message