hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mukul Kumar Singh (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDDS-638) enable ratis snapshots for HDDS datanodes
Date Fri, 19 Oct 2018 16:32:00 GMT

    [ https://issues.apache.org/jira/browse/HDDS-638?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16657038#comment-16657038

Mukul Kumar Singh commented on HDDS-638:

Thanks for taking a look at the patch [~jnp].

With this patch, the snapshot file generated for TermIndex (t:1, i:4) is of the name sm/snapshot.1_4.
So the snapshot file is unique each time, Also as part of Raft code, ServerState#initStatemachine,
the last applied index is read from the term index, which is inturn formed from the file name.

The following log snippet explain the file names and recovery on restart.
2018-10-17 21:39:19,532 INFO  ratis.ContainerStateMachine (ContainerStateMachine.java:takeSnapshot(195))
- Taking snapshot at termIndex:(t:1, i:4)
2018-10-17 21:39:19,534 INFO  ratis.ContainerStateMachine (ContainerStateMachine.java:takeSnapshot(198))
- Taking a snapshot to file /Users/msingh/code/apache/ozone/ozone_review/hadoop-ozone/tools/target/test-dir/MiniOzoneClusterImpl-65253a3a-c5ca-4e5e-b7b5-4ed4bcc92c99/datanode-0/data/ratis/a85420d0-71d2-497b-8890-b0689bf55a6e/sm/snapshot.1_4
2018-10-17 21:39:19,534 INFO  impl.RaftServerImpl (ServerState.java:close(381)) - 84aca8dc-41f8-4a21-a957-8d2cdd51d75e
closes. The last applied log index is 4

After restart, the 4 is the loaded index.
2018-10-17 21:39:26,535 INFO  ratis.ContainerStateMachine (ContainerStateMachine.java:addRequest(452))
- Setting the last applied index to 4

> enable ratis snapshots for HDDS datanodes
> -----------------------------------------
>                 Key: HDDS-638
>                 URL: https://issues.apache.org/jira/browse/HDDS-638
>             Project: Hadoop Distributed Data Store
>          Issue Type: Bug
>          Components: Ozone Datanode
>    Affects Versions: 0.3.0
>            Reporter: Mukul Kumar Singh
>            Assignee: Mukul Kumar Singh
>            Priority: Blocker
>         Attachments: HDDS-638.001.patch
> Currently on a restart, a hdds datanode, starts applying log entries from the start of
the log.
> This should can be avoided by taking a ratis snapshot to persist the last stable state
and on restart the datanodes start applying log from that index.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org

View raw message