hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Nanda kumar (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDDS-868) Handle quasi closed container replicas in SCM
Date Mon, 26 Nov 2018 10:10:00 GMT

    [ https://issues.apache.org/jira/browse/HDDS-868?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16698702#comment-16698702

Nanda kumar commented on HDDS-868:

Thanks [~msingh] & [~jnp] for the review. Addressed the review comments in patch v001.
{quote}ContainerInfo.java:192, lets add couple of checks while updating the sequenceId, a)
the new sequence id is always greater than the previously stored sequence id and b) for a
closed container, the sequence id is never changed.
Checks have been added.
{quote}ContainerStateManager.java:168, a quasi close container can only be moved to close
on a force close, lets add this to the diagram.
Done. Also added {{HddsProtos.LifeCycleEvent.FORCE_CLOSE}}
{quote}ReportHandler.java, For deleting, deleted and default, lets log and throw and exception
for now.
{quote}fixContainerReplica can be named as sendReplicaCommands, as the action taken here is
about re-sending replica commands
{quote}IncrementalContainerReportHandler and ContainerReportHandler can share the same code
for one replica, via updateContainerState.
{quote}Is sequenceId same as BCS? It could be different for some replicas for certain node
failure scenarios.
Yes, sequenceId and BCS are the same. Until we close the container the sequenceId can change
and we will update it only if the ID reported by the replica is greater than the ID known
to SCM. Once the container is closed, sequenceId cannot be updated.
{quote}containerInfo.setUsedBytes and containerInfo.setNumberOfKeys: Do they track maximum
number of bytes and keys reported by any replica; different replicas would report different
Yes, different replicas can report different values. We update these values in SCM only if
a replica reports a higher value than the one which SCM already saw.

> Handle quasi closed container replicas in SCM
> ---------------------------------------------
>                 Key: HDDS-868
>                 URL: https://issues.apache.org/jira/browse/HDDS-868
>             Project: Hadoop Distributed Data Store
>          Issue Type: Improvement
>          Components: SCM
>            Reporter: Nanda kumar
>            Assignee: Nanda kumar
>            Priority: Major
>         Attachments: HDDS-868.000.patch, HDDS-868.001.patch
> In case of pipeline failure the containers will be quais closed by datanode. SCM has
to understand that the container replica is quasi closed and based on the block commit sequence
Id SCM should identify the latest replica and force close them now.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org

View raw message