hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Elek, Marton (JIRA)" <j...@apache.org>
Subject [jira] [Created] (HDDS-91) Calculate under/over replicated containers from the container reports
Date Sun, 20 May 2018 07:26:00 GMT
Elek, Marton created HDDS-91:

             Summary: Calculate under/over replicated containers from the container reports
                 Key: HDDS-91
                 URL: https://issues.apache.org/jira/browse/HDDS-91
             Project: Hadoop Distributed Data Store
          Issue Type: Bug
          Components: SCM
            Reporter: Elek, Marton
            Assignee: Elek, Marton
             Fix For: 0.2.1

In the current InProgressPool we calculate the existing replica numbers for all the containers
based on the container reports. But we don't do anything in case of missing replicase.

This patch is the initial step to process the reported data by comparing the reported replica
numbers with the state saved in the Mapping database.

I prerefer to do smaller patches instead of one big one, so this patch doesn't solve over/under
replcation the problem yet just detect it.

1. It integrates the EventQueue with the scm and makes it available to the ContainerSupervisor
(constructor + field changes)

2. In finalizeReconciliation it sends events to compare expected and current replicase (expected
replicas are from the ContainerMapping)

3. Will send a new event in case of under/over replication.

Further works are needed to react to the new events and send delete/copy container commands
to the datanode. It also requires more information about the current in-progress replication:
If we alread asked a new datanode to replicate the container we need to save it to a map to
make the call idempotent: on the next container replication we should not request an other
replication. I would prefer to put this additional information to the ContainerMapping instead
of a new map.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org

View raw message