hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Anu Engineer (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-12387) Ozone: Support Ratis as a first class replication mechanism
Date Thu, 07 Sep 2017 20:40:01 GMT

    [ https://issues.apache.org/jira/browse/HDFS-12387?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16157609#comment-16157609

Anu Engineer commented on HDFS-12387:

[~nandakumar131] Thanks for your valuable comments.

bq. Case 1) If another client does an allocateBlock call in between step 1 and 2, we will
return AllocatedBlock with create container flag set to true. This will cause two clients
to trigger create container call to same pipeline and one will result in CONTAINER_EXISTS
Completely agree, we will need to add the code in the client that says if we get this exception
we should continue or another option that we have considered is to add a lease manager which
keeps a lease on the container that is being given out to a client for creation. But I completely
agree with your point this needs to be fixed and it is skipped in this patch for time being.
Thanks for flagging it.

bq. Case 2) If container creation call (step 3) fails with an exception, we will never revert
the container state from CREATING to ALLOCATED.
You are right again, the idea -- again missing code -- this patch is already 153 KB, is that
if we get a timeout, we will move to the state of delete this container.  We do need to define
the time out and the container delete path.We will rely on the delete path that [~cheersyang]
is building to do that. Hence it is missing in this patch. Just to make sure we are on the
same page, once a container is given out to a client for CREATING state, the state machine
will wait for some time, say 30 mins, if the creation does happen in that duration then that
container will get deleted both from SCM and Datandoes. The code for the time keeper will
part of the lease manager that we discussed above. 

So case 2, case 3, all are handled by this catch all meachnism. When we have the lease manager
( I might just update the patch, since you have already finished reading it once, adding a
bit more code would not hurt) then the time based state can be handled. So if we time out,
we will delete the container both from SCM and datanodes.

bq. No check is done before setting the pipeline replication type as ReplicationType.STAND_ALONE

Thanks for catching that, i will fix it in the next patch.

> Ozone: Support Ratis as a first class replication mechanism
> -----------------------------------------------------------
>                 Key: HDFS-12387
>                 URL: https://issues.apache.org/jira/browse/HDFS-12387
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: ozone
>    Affects Versions: HDFS-7240
>            Reporter: Anu Engineer
>            Assignee: Anu Engineer
>            Priority: Critical
>              Labels: ozoneMerge
>         Attachments: HDFS-12387-HDFS-7240.001.patch
> Ozone container layer supports pluggable replication policies. This JIRA brings Apache
Ratis based replication to Ozone.  Apache Ratis is a java implementation of Raft protocol.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org

View raw message