hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chen Liang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-11780) Ozone: KSM : Add putKey
Date Thu, 25 May 2017 19:00:07 GMT

    [ https://issues.apache.org/jira/browse/HDFS-11780?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16025228#comment-16025228

Chen Liang commented on HDFS-11780:

Thanks [~cheersyang] for the comments!

bq. why KsmKeyBlock has only a containerName but not a PipeLine
The reason is that, by design, maintaining the mapping from container name to actual container
instance should be the responsibility of SCM, while KSM should not be supposed to understand
the internal of container implementation. Also, Another important thing is that, {{KsmKeyBlock}}
(now KsmKeyInfo) is written to ksm.db for getKey(), keeping one {{Pipeline}} instance for
every single key can be too expensive, when storing just the container name is sufficient

Nonetheless, a downside of storing just the name rather than Pipeline is that, client side
needs to make another call to SCM to look up Pipeline given container name. I think this can
be optimized by changing XceiverClientManager to cache container connections by containerName
instead of Pipeline. Will probably follow up in another JIRA.

bq. And I don't understand the comment...

Sorry for the confusion, will elaborate in next path... What I was trying to say is that this
call to KSM is only making sure a block is allocated, but putKey is not complete yet. After
this allocated block gets returned to client side, client side {{DistributedStorageHandler}}
will make another call to datanode to actually writes the data. So KSM is in some sense making
the assumption that {{DistributedStorageHandler}} will complete the writes successfully. In
the case that the block is allocated on KSM and returned to {{DistributedStorageHandler}},
but {{DistributedStorageHandler}} failed the following writes for any reason, we may end up
having an entry in ksm.db, but no such key in that container.

(Also I thought allocateKeyBlock has been changed to allocateKey, but seems it's not... will
update also)

> Ozone: KSM : Add putKey
> -----------------------
>                 Key: HDFS-11780
>                 URL: https://issues.apache.org/jira/browse/HDFS-11780
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: ozone
>            Reporter: Anu Engineer
>            Assignee: Chen Liang
>         Attachments: HDFS-11780-HDFS-7240.001.patch, HDFS-11780-HDFS-7240.002.patch,
HDFS-11780-HDFS-7240.003.patch, HDFS-11780-HDFS-7240.004.patch
> Support putting a key into an Ozone bucket. 

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org

View raw message