hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Weiwei Yang (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (HDFS-11569) Ozone: Implement listKey function for KeyManager
Date Thu, 06 Apr 2017 09:19:41 GMT

    [ https://issues.apache.org/jira/browse/HDFS-11569?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15958233#comment-15958233
] 

Weiwei Yang edited comment on HDFS-11569 at 4/6/17 9:18 AM:
------------------------------------------------------------

Hi [~anu]

Thanks for your comments, apparently I was missing the handler part, thanks for pointing this
out. Like you mentioned, I don't think this jira will implement {{DistributedStorageHandler#listKeys}},
because the list key operation is supposed to route to KSM first, then get container location
from SCM. However KSM is not ready yet,  so lets use this jira to track the work of the container
layer.

About the pagination, you are making a good point. It looks better to simply honor  the arguments
{{prefix}}, {{prevKey}} and {{maxNumOfKeys}}.  The default value for max number of keys is
1000 to avoid returning too many entries a time. Works like following

# Client makes a listKey request to ozone front end
# Ozone handler, i.e {{DistributedStorageHandler}}, handles the request, it gets the arguments
from {{ListArgs}}
# Ozone handler looks up container locations from {{KSM}} where a range of keys reside.
# Ozone handler reads keys from those containers via {{KeyManager}} interface.
# Ozone handler merges results from multiple containers and return that to client.

This jira only addresses #4, rest of will need to be implemented when KSM is ready. Please
let me know if this approach looks good to you or not. Thank you.


was (Author: cheersyang):
Hi [~anu]

Thanks for your comments, apparently I missed the web handler part. I will implement this.
And I noticed the deleteKey function has been implemented in {{KeyManagerImpl}}, but not yet
implemented in {{DistributedStorageHandler}}, we need to get that done too (maybe in another
jira), right?

About the pagination, you are making a good point. It looks better to simply honor  the arguments
{{prefix}}, {{prevKey}} and {{maxKeys}}, send them to container layer and return desired set
of keys. That means we do not need pagination in server side, instead we let client side to
request proper size of results. And we set {{maxKeys}} a default value 1000. Please let me
know if this approach looks good to you or not. Thank you.

> Ozone: Implement listKey function for KeyManager
> ------------------------------------------------
>
>                 Key: HDFS-11569
>                 URL: https://issues.apache.org/jira/browse/HDFS-11569
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: ozone
>            Reporter: Weiwei Yang
>            Assignee: Weiwei Yang
>         Attachments: HDFS-11569-HDFS-7240.001.patch, HDFS-11569-HDFS-7240.002.patch,
HDFS-11569-HDFS-7240.003.patch, HDFS-11569-HDFS-7240.004.patch, HDFS-11569-HDFS-7240.005.patch
>
>
> List keys by prefix from a container. This will need to support pagination for the purpose
of small object support. So the listKey function returns something like ListKeyResult, client
can iterate the object to get pagination results.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message