hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Weiwei Yang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDFS-11782) Ozone: KSM: Add listKey
Date Sat, 10 Jun 2017 14:58:18 GMT

    [ https://issues.apache.org/jira/browse/HDFS-11782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16045562#comment-16045562
] 

Weiwei Yang commented on HDFS-11782:
------------------------------------

Hi [~linyiqun]

Thanks for working on this, the patch overall looks good to me. But there is one thing I want
to double check with you

According to the [design doc|https://issues.apache.org/jira/secure/attachment/12799549/ozone_user_v0.pdf],
the listKey returns a list of keys, each key contains following info

# version
# md5hash
# createdOn
# size
# keyName
# dataFileName

this is encapsulated in class {{org.apache.hadoop.ozone.web.response.KeyInfo}}, this is the
correct return value for {{DistributedStorageHandler#listKeys()}} in your patch, however the
info seems incomplete. KSM in your patch returns a list of {{org.apache.hadoop.ksm.helpers.KsmKeyInfo}},
that was added by the patch for {{putKey}}, it only has info required to for addKey operation,

# volumeName
# bucketName
# keyName
# containerName
# blockID
# dataSize
# shouldCreateContainer

it doesn't have persisted key info. This is not a problem with your patch, which I think this
is because {{putKey}} was not fully implemented. Right now {{putKey}} directly updates the
KSM meta store with {{KsmKeyInfo}}, so KSM meta store doesn't have complete key info. To me,
it seems the step of COMMIT a key is missing. After a key object write succeed, it needs to
commit a key to KSM namespace with the {{KeyInfo}}, including the timestamp it created, md5hash,
size etc, just like what discussed in HDFS-11886. So I think we properly need to get HDFS-11886
done before getting this one completely work, or track the remain work in HDFS-11886. What
do you think [~linyiqun] and [~anu] ?

Some other nits

# *KeyManager*: java doc warning line 76 incorrect param name
# *KeySpaceManagerProtocol*: java doc warning line 195 incorrect param name

Thank you.

> Ozone: KSM: Add listKey
> -----------------------
>
>                 Key: HDFS-11782
>                 URL: https://issues.apache.org/jira/browse/HDFS-11782
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: HDFS-7240
>    Affects Versions: ozone
>            Reporter: Anu Engineer
>            Assignee: Yiqun Lin
>         Attachments: HDFS-11782-HDFS-7240.001.patch
>
>
> Add support for listing keys in a bucket. Just like other 2 list operations, this API
supports paging via, prevKey, prefix and maxKeys.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message