hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Anu Engineer (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (HDFS-11926) Ozone: Implement a common helper to return a range of KVs in levelDB
Date Tue, 06 Jun 2017 18:11:18 GMT

    [ https://issues.apache.org/jira/browse/HDFS-11926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16039281#comment-16039281
] 

Anu Engineer edited comment on HDFS-11926 at 6/6/17 6:11 PM:
-------------------------------------------------------------

[~cheersyang] Thanks for the update, it looks much better. Three questions / Comments.

# if count is 0 or less than zero, I think we should just throw an IllegalArgumentException.
If there is a magic semantics of translating a -1 to max int, it should be done in the caller
function, not in a generic helper. My point is that, this semantics might not be applicable
to all listing operations and we should just keep what is truly generic in this function.
# We do a DbIter.seekToFirst -- shouldn't we do that if and only if the startKey == null ?
It looks like it is a frivolous operation if startKey argument is not null.
# Even though I did suggest that we should throw if cannot find the startKey, We also need
a plan to handle the situation where someone is iterating a bucket with concurrent deletes
going on, if we always throw then user will not be able to make any forward progress(For this
patch, I am good with this, we should file a tracking JIRA if we are committing this). So
we should tell user how to back-off, that is take the last listing output and try the previous
key, or restart from the last successful startKey or retry the operations. In other words,
we need to cleanly define the semantics and usage of what the user needs to do if and when
they encounter this error.

One more minor suggestion, Would it be possible to log the time taken to execute this function,
I wanted to make sure this does not degrade then there are lots of writers present. You don't
need to test it, but just keep logging the time taken -- if you want,  feel free to write
this as a debug call or return the time taken and keep a metric.




was (Author: anu):
[~cheersyang] Thanks for the update, it looks much better. Three questions / Comments.

# if count is 0 or less than zero, I think we should just throw an IllegalArgumentException.
If there is a magic semantics of translating a -1 to max int, it should be done in the caller
function, not in a generic helper. My point is that, this semantics might not be applicable
to all listing operations and we should just keep what is truly generic in this function.
# We do a DbIter.seekToFirst -- shouldn't we do that if and only if the startKey == null ?
It looks like it is a frivolous operation if startKey argument is not null.
# Even though I did suggest that we should throw if cannot find the startKey, We also need
a plan to handle the situation where someone is iterating a bucket with concurrent deletes
going on, if we always throw then user will not be able to make any forward progress(For this
patch, I am good with this, we should file a tracking JIRA if we are committing this). So
we should tell user how to back-off, that is take the last listing output and try the previous
key, or restart from the last successful startKey or retry the operations. In other words,
we need to cleanly define the semantics and usage of what the user needs to do if and when
they encounter this error.


> Ozone: Implement a common helper to return a range of KVs in levelDB
> --------------------------------------------------------------------
>
>                 Key: HDFS-11926
>                 URL: https://issues.apache.org/jira/browse/HDFS-11926
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: ozone
>            Reporter: Weiwei Yang
>            Assignee: Weiwei Yang
>            Priority: Blocker
>         Attachments: HDFS-11926-HDFS-7240.001.patch, HDFS-11926-HDFS-7240.002.patch,
HDFS-11926-HDFS-7240.003.patch
>
>
> There are quite some *LIST* operations need to get a range of keys or values from levelDB,
and filter entries with key prefix. 
> # HDFS-11782 listKeys
> # HDFS-11779 listBuckets
> # HDFS-11773 listVolumes
> # HDFS-11679 listContainers
> we need to implement a common utility for them.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message