hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Weiwei Yang (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (HDFS-12506) Ozone: ListBucket is too slow
Date Mon, 25 Sep 2017 08:21:00 GMT

    [ https://issues.apache.org/jira/browse/HDFS-12506?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16178712#comment-16178712
] 

Weiwei Yang edited comment on HDFS-12506 at 9/25/17 8:20 AM:
-------------------------------------------------------------

The javadoc warnings seems to be a false alarm, it doesn't understand doc reference

{code}
/** 
 * {@link #getRangeKVs(byte[], int, MetadataKeyFilter...)}
 **/
{code}

which works perfectly on my intellji IDE. I manually run {{mvn javadoc:javadoc}}, I got following
warnings

{noformat}
[WARNING] /Users/yangwwei/IdeaProjects/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/ozone/container/common/transport/server/ratis/ContainerStateMachine.java:45:
warning - Tag @link: reference not found: org.apache.ratis.statemachine.StateMachine
[WARNING] /Users/yangwwei/IdeaProjects/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/checker/AbstractFuture.java:1274:
warning - Tag @link: can't find newDirectExecutorService() in org.apache.hadoop.hdfs.server.datanode.checker.AbstractFuture
[WARNING] /Users/yangwwei/IdeaProjects/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/checker/AbstractFuture.java:1274:
warning - Tag @link: reference not found: CallerRunsPolicy
[WARNING] /Users/yangwwei/IdeaProjects/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/ozone/container/common/transport/server/XceiverServerHandler.java:75:
warning - Tag @link: reference not found: ChannelHandlerContext#fireExceptionCaught(Throwable)
[WARNING] /Users/yangwwei/IdeaProjects/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/ozone/protocolPB/KeySpaceManagerProtocolServerSideTranslatorPB.java:95:
warning - Tag @link: reference not found: org.apache.hadoop.ozone.ksm.protocolPB.KeySpaceManagerProtocolPB
[WARNING] /Users/yangwwei/IdeaProjects/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/ozone/ksm/BucketManager.java:76:
warning - Tag @link: reference not found: KsmBucketInfo
[WARNING] /Users/yangwwei/IdeaProjects/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/ozone/ksm/BucketManager.java:76:
warning - Tag @link: reference not found: KsmBucketInfo
[WARNING] /Users/yangwwei/IdeaProjects/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/ozone/ksm/KSMMetadataManager.java:164:
warning - Tag @link: reference not found: KsmBucketInfo
[WARNING] /Users/yangwwei/IdeaProjects/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/ozone/ksm/KSMMetadataManager.java:164:
warning - Tag @link: reference not found: KsmBucketInfo
[WARNING] /Users/yangwwei/IdeaProjects/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/ozone/ksm/KeyManager.java:101:
warning - Tag @link: reference not found: KsmKeyInfo
[WARNING] /Users/yangwwei/IdeaProjects/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/ozone/ksm/KSMMetadataManager.java:188:
warning - Tag @link: reference not found: KsmKeyInfo
[WARNING] /Users/yangwwei/IdeaProjects/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/ozone/web/netty/RequestContentObjectStoreChannelHandler.java:42:
warning - Tag @link: reference not found: HttpContent
[WARNING] /Users/yangwwei/IdeaProjects/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/ozone/web/netty/RequestDispatchObjectStoreChannelHandler.java:44:
warning - Tag @link: reference not found: HttpRequest
{noformat}

It looks like javadoc plugin doesn't recognize these links. I think we can commit this, this
2 links are useful to help people know the difference of two methods. [~linyiqun], any thought
on the java doc warnings?


was (Author: cheersyang):
The javadoc warnings seems to be a false alarm, it doesn't understand doc reference

{code}
/** 
 * {@link #getRangeKVs(byte[], int, MetadataKeyFilter...)}
 **/
{code}

which works perfectly on my intellji IDE. I manually run {{mvn javadoc:javadoc}}, I got following
warnings

{noformat}
[WARNING] /Users/yangwwei/IdeaProjects/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/ozone/container/common/transport/server/ratis/ContainerStateMachine.java:45:
warning - Tag @link: reference not found: org.apache.ratis.statemachine.StateMachine
[WARNING] /Users/yangwwei/IdeaProjects/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/checker/AbstractFuture.java:1274:
warning - Tag @link: can't find newDirectExecutorService() in org.apache.hadoop.hdfs.server.datanode.checker.AbstractFuture
[WARNING] /Users/yangwwei/IdeaProjects/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/datanode/checker/AbstractFuture.java:1274:
warning - Tag @link: reference not found: CallerRunsPolicy
[WARNING] /Users/yangwwei/IdeaProjects/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/ozone/container/common/transport/server/XceiverServerHandler.java:75:
warning - Tag @link: reference not found: ChannelHandlerContext#fireExceptionCaught(Throwable)
[WARNING] /Users/yangwwei/IdeaProjects/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/ozone/protocolPB/KeySpaceManagerProtocolServerSideTranslatorPB.java:95:
warning - Tag @link: reference not found: org.apache.hadoop.ozone.ksm.protocolPB.KeySpaceManagerProtocolPB
[WARNING] /Users/yangwwei/IdeaProjects/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/ozone/ksm/BucketManager.java:76:
warning - Tag @link: reference not found: KsmBucketInfo
[WARNING] /Users/yangwwei/IdeaProjects/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/ozone/ksm/BucketManager.java:76:
warning - Tag @link: reference not found: KsmBucketInfo
[WARNING] /Users/yangwwei/IdeaProjects/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/ozone/ksm/KSMMetadataManager.java:164:
warning - Tag @link: reference not found: KsmBucketInfo
[WARNING] /Users/yangwwei/IdeaProjects/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/ozone/ksm/KSMMetadataManager.java:164:
warning - Tag @link: reference not found: KsmBucketInfo
[WARNING] /Users/yangwwei/IdeaProjects/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/ozone/ksm/KeyManager.java:101:
warning - Tag @link: reference not found: KsmKeyInfo
[WARNING] /Users/yangwwei/IdeaProjects/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/ozone/ksm/KSMMetadataManager.java:188:
warning - Tag @link: reference not found: KsmKeyInfo
[WARNING] /Users/yangwwei/IdeaProjects/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/ozone/web/netty/RequestContentObjectStoreChannelHandler.java:42:
warning - Tag @link: reference not found: HttpContent
[WARNING] /Users/yangwwei/IdeaProjects/hadoop/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/ozone/web/netty/RequestDispatchObjectStoreChannelHandler.java:44:
warning - Tag @link: reference not found: HttpRequest
{noformat}

It looks like javadoc plugin doesn't recognize these links. I think we can commit this, this
2 links are useful to help people know the difference of two methods.

> Ozone: ListBucket is too slow
> -----------------------------
>
>                 Key: HDFS-12506
>                 URL: https://issues.apache.org/jira/browse/HDFS-12506
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: ozone
>            Reporter: Weiwei Yang
>            Assignee: Weiwei Yang
>            Priority: Blocker
>              Labels: ozoneMerge, performance
>         Attachments: HDFS-12506-HDFS-7240.001.patch, HDFS-12506-HDFS-7240.002.patch,
HDFS-12506-HDFS-7240.003.patch, HDFS-12506-HDFS-7240.004.patch, HDFS-12506-HDFS-7240.005.patch,
HDFS-12506-HDFS-7240.006.patch, HDFS-12506-HDFS-7240.007.patch
>
>
> Generated 3 million keys in ozone, and run {{listBucket}} command to get a list of buckets
under a volume,
> {code}
> bin/hdfs oz -listBucket http://15oz1.fyre.ibm.com:9864/vol-0-15143 -user wwei
> {code}
> this call spent over *15 seconds* to finish. The problem was caused by the inflexible
structure of KSM DB. Right now {{ksm.db}} stores keys like following
> {code}
> /v1/b1
> /v1/b1/k1
> /v1/b1/k2
> /v1/b1/k3
> /v1/b2
> /v1/b2/k1
> /v1/b2/k2
> /v1/b2/k3
> /v1/b3
> /v1/b4
> {code}
> keys are sorted in nature order so when we do list buckets under a volume e.g /v1, we
need to seek to /v1 point and start to iterate and filter keys, this ends up with scanning
all keys under volume /v1. The problem with this design is we don't have an efficient approach
to locate all buckets without scanning the keys.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org


Mime
View raw message