hadoop-hdfs-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hudson (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HDDS-898) Continue token should contain the previous dir in Ozone s3g object list
Date Thu, 17 Jan 2019 01:52:00 GMT

    [ https://issues.apache.org/jira/browse/HDDS-898?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16744601#comment-16744601

Hudson commented on HDDS-898:

SUCCESS: Integrated in Jenkins build Hadoop-trunk-Commit #15782 (See [https://builds.apache.org/job/Hadoop-trunk-Commit/15782/])
HDDS-898. Continue token should contain the previous dir in Ozone s3g (bharat: rev 54b11de2c0254eb0b8dc45dc9ce67758697df846)
* (edit) hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/endpoint/BucketEndpoint.java
* (add) hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/util/TestContinueToken.java
* (edit) hadoop-ozone/s3gateway/src/test/java/org/apache/hadoop/ozone/s3/endpoint/TestBucketGet.java
* (add) hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/util/ContinueToken.java
* (edit) hadoop-ozone/s3gateway/src/main/java/org/apache/hadoop/ozone/s3/util/S3utils.java

> Continue token should contain the previous dir in Ozone s3g object list
> -----------------------------------------------------------------------
>                 Key: HDDS-898
>                 URL: https://issues.apache.org/jira/browse/HDDS-898
>             Project: Hadoop Distributed Data Store
>          Issue Type: Sub-task
>          Components: S3
>    Affects Versions: 0.3.0
>            Reporter: Elek, Marton
>            Assignee: Elek, Marton
>            Priority: Major
>             Fix For: 0.4.0
>         Attachments: HDDS-898.001.patch, HDDS-898.002.patch, HDDS-898.003.patch
> Let's imagine we have the following keys:
> test/dir1/file1
> test/dir2/file1
> test/dir2/file2
> test/dir3/file1
> With the object list endpoint (separator=/) we will return with a list where the directories
are also added (they are added during the iteration):
> *test/dir1 (directory/prefix entry)
> *test/dir2 (directory/prefix entry)
> *test/dir3 (directory/prefix entry)
> test/dir3/file1
> Now limit the results to 2:
> First call:
> test/dir1/file1 --> this should be added to the results as dir1/ (first result)
> test/dir2/file1 --> this should be added to the restuls as dir2/ (second result)
> the iteration can be continued from key: test/dir2/file2
> Second call, with continue token, continue the iteration from test/dir2/file2
> test/dir2/file2 --> this will be added as dir2 (!!! duplicate here !!!) as we have
no information if it has already been added or not
> test/dir3/file1 --> will be dir3
> Summary: we don't know if the dynamic dir entry is already added or not.
> Solution: we can add this information to the encoded continue token and decode it at
the next iteration.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: hdfs-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: hdfs-issues-help@hadoop.apache.org

View raw message