hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kai Zheng (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-13768) AliyunOSS: handle deleteDirs reliably when too many objects to delete
Date Thu, 03 Nov 2016 02:19:58 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-13768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15631263#comment-15631263
] 

Kai Zheng commented on HADOOP-13768:
------------------------------------

Looking at the following codes:
{code}
+    List<String> deleteFailed = keysToDelete;
+    while(CollectionUtils.isNotEmpty(deleteFailed)) {
+      List<String> l = new ArrayList<>();
+      List<List<String>> smallerLists = Lists.partition(deleteFailed, 1000);
+      for (List<String> smallerList : smallerLists) {
+        DeleteObjectsRequest deleteRequest =
+            new DeleteObjectsRequest(bucketName);
+        deleteRequest.setKeys(smallerList);
+        deleteRequest.setQuiet(true);
+        DeleteObjectsResult result = ossClient.deleteObjects(deleteRequest);
+        l.addAll(result.getDeletedObjects());
+      }
+      deleteFailed = l;
+      tries++;
+      if (tries == retry) {
+        break;
+      }
+    }
{code}

1. Please give {{l}} a more readable name.
2. Can you give some comments to explain some bit about the procedure? I (probably others)
wouldn't know why it's like that without querying the SDK's manual. I know it now, there're
2 modes in the {{ossClient.deleteObjects}} operation, one mode to return successfully deleted
objects, and the other returning the deleting-failed objects. You're using the latter, and
use a loop to try some times to delete and delete the failed-to-delete objects.

> AliyunOSS: handle deleteDirs reliably when too many objects to delete
> ---------------------------------------------------------------------
>
>                 Key: HADOOP-13768
>                 URL: https://issues.apache.org/jira/browse/HADOOP-13768
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: fs
>    Affects Versions: 3.0.0-alpha2
>            Reporter: Genmao Yu
>            Assignee: Genmao Yu
>             Fix For: 3.0.0-alpha2
>
>         Attachments: HADOOP-13768.001.patch, HADOOP-13768.002.patch
>
>
> Note in Aliyun OSS SDK, DeleteObjectsRequest has 1000 objects limit. This needs to improve
{{deleteDirs}} operation to make it pass when more objects than the limit to delete.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message