Mailing-List: contact common-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: common-issues@hadoop.apache.org
Date: Tue, 30 Sep 2014 16:54:34 +0000 (UTC)
From: "Charles Lamb (JIRA)" <jira@apache.org>
To: common-issues@hadoop.apache.org
Message-ID: <JIRA.12721679.1403011947000.156296.1412096074679@Atlassian.JIRA>
In-Reply-To: <JIRA.12721679.1403011947000@Atlassian.JIRA>
References: <JIRA.12721679.1403011947000@Atlassian.JIRA>
 <JIRA.12721679.1403011947632@arcas>
Subject: [jira] [Commented] (HADOOP-10714) AmazonS3Client.deleteObjects()
 need to be limited to 1000 entries per call
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable


    [ https://issues.apache.org/jira/browse/HADOOP-10714?page=3Dcom.atlassi=
an.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=3D14=
153394#comment-14153394 ]=20

Charles Lamb commented on HADOOP-10714:
---------------------------------------

bq. charles =E2=80=94you happy with this?

Yes, +1 (non-binding) from me.


> AmazonS3Client.deleteObjects() need to be limited to 1000 entries per cal=
l
> -------------------------------------------------------------------------=
-
>
>                 Key: HADOOP-10714
>                 URL: https://issues.apache.org/jira/browse/HADOOP-10714
>             Project: Hadoop Common
>          Issue Type: Bug
>          Components: fs/s3
>    Affects Versions: 2.5.0
>            Reporter: David S. Wang
>            Assignee: Juan Yu
>            Priority: Critical
>              Labels: s3
>         Attachments: HADOOP-10714-007.patch, HADOOP-10714-1.patch, HADOOP=
-10714.001.patch, HADOOP-10714.002.patch, HADOOP-10714.003.patch, HADOOP-10=
714.004.patch, HADOOP-10714.005.patch, HADOOP-10714.006.patch
>
>
> In the patch for HADOOP-10400, calls to AmazonS3Client.deleteObjects() ne=
ed to have the number of entries at 1000 or below. Otherwise we get a Malfo=
rmed XML error similar to:
> com.amazonaws.services.s3.model.AmazonS3Exception: Status Code: 400, AWS =
Service: Amazon S3, AWS Request ID: 6626AD56A3C76F5B, AWS Error Code: Malfo=
rmedXML, AWS Error Message: The XML you provided was not well-formed or did=
 not validate against our published schema, S3 Extended Request ID: DOt6C+Y=
84mGSoDuaQTCo33893VaoKGEVC3y1k2zFIQRm+AJkFH2mTyrDgnykSL+v
> at com.amazonaws.http.AmazonHttpClient.handleErrorResponse(AmazonHttpClie=
nt.java:798)
> at com.amazonaws.http.AmazonHttpClient.executeHelper(AmazonHttpClient.jav=
a:421)
> at com.amazonaws.http.AmazonHttpClient.execute(AmazonHttpClient.java:232)
> at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:35=
28)
> at com.amazonaws.services.s3.AmazonS3Client.invoke(AmazonS3Client.java:34=
80)
> at com.amazonaws.services.s3.AmazonS3Client.deleteObjects(AmazonS3Client.=
java:1739)
> at org.apache.hadoop.fs.s3a.S3AFileSystem.rename(S3AFileSystem.java:388)
> at org.apache.hadoop.hbase.snapshot.ExportSnapshot.run(ExportSnapshot.jav=
a:829)
> at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
> at org.apache.hadoop.hbase.snapshot.ExportSnapshot.innerMain(ExportSnapsh=
ot.java:874)
> at org.apache.hadoop.hbase.snapshot.ExportSnapshot.main(ExportSnapshot.ja=
va:878)
> Note that this is mentioned in the AWS documentation:
> http://docs.aws.amazon.com/AmazonS3/latest/API/multiobjectdeleteapi.html
> "The Multi-Object Delete request contains a list of up to 1000 keys that =
you want to delete. In the XML, you provide the object key names, and optio=
nally, version IDs if you want to delete a specific version of the object f=
rom a versioning-enabled bucket. For each key, Amazon S3=E2=80=A6.=E2=80=9D
> Thanks to Matteo Bertozzi and Rahul Bhartia from AWS for identifying the =
problem.


--
This message was sent by Atlassian JIRA
(v6.3.4#6332)