Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 77FBA200CBD for ; Thu, 6 Jul 2017 16:09:02 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id 76676166742; Thu, 6 Jul 2017 14:09:02 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 946B8166740 for ; Thu, 6 Jul 2017 16:09:01 +0200 (CEST) Received: (qmail 79875 invoked by uid 500); 6 Jul 2017 14:08:59 -0000 Mailing-List: contact user-help@spark.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list user@spark.apache.org Received: (qmail 79865 invoked by uid 99); 6 Jul 2017 14:08:59 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 06 Jul 2017 14:08:59 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id EFD6CC1F79 for ; Thu, 6 Jul 2017 14:08:58 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.299 X-Spam-Level: * X-Spam-Status: No, score=1.299 tagged_above=-999 required=6.31 tests=[HTML_MESSAGE=2, RCVD_IN_DNSWL_LOW=-0.7, SPF_PASS=-0.001] autolearn=disabled Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id jMfI8GWx7T_y for ; Thu, 6 Jul 2017 14:08:56 +0000 (UTC) Received: from us-smtp-delivery-102.mimecast.com (us-smtp-delivery-102.mimecast.com [216.205.24.102]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id 18D685F6C2 for ; Thu, 6 Jul 2017 14:08:55 +0000 (UTC) Received: from CAS080-CO-3.exch080.serverpod.net (cas080-co-3.exch080.serverdata.net [199.193.204.150]) (Using TLS) by us-smtp-1.mimecast.com with ESMTP id us-mta-77-24YMPm72OUqoV6DvvUhIXg-1; Thu, 06 Jul 2017 10:08:54 -0400 Received: from MBX080-W4-CO-2.exch080.serverpod.net (10.224.117.102) by MBX080-W4-CO-2.exch080.serverpod.net (10.224.117.102) with Microsoft SMTP Server (TLS) id 15.0.1178.4; Thu, 6 Jul 2017 07:08:52 -0700 Received: from MBX080-W4-CO-2.exch080.serverpod.net ([10.224.117.102]) by mbx080-w4-co-2.exch080.serverpod.net ([10.224.117.102]) with mapi id 15.00.1178.000; Thu, 6 Jul 2017 07:08:52 -0700 From: Steve Loughran To: Vadim Semenov CC: "everett@nuna.com" , user Subject: Re: Spark, S3A, and 503 SlowDown / rate limit issues Thread-Topic: Spark, S3A, and 503 SlowDown / rate limit issues Thread-Index: AQHS8RpEmpRVgFYrU0ypL4DVUPu2IaJFu5UAgAGaVoA= Date: Thu, 6 Jul 2017 14:08:51 +0000 Message-ID: References: In-Reply-To: Accept-Language: en-GB, en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-ms-exchange-messagesentrepresentingtype: 1 x-ms-exchange-transport-fromentityheader: Hosted x-source-routing-agent: Processed MIME-Version: 1.0 X-MC-Unique: 24YMPm72OUqoV6DvvUhIXg-1 Content-Type: multipart/alternative; boundary="_000_E16A829AAEF24A558C9CB531AC978414hortonworkscom_" archived-at: Thu, 06 Jul 2017 14:09:02 -0000 --_000_E16A829AAEF24A558C9CB531AC978414hortonworkscom_ Content-Type: text/plain; charset=WINDOWS-1252 Content-Transfer-Encoding: quoted-printable On 5 Jul 2017, at 14:40, Vadim Semenov > wrote: Are you sure that you use S3A? Because EMR says that they do not support S3A https://aws.amazon.com/premiumsupport/knowledge-center/emr-file-system-s3/ > Amazon EMR does not currently support use of the Apache Hadoop S3A file s= ystem. I think that the HEAD requests come from the `createBucketIfNotExists` in t= he AWS S3 library that checks if the bucket exists every time you do a PUT = request, i.e. creates a HEAD request. You can disable that by setting `fs.s3.buckets.create.enabled` to `false` http://docs.aws.amazon.com/emr/latest/ManagementGuide/emr-plan-upload-s3.ht= ml Yeah, I'd like to see the stack traces before blaming S3A and the ASF codeb= ase One thing I do know is that the shipping S3A client doesn't have any explic= it handling of 503/retry events. I know that: https://issues.apache.org/jir= a/browse/HADOOP-14531 There is some retry logic in bits of the AWS SDK related to file upload: th= at may log and retry, but in all the operations listing files, getting thei= r details, etc: no resilience to throttling. If it is surfacing against s3a, there isn't anything which can immediately = be done to fix it, other than "spread your data around more buckets". Do at= tach the stack trace you get under https://issues.apache.org/jira/browse/HA= DOOP-14381 though: I'm about half-way through the resilience code (& fault = injection needed to test it). The more where I can see problems arise, the = more confident I can be that those codepaths will be resilient. On Thu, Jun 29, 2017 at 4:56 PM, Everett Anderson > wrote: Hi, We're using Spark 2.0.2 + Hadoop 2.7.3 on AWS EMR with S3A for direct I/O f= rom/to S3 from our Spark jobs. We set mapreduce.fileoutputcommitter.algorit= hm.version=3D2 and are using encrypted S3 buckets. This has been working fine for us, but perhaps as we've been running more j= obs in parallel, we've started getting errors like Status Code: 503, AWS Service: Amazon S3, AWS Request ID: ..., AWS Error Co= de: SlowDown, AWS Error Message: Please reduce your request rate., S3 Exten= ded Request ID: ... We enabled CloudWatch S3 request metrics for one of our buckets and I was a= little alarmed to see spikes of over 800k S3 requests over a minute or so,= with the bulk of them HEAD requests. We read and write Parquet files, and most tables have around 50 shards/part= s, though some have up to 200. I imagine there's additional parallelism whe= n reading a shard in Parquet, though. Has anyone else encountered this? How did you solve it? I'd sure prefer to avoid copying all our data in and out of HDFS for each j= ob, if possible. Thanks! --_000_E16A829AAEF24A558C9CB531AC978414hortonworkscom_ Content-Type: text/html; charset=WINDOWS-1252 Content-ID: Content-Transfer-Encoding: quoted-printable
On 5 Jul 2017, at 14:40, Vadim Semenov <vadim.semenov@datadoghq.com&g= t; wrote:

Are you sure that you use S3A?
Because EMR says that they do not support S3A 

https://aws.amazon.com/premiumsuppor= t/knowledge-center/emr-file-system-s3/
> Amazon EMR does not currently support use of the Apache Hadoop S3A file sy= stem.

I think that the HEAD requests come from the `createBucketIfNotExist= s` in the AWS S3 library that checks if the bucket exists every time you do a PUT request, i.e. creates a HEAD request= .

You can disable that by setting `fs.s3.buckets.create.= enabled` to `false` 
http://docs.aws.amazon.com/emr/la= test/ManagementGuide/emr-plan-upload-s3.html



Yeah, I'd like to see the stack traces before blaming S3A and the ASF = codebase

One thing I do know is that the shipping S3A client doesn't have any e= xplicit handling of 503/retry events. I know that: https://issues.apach= e.org/jira/browse/HADOOP-14531

There is some retry logic in bits of the AWS SDK related to file uploa= d: that may log and retry, but in all the operations listing files, getting= their details, etc: no resilience to throttling.

If it is surfacing against s3a, there isn't anything which can immedia= tely be done to fix it, other than "spread your data around more bucke= ts". Do attach the stack trace you get under https://issues.apache= .org/jira/browse/HADOOP-14381 though: I'm about half-way through the resilience code (& fault injection need= ed to test it). The more where I can see problems arise, the more confident= I can be that those codepaths will be resilient.


On Thu, Jun 29, 2017 at 4:56 PM, Everett Anderso= n <everett@nuna.com.invalid> wrote:
Hi,

We're using Spark 2.0.2 + Hadoop 2.7.3 on AWS EMR with = S3A for direct I/O from/to S3 from our Spark jobs. We set mapreduce.fileoutputcommitter.algorithm.version=3D2 and= are using encrypted S3 buckets.

This has been working fine for us, but perhaps as we've bee= n running more jobs in parallel, we've started getting errors like

Status Code:= 503, AWS Service: Amazon S3, AWS Request ID: ..., AWS Error Code: SlowDown= , AWS Error Message: Please reduce your request rate., S3 Extended Request = ID: ...

We enabled CloudWatch S3 request metrics for one of our buc= kets and I was a little alarmed to see spikes of over 800k S3 requests over= a minute or so, with the bulk of them HEAD requests.

We read and write Parquet files, and most tables have aroun= d 50 shards/parts, though some have up to 200. I imagine there's additional= parallelism when reading a shard in Parquet, though.

Has anyone else encountered this? How did you solve it?

I'd sure prefer to avoid copying all our data in and out of= HDFS for each job, if possible.

Thanks!



--_000_E16A829AAEF24A558C9CB531AC978414hortonworkscom_--