hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From GitBox <...@apache.org>
Subject [GitHub] [hadoop] steveloughran opened a new pull request #1826: HADOOP-16823. Manage S3 Throttling exclusively in S3A client.
Date Fri, 31 Jan 2020 11:48:02 GMT
steveloughran opened a new pull request #1826: HADOOP-16823. Manage S3 Throttling exclusively
in S3A client. 
URL: https://github.com/apache/hadoop/pull/1826
 
 
   Currently AWS S3 throttling is initially handled in the AWS SDK, only reaching the S3 client
code after it has given up.
   
   This means we don't always directly observe when throttling is taking place.
   
   * disables throttling retries in the AWS client Library *for S3 only*
   * add a quantile for the S3 throttle events, as DDB has
   * isolate counters of s3 and DDB throttle events to classify issues better
   * improvements to DDB throttling handling and testing
   
   1. Because we are taking over the AWS retries, we need to expand the initial delay en retries
and the number of retries we should support before giving up.
   1. I can split the DDB and S3 side of this patch...they came in together as once I turned
off throttling across all AWS client configs, scale tests against a 10 TPS DDB table showed
we weren't retrying adequately in some of the tests, and retrying inefficiently in listChildren.
   
   
   (reinstatment of #1814 which was accidentally closed)
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message