hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-15076) Enhance s3a troubleshooting docs, add perf section
Date Fri, 12 Jan 2018 11:40:00 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-15076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16323851#comment-16323851
] 

Steve Loughran commented on HADOOP-15076:
-----------------------------------------

thanks: always good to have feedback,

I'll take on all the perf stuff

w.r.t the "heavyhanded" change in troubleshooting, yes it is that blunt. You can look at all
the JIRAs off the previous S3a changes and see that something invariably fails, including
changed APIs stopping linking working. I know minor diffs are more nuanced, but I don't want
to encourage people to go that way, as even point releases change things (HADOOP-14283, HADOOP-13050,
HADOOP-13044)

Saying  "a point release may work" isn't valid unless we say "a point release may update but
you can't be sure without running all the hadoop-aws test suite with s3guard and scale enabled
then build downstream projects and run any tests you have to push them against s3 to see that
all is well". 

I'm trying to make clear that no, JAR-swap-debugging isn't going to do any good. Which I believe,
based on experience, is generally the case.

> Enhance s3a troubleshooting docs, add perf section
> --------------------------------------------------
>
>                 Key: HADOOP-15076
>                 URL: https://issues.apache.org/jira/browse/HADOOP-15076
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: documentation, fs/s3
>    Affects Versions: 2.8.2
>            Reporter: Steve Loughran
>            Assignee: Abraham Fine
>         Attachments: HADOOP-15076-001.patch, HADOOP-15076-002.patch, HADOOP-15076-003.patch,
HADOOP-15076-004.patch
>
>
> A recurrent theme in s3a-related JIRAs, support calls etc is "tried upgrading the AWS
SDK JAR and then I got the error ...". We know here "don't do that", but its not something
immediately obvious to lots of downstream users who want to be able to drop in the new JAR
to fix things/add new features
> We need to spell this out quite clearlyi "you cannot safely expect to do this. If you
want to upgrade the SDK, you will need to rebuild the whole of hadoop-aws with the maven POM
updated to the latest version, ideally rerunning all the tests to make sure something hasn't
broken. 
> Maybe near the top of the index.md file, along with "never share your AWS credentials
with anyone"



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message