hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-15076) Enhance s3a troubleshooting docs, add perf section
Date Fri, 12 Jan 2018 17:56:00 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-15076?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16324290#comment-16324290

Steve Loughran commented on HADOOP-15076:

bq.  Isn't the commit protocol being slow part of "performance"? Can this be rephrased?

more "people complain think their jobs slow", when they should be worrying that the data may
actually be invalid. That is, yes, slow is the performance, but the real problem is that commit
algorithms based on atomic renames don't work if renames aren't atomic, even if they are fast.

Attached Patch 5

* intro has a table
* moved commit section down past I/O
* fadvise options reordered
* added a bit on good code too
* spell check

> Enhance s3a troubleshooting docs, add perf section
> --------------------------------------------------
>                 Key: HADOOP-15076
>                 URL: https://issues.apache.org/jira/browse/HADOOP-15076
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: documentation, fs/s3
>    Affects Versions: 2.8.2
>            Reporter: Steve Loughran
>            Assignee: Steve Loughran
>         Attachments: HADOOP-15076-001.patch, HADOOP-15076-002.patch, HADOOP-15076-003.patch,
HADOOP-15076-004.patch, HADOOP-15076-005.patch
> A recurrent theme in s3a-related JIRAs, support calls etc is "tried upgrading the AWS
SDK JAR and then I got the error ...". We know here "don't do that", but its not something
immediately obvious to lots of downstream users who want to be able to drop in the new JAR
to fix things/add new features
> We need to spell this out quite clearlyi "you cannot safely expect to do this. If you
want to upgrade the SDK, you will need to rebuild the whole of hadoop-aws with the maven POM
updated to the latest version, ideally rerunning all the tests to make sure something hasn't
> Maybe near the top of the index.md file, along with "never share your AWS credentials
with anyone"

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org

View raw message