hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-15642) Update to latest/recent version of aws-sdk
Date Mon, 30 Jul 2018 21:23:00 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-15642?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16562525#comment-16562525
] 

Steve Loughran commented on HADOOP-15642:
-----------------------------------------

Proposed process for SDK update, something I'd add to the testing.md file along with an intro
"updating the AWS SDK is non-trivial and can take a while to bed in"

# plan this a few weeks before are release, to allow for (a) time to see what the issues are,
(b) time to roll back and (c) possibly time to get any big issues fixed in the AWS Release
cycle.
# increase {{aws-java-sdk.version}} to the new version
# rebuild, retest
# distinguish test failures from our code being wrong (repair) vs incompatible AWS SDK changes
(file)
# look at logs of generated test runs. Are new warning messages appearing? Stack traces? 
# build hadoop CLI, perform manual operations. Again, does everything "look" right
# build and run any downstream tests you have to hand
# do a mvn dependency dump of hadoop-aws: has the new shaded SDK forgotten to shade everything?
If so: file issue with AWS; exclude this dependency (HADOOP-15264)

Policy w.r.t deprecated items
* it depends. Treat as a hint more than a demand, but do consider why the old version is being
deprecated.

Policy w.r.t backporting SDK updates
* Updates allowed for branch-2 and trunk
* No to <=2.8.x (unshaded, separate JAR era)
* Maybe to 2.9.x, 3.0.x, 3.1.x, if there are significant security implications or other major
bug fixes.



> Update to latest/recent version of aws-sdk
> ------------------------------------------
>
>                 Key: HADOOP-15642
>                 URL: https://issues.apache.org/jira/browse/HADOOP-15642
>             Project: Hadoop Common
>          Issue Type: Sub-task
>          Components: build, fs/s3
>    Affects Versions: 3.2.0
>            Reporter: Steve Loughran
>            Priority: Blocker
>         Attachments: Screen Shot 2018-07-30 at 14.11.22.png
>
>
> Move to a later version of the AWS SDK library for a different set of features and issues.
> One thing which doesn't work on the one we ship with is the ability to create assumed
role sessions >1h, as there's a check in the client lib for role-duration <= 3600 seconds.
I'll assume more recent SDKs delegate duration checks to the far end.
> see: [https://aws.amazon.com/about-aws/whats-new/2018/03/longer-role-sessions/]
> * assuming later versions will extend assumed role life, docs will need changing, 
> * Adding a test in HADOOP-15583 which expects an error message if you ask for a duration
of 3h; this should act as the test to see what happens.
> * think this time would be good to explicitly write down the SDK update process



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org


Mime
View raw message