hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HADOOP-15808) Harden Token service loader use
Date Wed, 28 Nov 2018 11:20:02 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-15808?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16701734#comment-16701734

Steve Loughran commented on HADOOP-15808:

Revisited this to see about changing message for token loading.
I'm actually reluctant to do this because of the following classpath isn't as rare as you

* hadoop-aws on CP, aws-sdk not. 

We saw that with the FileSystem service loader, logging warnings created too many stack traces,
issues related to "why am I being told about s3 when I'm working with HDFS, etc".

I would like to keep this noise out. Now, it will hide the issue "there's an implementation
declared which isn't going to load", but if you are really looking for a token, the fact it
won't load will soon surface. The problem is that we've lost some of the root cause. Really
you'd want to cache the stack and only print it in a specific failure, e.g when a token can't
be decoded because its kind is unrecognised. 

But I can't see an easy way to do that with the current code.

the one thing we could do is cache the fact that 1+ warning was found and then in {{Token.decodeIdentifier()}},
when there's no match for a token kind, log that fact.

The problem I have is
* this stuff gets used in a lot of places
* its currently fairly brittle code: one bad class and *nothing* will load.
* A patch to really harden this stuff with null checks everywhere didn't take

I'm confident that the current design is going to fail badly once HADOOP-14556 goes in. I've
looked to see if I can get away with making sure the token identifier has no indirect refs
to any AWS libraries, but it's a doomed exercise: no way to test, and it'll inevitably regress.

> Harden Token service loader use
> -------------------------------
>                 Key: HADOOP-15808
>                 URL: https://issues.apache.org/jira/browse/HADOOP-15808
>             Project: Hadoop Common
>          Issue Type: Improvement
>          Components: security
>    Affects Versions: 2.9.1, 3.1.2
>            Reporter: Steve Loughran
>            Assignee: Steve Loughran
>            Priority: Major
>         Attachments: HADOOP-15808-001.patch, HADOOP-15808-002.patch, HADOOP-15808-003.patch
> The Hadoop token service loading (identifiers, renewers...) works provided there's no
problems loading any registered implementation. If there's a classloading or classcasting
problem, the exception raised will stop all token support working; possibly the application
not starting.
> This matters for S3A/HADOOP-14556 as things may not load if aws-sdk isn't on the classpath.
It probably lurks in the wasb/abfs support too, but things have worked there because the installations
with DT support there have always had correctly set up classpaths.
> Fix: do what we did for the FS service loader. Catch failures to instantiate a service
provider impl and skip it

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org

View raw message