flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ken Krugler (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (FLINK-7477) Use "hadoop classpath" to augment classpath when available
Date Wed, 31 Jan 2018 22:11:01 GMT

    [ https://issues.apache.org/jira/browse/FLINK-7477?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16347685#comment-16347685

Ken Krugler commented on FLINK-7477:

I posted to the mailing list about an issue that this change seemed to create for me, but
didn't hear back.
{quote}With Flink 1.4 and FLINK-7477, I ran into a problem with jar versions for HttpCore,
when using the AWS SDK to read from S3.
I believe the issue is that even when setting classloader.resolve-order to child-first in
flink-conf.yaml, the change to put all jars returned by “hadoop classpath” on the classpath
means that classes in these jars are found before the classes in my shaded Flink uber jar.
If I ensure that I don’t have the “hadoop” command set up on my Bash path, then I don’t
run into this issue.
Does this make sense, or is there something else going on that I can fix to avoid this situation?{quote}
Any input? Thanks...Ken

> Use "hadoop classpath" to augment classpath when available
> ----------------------------------------------------------
>                 Key: FLINK-7477
>                 URL: https://issues.apache.org/jira/browse/FLINK-7477
>             Project: Flink
>          Issue Type: Bug
>          Components: Startup Shell Scripts
>            Reporter: Aljoscha Krettek
>            Assignee: Aljoscha Krettek
>            Priority: Major
>             Fix For: 1.4.0
> Currently, some cloud environments don't properly put the Hadoop jars into {{HADOOP_CLASSPATH}}
(or don't set {{HADOOP_CLASSPATH}}) at all. We should check in {{config.sh}} if the {{hadoop}}
binary is on the path and augment our {{INTERNAL_HADOOP_CLASSPATHS}} with the result of {{hadoop
classpath}} in our scripts.
> This will improve the out-of-box experience of users that otherwise have to manually

This message was sent by Atlassian JIRA

View raw message