flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Kai Chen (Jira)" <j...@apache.org>
Subject [jira] [Updated] (FLINK-23449) YarnTaskExecutorRunner does not contains MapReduce classes
Date Wed, 21 Jul 2021 11:29:00 GMT

     [ https://issues.apache.org/jira/browse/FLINK-23449?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Kai Chen updated FLINK-23449:
-----------------------------
    Description: 
I followed instructions described in [https://ci.apache.org/projects/flink/flink-docs-release-1.11/dev/table/hive] and
tested hive streaming sink, met this exception  

      Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.mapred.JobConf

[http://apache-flink.147419.n8.nabble.com/Flink-td7866.html] met the same problem.

 

I checked TM jvm envs and the code and found that flink only set up YARN_APPLICATION_CLASSPATH,
but without MAPREDUCE_APPLICATION_CLASSPATH.

See: [https://github.com/apache/flink/blob/ed39fb2efc790af038c1babd4a48847b7b39f91e/flink-yarn/src/main/java/org/apache/flink/yarn/Utils.java#L119]

 

I think we should add MAPREDUCE_APPLICATION_CLASSPATH as well, as the same as spark does.

  was:
I followed instructions described in [https://ci.apache.org/projects/flink/flink-docs-release-1.11/dev/table/hive] and
tested hive streaming sink, met this exception  

      Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.mapred.JobConf

[http://apache-flink.147419.n8.nabble.com/Flink-td7866.html] met the same problem.

 

I checked TM jvm envs and the code and found that flink only set up YARN_APPLICATION_CLASSPATH,
but without MAPREDUCE_APPLICATION_CLASSPATH.

See: [https://github.com/apache/flink/blob/ed39fb2efc790af038c1babd4a48847b7b39f91e/flink-yarn/src/main/java/org/apache/flink/yarn/Utils.java#L119]

 

I think we should add MAPREDUCE_APPLICATION_CLASSPATH as well, as the same as spark does。


> YarnTaskExecutorRunner does  not contains MapReduce classes
> -----------------------------------------------------------
>
>                 Key: FLINK-23449
>                 URL: https://issues.apache.org/jira/browse/FLINK-23449
>             Project: Flink
>          Issue Type: Bug
>          Components: Deployment / YARN
>    Affects Versions: 1.11.3
>         Environment: flink-1.11
> flink on yarn cluster
> jdk1.8
> hive1.2.1
> hadoop2.7
> hadoop classes is provided with  {{export HADOOP_CLASSPATH=`hadoop classpath` when
submitting test APP. (described in [https://ci.apache.org/projects/flink/flink-docs-release-1.11/ops/deployment/hadoop.html] )}}
> {{}}
>            Reporter: Kai Chen
>            Priority: Major
>
> I followed instructions described in [https://ci.apache.org/projects/flink/flink-docs-release-1.11/dev/table/hive] and
tested hive streaming sink, met this exception  
>       Caused by: java.lang.ClassNotFoundException: org.apache.hadoop.mapred.JobConf
> [http://apache-flink.147419.n8.nabble.com/Flink-td7866.html] met the same problem.
>  
> I checked TM jvm envs and the code and found that flink only set up YARN_APPLICATION_CLASSPATH,
but without MAPREDUCE_APPLICATION_CLASSPATH.
> See: [https://github.com/apache/flink/blob/ed39fb2efc790af038c1babd4a48847b7b39f91e/flink-yarn/src/main/java/org/apache/flink/yarn/Utils.java#L119]
>  
> I think we should add MAPREDUCE_APPLICATION_CLASSPATH as well, as the same as spark does.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Mime
View raw message