From user-return-33825-archive-asf-public=cust-asf.ponee.io@flink.apache.org Tue Mar 31 06:46:55 2020 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [207.244.88.153]) by mx-eu-01.ponee.io (Postfix) with SMTP id F2E04180181 for ; Tue, 31 Mar 2020 08:46:54 +0200 (CEST) Received: (qmail 27310 invoked by uid 500); 31 Mar 2020 06:46:53 -0000 Mailing-List: contact user-help@flink.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list user@flink.apache.org Received: (qmail 27300 invoked by uid 99); 31 Mar 2020 06:46:53 -0000 Received: from Unknown (HELO mailrelay1-lw-us.apache.org) (10.10.3.159) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 31 Mar 2020 06:46:53 +0000 Received: from mail-oi1-f170.google.com (mail-oi1-f170.google.com [209.85.167.170]) by mailrelay1-lw-us.apache.org (ASF Mail Server at mailrelay1-lw-us.apache.org) with ESMTPSA id DE155F64 for ; Tue, 31 Mar 2020 06:46:52 +0000 (UTC) Received: by mail-oi1-f170.google.com with SMTP id m14so18081153oic.0 for ; Mon, 30 Mar 2020 23:46:52 -0700 (PDT) X-Gm-Message-State: ANhLgQ1p8A/z1Ln5lTo1C24N29tuts6wqRqCc6a8utIJ5RN/GObH7wDJ HUpLQPoDzcOQWJmpbXDgpRYgnKzDZbfqIaZJ6s8= X-Google-Smtp-Source: ADFU+vsrqkBOyCyRb2O0Isyq0Tw8gODl1nwr+nIW7RK9MDCumLGWx92iLNbXSwEAWdBYEEufL5VRFKWcwuu3CTPRluk= X-Received: by 2002:aca:cd0e:: with SMTP id d14mr1022080oig.167.1585637212261; Mon, 30 Mar 2020 23:46:52 -0700 (PDT) MIME-Version: 1.0 References: In-Reply-To: From: Robert Metzger Date: Tue, 31 Mar 2020 08:46:36 +0200 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: Log file environment variable 'log.file' is not set. To: Vitaliy Semochkin Cc: user Content-Type: multipart/alternative; boundary="000000000000c79cd805a220ee60" --000000000000c79cd805a220ee60 Content-Type: text/plain; charset="UTF-8" Hey Vitaliy, Check this documentation on how to use Flink with Hadoop: https://ci.apache.org/projects/flink/flink-docs-release-1.10/ops/deployment/hadoop.html For your setup, I would recommend referencing the Hadoop jars from your Hadoop vendor by setting export HADOOP_CLASSPATH=`hadoop classpath` Is it possible that the files on your cluster are Flink 1.7.0 files, while your Flink job maven project has Flink 1.10 dependencies? On your server, what version do the flink jar files in lib/ have? If your are launching Flink like this... ./bin/flink run -m yarn-cluster -p 4 -yjm 1024m -ytm 4096m ./my/project.jar ... it will use the files in lib/ for starting Flink. Best, Robert On Mon, Mar 30, 2020 at 5:39 PM Vitaliy Semochkin wrote: > Hello Robert, > Thank you for quick response! > Indeed logs says the hadoop version is 2.4.1 this is probably because of > > https://github.com/apache/flink/blob/b17a597dec80e590db2beedda446aa3cae9920dd/pom.xml#L96 > How can I make 1.10 to work with my current hadoop version? > > Regarding flink reporting in logs its 1.7.0 > org.apache.flink.runtime.entrypoint.ClusterEntrypoint - Starting > YarnJobClusterEntrypoint (Version: 1.7.0 > while I'm using 1.10 and this is application cluster (everything is > bundled and we don't have session cluster running). > Here are the whole dependencies list: > mvn dependency:tree | grep flink | cut -d'-' -f2- > org.apache.flink:flink-yarn_2.11:jar:1.10.0:runtime > org.apache.flink:flink-clients_2.11:jar:1.10.0:compile > org.apache.flink:flink-optimizer_2.11:jar:1.10.0:compile > org.apache.flink:flink-shaded-hadoop-2:jar:2.4.1-9.0:runtime > org.apache.flink:force-shading:jar:1.10.0:compile > org.apache.flink:flink-runtime_2.11:jar:1.10.0:runtime > org.apache.flink:flink-core:jar:1.10.0:compile > org.apache.flink:flink-annotations:jar:1.10.0:compile > org.apache.flink:flink-metrics-core:jar:1.10.0:compile > org.apache.flink:flink-java:jar:1.10.0:compile > org.apache.flink:flink-queryable-state-client-java:jar:1.10.0:runtime > org.apache.flink:flink-hadoop-fs:jar:1.10.0:runtime > org.apache.flink:flink-shaded-netty:jar:4.1.39.Final-9.0:compile > org.apache.flink:flink-shaded-guava:jar:18.0-9.0:compile > org.apache.flink:flink-shaded-asm-7:jar:7.1-9.0:compile > org.apache.flink:flink-shaded-jackson:jar:2.10.1-9.0:compile > org.apache.flink:flink-jdbc_2.11:jar:1.10.0:compile > org.apache.flink:flink-hbase_2.11:jar:1.10.0:compile > org.apache.flink:flink-runtime-web_2.11:jar:1.10.0:compile > As you can see all flink related libs are 1.10. > > Can you please tell which class in flinks identifies the version(I'll try > to debug it locally)? > > Regards, > Vitaliy > > > On Mon, Mar 30, 2020 at 5:10 PM Robert Metzger > wrote: > >> Hey Vitaliy, >> is it okay for you if we keep the discussion on the list, so that others >> can chime in to help, and that Google can index the conversation, in case >> somebody else has a similar problem? >> >> I just checked, and Flink on YARN in Flink 1.10 does set the >> property correctly. Maybe in Flink 1.7, accessing the logs in the web ui >> was not yet supported. >> >> You said in your email, that you are using Flink 1.10, however, your logs >> state that you are running Flink 1.7.0. >> It also seems that you have the Hadoop 2.4.1 dependencies of Flink, but >> your Hadoop environment is Hadoop 2.7.3. I believe this error is caused by >> that version mismatch: >> >> Caused by: java.lang.IllegalAccessError: tried to access method >>> org.apache.hadoop.yarn.client.ConfiguredRMFailoverProxyProvider.getProxyInternal()Ljava/lang/Object; >>> from class >>> org.apache.hadoop.yarn.client.RequestHedgingRMFailoverProxyProvider >> >> >> >> >> On Mon, Mar 30, 2020 at 1:58 PM Vitaliy Semochkin >> wrote: >> >>> Hello Robert, >>> >Where exactly are you seeing the "Log file environment variable >>> 'log.file' is not set." message? >>> I get this message when I check yarn logs. >>> >>> >Can you post some context around it? (is this shown from the command >>> line? what are the arguments? is it shown in a file? >>> I'm creating an application cluster from a java application using flink >>> 1.10. I didn't have this issue with 1.8.1 version, >>> however when I upgraded configuration slightly changed, e.g. >>> ClusterSpecification taskManagerMemoryMB is now ignored, and should be set >>> via fink Configuration. >>> (though it's value still validated, but no longer used after that). >>> My main issue, is that Application seems to fail to start properly, it >>> seems that JobMaster fails to connect to ResourceManager, but I can't >>> figure out why. >>> The yarn log is attached. >>> >>> I'll appreciate if you tell me to which direction I should dig. >>> >>> Regards, >>> Vitaliy >>> >>> >>> On Mon, Mar 30, 2020 at 12:00 PM Robert Metzger >>> wrote: >>> >>>> Hey, >>>> which Flink version are you using? >>>> >>>> Where exactly are you seeing the "Log file environment variable >>>> 'log.file' is not set." message? Can you post some context around it? (is >>>> this shown from the command line? what are the arguments? is it shown in a >>>> file? >>>> >>>> Usually, the "log.file" property is used to pass the name of the log >>>> file into the log4j configuration. If this property is not set, I have to >>>> assume that you are using modified or custom scripts, or you are executing >>>> Flink in an environment that fails to set the property. >>>> >>>> When running Flink on YARN, the JobManager logs are stored on the >>>> machine running the JobManager. The logs accessible through "yarn logs" are >>>> the same as you would see in the JM interface. >>>> >>>> Best, >>>> Robert >>>> >>>> >>>> >>>> On Sun, Mar 29, 2020 at 11:22 PM Vitaliy Semochkin < >>>> vitaliy.se@gmail.com> wrote: >>>> >>>>> Hello Yun, >>>>> >>>>> I see this error reported by: >>>>> *org.apache.flink.runtime.webmonitor.WebMonitorUtils* - *JobManager >>>>> log files are unavailable in the web dashboard. Log file location not found >>>>> in environment variable 'log.file' or configuration key 'Key: >>>>> 'web.log.path' , default: null (fallback keys: >>>>> [{key=jobmanager.web.log.path, isDeprecated=true}])'.* >>>>> >>>>> I wonder where the JobManager files are stored in case running on a >>>>> YARN cluster? >>>>> Are these logs same to those I get via yarn logs -applicationId? >>>>> >>>>> Regards, >>>>> Vitaliy >>>>> >>>>> >>>>> >>>>> On Sun, Mar 29, 2020 at 8:24 PM Yun Tang wrote: >>>>> >>>>>> Hi Vitaliy >>>>>> >>>>>> Property of 'log.file' would be configured if you have uploaded >>>>>> 'logback.xml' or 'log4j.properties' [1]. >>>>>> The file would contain logs of job manager or task manager which is >>>>>> decided by the component itself. And as you can see, this is only a local >>>>>> file path, I am afraid this cannot understand hdfs paths. >>>>>> >>>>>> >>>>>> [1] >>>>>> https://github.com/apache/flink/blob/ae3b0ff80b93a83a358ab474060473863d2c30d6/flink-runtime/src/main/java/org/apache/flink/runtime/clusterframework/BootstrapTools.java#L420 >>>>>> >>>>>> Best >>>>>> Yun Tang >>>>>> ------------------------------ >>>>>> *From:* Vitaliy Semochkin >>>>>> *Sent:* Sunday, March 29, 2020 4:32 >>>>>> *To:* user >>>>>> *Subject:* Log file environment variable 'log.file' is not set. >>>>>> >>>>>> Hi, >>>>>> >>>>>> When I launch Flink Application Cluster I keep getting a message >>>>>> " Log file environment variable 'log.file' is not set." >>>>>> >>>>>> I use console logging via log4j >>>>>> and I read logs via yarn logs -applicationId .... >>>>>> >>>>>> What's the purpose of log.file property? >>>>>> What this file will contain and on which host should I search for the >>>>>> log? >>>>>> Does this property understands hdfs paths? >>>>>> >>>>>> Regards, >>>>>> Vitaliy >>>>>> >>>>> --000000000000c79cd805a220ee60 Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
On Mon, Mar 30, 2020 at 5:3= 9 PM Vitaliy Semochkin <vitaliy.= se@gmail.com> wrote:
Hello Robert,
Thank you for qui= ck response!
Indeed logs says the hadoop version is 2.4.1 this is= probably because of
How can I make 1.10 to work with my = current hadoop version?

Regarding flink reporting = in logs its 1.7.0
org.apache.flink.runtime.entrypoint.Cluster= Entrypoint =C2=A0 =C2=A0 =C2=A0 =C2=A0 - =C2=A0Starting YarnJobClusterEntry= point (Version: 1.7.0
while I'm using 1.10 and this is appli= cation cluster (everything is bundled and we don't have session cluster= running).
Here are the whole dependencies list:
mvn dependenc= y:tree | grep flink | cut -d'-' -f2-
=C2=A0org.apache.flink:flin= k-yarn_2.11:jar:1.10.0:runtime
=C2=A0org.apache.flink:flink-clients_2.11= :jar:1.10.0:compile
=C2=A0org.apache.flink:flink-optimizer_2.11:jar:1.10= .0:compile
=C2=A0org.apache.flink:flink-shaded-hadoop-2:jar:2.4.1-9.0:ru= ntime
=C2=A0org.apache.flink:force-shading:jar:1.10.0:compile
=C2=A0o= rg.apache.flink:flink-runtime_2.11:jar:1.10.0:runtime
=C2=A0org.apache.f= link:flink-core:jar:1.10.0:compile
=C2=A0org.apache.flink:flink-annotati= ons:jar:1.10.0:compile
=C2=A0org.apache.flink:flink-metrics-core:jar:1.1= 0.0:compile
=C2=A0org.apache.flink:flink-java:jar:1.10.0:compile
=C2= =A0org.apache.flink:flink-queryable-state-client-java:jar:1.10.0:runtime=C2=A0org.apache.flink:flink-hadoop-fs:jar:1.10.0:runtime
=C2=A0org.apa= che.flink:flink-shaded-netty:jar:4.1.39.Final-9.0:compile
=C2=A0org.apac= he.flink:flink-shaded-guava:jar:18.0-9.0:compile
=C2=A0org.apache.flink:= flink-shaded-asm-7:jar:7.1-9.0:compile
=C2=A0org.apache.flink:flink-shad= ed-jackson:jar:2.10.1-9.0:compile
=C2=A0org.apache.flink:flink-jdbc_2.11= :jar:1.10.0:compile
=C2=A0org.apache.flink:flink-hbase_2.11:jar:1.10.0:c= ompile
=C2=A0org.apache.flink:flink-runtime-web_2.11:jar:1.10.0:compile<= br>
As you can see all flink related libs are 1.10.
Can you please tell which class in flinks identifies the versio= n(I'll try to debug it locally)?

Regards,=
Vitaliy


On Mon, Mar 30, 2020 at 5:10 PM Robert Metz= ger <rmetzger@a= pache.org> wrote:
Hey Vitaliy,
is it okay for you if we keep = the discussion on the list, so that others can chime in to help, and that G= oogle can index the conversation, in case somebody else has a similar probl= em?

I just checked, and Flink on YARN in Flink 1.1= 0 does set the property=C2=A0correctly. Maybe in Flink 1.7, accessing the l= ogs in the web ui was not yet supported.

You said = in your email, that you are using Flink 1.10, however, your logs state that= you are running Flink 1.7.0.
It also seems that you have the Had= oop 2.4.1 dependencies of Flink, but your Hadoop environment is Hadoop 2.7.= 3. I believe this error is caused by that version mismatch:=C2=A0

Caused by: jav= a.lang.IllegalAccessError: tried to access method org.apache.hadoop.yarn.cl= ient.ConfiguredRMFailoverProxyProvider.getProxyInternal()Ljava/lang/Object;= from class org.apache.hadoop.yarn.client.RequestHedgingRMFailoverProxyProv= ider



On Mon, Mar 30, 2020 at 1:58= PM Vitaliy Semochkin <vitaliy.se@gmail.com> wrote:
Hello Robert,
= >Where exactly are you seeing the "Log file environment variable=20 'log.file' is not set." message?
I get this message = when I check yarn logs.

>Can you post some = context around it?=20 (is this shown from the command line? what are the arguments? is it=20 shown in a file?
I'm creating an application cluster from a j= ava application using flink 1.10. I didn't have this issue with 1.8.1 v= ersion,
however when I upgraded configuration slightly chang= ed, e.g. ClusterSpecification taskManagerMemoryMB is now ignored, and shoul= d be set via fink Configuration.
(though it's value still val= idated, but no longer used after that).
My main issue, is that Ap= plication seems to fail to start properly,=C2=A0 it seems that JobMaster fa= ils to connect to ResourceManager, but I can't figure out why.
The yarn log=C2=A0 is attached.

I'll appreci= ate if you tell me to which direction I should dig.

Regards,
Vitaliy


On Mon, Mar 30, 2= 020 at 12:00 PM Robert Metzger <rmetzger@apache.org> wrote:
Hey,
which Flink v= ersion are you using?

Where exactly are you seeing= the "Log file environment variable 'log.file' is not set.&quo= t; message? Can you post some context around it? (is this shown from the co= mmand line? what are the arguments? is it shown in a file?

Usually, the "log.file" property is used to pass the nam= e of the log file into the log4j configuration. If this property is not set= , I have to assume that you are using modified or custom scripts, or you ar= e executing Flink in an environment that fails to set the property.

When running Flink on YARN, the JobManager logs are store= d on the machine running the JobManager. The logs accessible through "= yarn logs" are the same as you would see in the JM interface.

Best,
Robert


=

= On Sun, Mar 29, 2020 at 11:22 PM Vitaliy Semochkin <vitaliy.se@gmail.com> wrote:
=
Hello Yun,

I see this error reported by:
org.apache.flink.runtime.webmonitor.WebMonitorUtils=C2=A0 - JobManager log files are unavailable in the web dashboard. Log file locati= on not found in environment variable 'log.file' or configuration ke= y 'Key: 'web.log.path' , default: null (fallback keys: [{key=3D= jobmanager.web.log.path, isDeprecated=3Dtrue}])'.

=
I wonder where the JobManager files are stored in case runni= ng on a YARN cluster?
Are these logs same to those I get via yarn logs -app= licationId?

Regards,
Vitaliy

=


On Sun, Mar 29, 2020 at 8:24 PM Yun Tang &l= t;myasuka@live.com> wrote:
Hi Vitaliy

Property of 'log.file' would be configured if you have uploaded = 9;logback.xml' or 'log4j.properties' [1].
The file would contain logs of job manager or task manager which is decided= by the component itself. And as you can see, this is only a local file pat= h, I am afraid this cannot understand hdfs paths.

Best
Yun Tang
From: Vitali= y Semochkin <v= italiy.se@gmail.com>
Sent: Sunday, March 29, 2020 4:32
To: user <user@flink.apache.org>
Subject: Log file environment variable 'log.file' is not set= .
=C2=A0
Hi,

When I launch Flink Application Cluster I keep getting a message
" Log file environment variable 'log.file' is not set.&qu= ot;

I use console logging via log4j=C2=A0
and I read logs via yarn logs -applicationId ....

What's the purpose of log.file property?
What this file will contain and on which host should I search for the = log?
Does this property understands hdfs paths?

Regards,
Vitaliy
--000000000000c79cd805a220ee60--