flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Theofilos Kakantousis <t...@kth.se>
Subject Re: Application log on Yarn FlinkCluster
Date Wed, 15 Jun 2016 08:18:15 GMT
Hi,

By yarn aggregated log I mean Yarn log aggregation is enabled and the 
log I'm referring to is the one returned by `yarn logs -applicationId 
<id>`. When running a Spark job for example on the same setup, the yarn 
aggregated log contains all the information printed out by the application.

Cheers,
Theofilos

On 6/15/2016 10:14 AM, Maximilian Michels wrote:
> Please use the `yarn logs -applicationId <id>` to retrieve the logs. 
> If you have enabled log aggregation, this will give you all container 
> logs concatenated.
>
> Cheers,
> Max
>
> On Wed, Jun 15, 2016 at 12:24 AM, Theofilos Kakantousis <tkak@kth.se 
> <mailto:tkak@kth.se>> wrote:
>
>     Hi Max,
>
>     The runBlocking(..) problem was due to a Netty dependency issue on
>     my project, it works fine now :)
>
>     To pinpoint the logging issue, I just ran a single flink job on
>     yarn as per the documentation "./bin/flink run -m yarn-cluster -yn
>     2 ./examples/streaming/Iteration.jar" and I have the same
>     issue.During the job I can see in the containers the taskmanager
>     logs, and a sample output from the taskmanager.out is the following:
>     "cat
>     /srv/hadoop/logs/userlogs/application_1465901188070_0037/container_1465901188070_0037_01_000002/taskmanager.out
>
>     2> ((49,1),3)
>     2> ((25,11),4)
>     2> ((46,44),2
>     .."
>
>     However, the yarn aggregated log contains only the jobmanager
>     output. Is this expected or could it indicate a problem with my
>     hadoop logging configuration not picking up taskmanager logs?
>
>     Cheers,
>     Theofilos
>
>
>     On 6/13/2016 12:13 PM, Maximilian Michels wrote:
>
>         Hi Theofilos,
>
>         Flink doesn't send the local client output to the Yarn cluster. I
>         think this will only change once we move the entire execution
>         of the
>         Job to the cluster framework. All output of the actual Flink job
>         should be within the JobManager or TaskManager logs.
>
>         There is something wrong with the network communication if the
>         Client
>         doesn't return from `runBlocking(..)`. Would be interesting to
>         take a
>         look at the logs to find out why that could be.
>
>         Cheers,
>         Max
>
>
>         On Sat, Jun 11, 2016 at 1:53 PM, Theofilos Kakantousis
>         <tkak@kth.se <mailto:tkak@kth.se>> wrote:
>
>             Hi Robert,
>
>             Thanks for the prompt reply. I'm using the IterateExample
>             from Flink
>             examples. In the yarn log I get entries for the
>             YarnJobManager and
>             ExecutionGraph, but I was wondering if there is a way to
>             push all the
>             logging that the client produces into the yarn log.
>             Including the System.out
>             calls. Is there a way to modify the example to use a
>             logging framework to
>             achieve it?
>
>             Also when I submit the program using the Client
>             runBlocking method, although
>             I see in the taskmanager and jobmanager log that the
>             application has
>             finished, the runBlocking method does not return. Should I
>             call it in a
>             separate thread?
>
>             Cheers,
>             Theofilos
>
>             On 2016-06-10 22:12, Robert Metzger wrote:
>
>             Hi Theofilos,
>
>             how exactly are you writing the application output?
>             Are you using a logging framework?
>             Are you writing the log statements from the open(), map(),
>             invoke() methods
>             or from some constructors? (I'm asking since different
>             parts are executed on
>             the cluster and locally).
>
>             On Fri, Jun 10, 2016 at 4:00 PM, Theofilos Kakantousis
>             <tkak@kth.se <mailto:tkak@kth.se>> wrote:
>
>                 Hi all,
>
>                 Flink 1.0.3
>                 Hadoop 2.4.0
>
>                 When running a job on a Flink Cluster on Yarn, the
>                 application output is
>                 not included in the Yarn log. Instead, it is only
>                 printed in the stdout from
>                 where I run my program.  For the jobmanager, I'm using
>                 the log4j.properties
>                 file from the flink/conf directory. Yarn log
>                 aggregation is enabled and the
>                 YarnJobManager log is printed in the yarn log. The
>                 application is submitted
>                 by a Flink Client to the FlinkYarnCluster using a
>                 PackagedProgram.
>
>                 Is this expected behavior and if so, is there a way to
>                 include the
>                 application output in the Yarn aggregated log? Thanks!
>
>                 Cheers,
>                 Theofilos
>
>
>
>


Mime
View raw message