flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Theofilos Kakantousis <t...@kth.se>
Subject Re: Application log on Yarn FlinkCluster
Date Wed, 15 Jun 2016 16:49:17 GMT
Great then, I will look into my configuration. Thanks for your help!

Cheers,
Theofilos

On 6/15/2016 2:00 PM, Maximilian Michels wrote:
> You should also see TaskManager output in the logs. I just verified 
> this using Flink 1.0.3 with Hadoop 2.7.1. I executed the Iterate 
> example and it aggregated correctly including the TaskManager logs.
>
> I'm wondering, is there anything in the Hadoop logs of the 
> Resourcemanager/Nodemanager that could indicate a transfer failure?
>
> On Wed, Jun 15, 2016 at 10:18 AM, Theofilos Kakantousis <tkak@kth.se 
> <mailto:tkak@kth.se>> wrote:
>
>     Hi,
>
>     By yarn aggregated log I mean Yarn log aggregation is enabled and
>     the log I'm referring to is the one returned by `yarn logs
>     -applicationId <id>`. When running a Spark job for example on the
>     same setup, the yarn aggregated log contains all the information
>     printed out by the application.
>
>     Cheers,
>     Theofilos
>
>
>     On 6/15/2016 10:14 AM, Maximilian Michels wrote:
>>     Please use the `yarn logs -applicationId <id>` to retrieve the
>>     logs. If you have enabled log aggregation, this will give you all
>>     container logs concatenated.
>>
>>     Cheers,
>>     Max
>>
>>     On Wed, Jun 15, 2016 at 12:24 AM, Theofilos Kakantousis
>>     <tkak@kth.se <mailto:tkak@kth.se>> wrote:
>>
>>         Hi Max,
>>
>>         The runBlocking(..) problem was due to a Netty dependency
>>         issue on my project, it works fine now :)
>>
>>         To pinpoint the logging issue, I just ran a single flink job
>>         on yarn as per the documentation "./bin/flink run -m
>>         yarn-cluster -yn 2 ./examples/streaming/Iteration.jar" and I
>>         have the same issue.During the job I can see in the
>>         containers the taskmanager logs, and a sample output from the
>>         taskmanager.out is the following:
>>         "cat
>>         /srv/hadoop/logs/userlogs/application_1465901188070_0037/container_1465901188070_0037_01_000002/taskmanager.out
>>
>>         2> ((49,1),3)
>>         2> ((25,11),4)
>>         2> ((46,44),2
>>         .."
>>
>>         However, the yarn aggregated log contains only the jobmanager
>>         output. Is this expected or could it indicate a problem with
>>         my hadoop logging configuration not picking up taskmanager logs?
>>
>>         Cheers,
>>         Theofilos
>>
>>
>>         On 6/13/2016 12:13 PM, Maximilian Michels wrote:
>>
>>             Hi Theofilos,
>>
>>             Flink doesn't send the local client output to the Yarn
>>             cluster. I
>>             think this will only change once we move the entire
>>             execution of the
>>             Job to the cluster framework. All output of the actual
>>             Flink job
>>             should be within the JobManager or TaskManager logs.
>>
>>             There is something wrong with the network communication
>>             if the Client
>>             doesn't return from `runBlocking(..)`. Would be
>>             interesting to take a
>>             look at the logs to find out why that could be.
>>
>>             Cheers,
>>             Max
>>
>>
>>             On Sat, Jun 11, 2016 at 1:53 PM, Theofilos Kakantousis
>>             <tkak@kth.se <mailto:tkak@kth.se>> wrote:
>>
>>                 Hi Robert,
>>
>>                 Thanks for the prompt reply. I'm using the
>>                 IterateExample from Flink
>>                 examples. In the yarn log I get entries for the
>>                 YarnJobManager and
>>                 ExecutionGraph, but I was wondering if there is a way
>>                 to push all the
>>                 logging that the client produces into the yarn log.
>>                 Including the System.out
>>                 calls. Is there a way to modify the example to use a
>>                 logging framework to
>>                 achieve it?
>>
>>                 Also when I submit the program using the Client
>>                 runBlocking method, although
>>                 I see in the taskmanager and jobmanager log that the
>>                 application has
>>                 finished, the runBlocking method does not return.
>>                 Should I call it in a
>>                 separate thread?
>>
>>                 Cheers,
>>                 Theofilos
>>
>>                 On 2016-06-10 22:12, Robert Metzger wrote:
>>
>>                 Hi Theofilos,
>>
>>                 how exactly are you writing the application output?
>>                 Are you using a logging framework?
>>                 Are you writing the log statements from the open(),
>>                 map(), invoke() methods
>>                 or from some constructors? (I'm asking since
>>                 different parts are executed on
>>                 the cluster and locally).
>>
>>                 On Fri, Jun 10, 2016 at 4:00 PM, Theofilos
>>                 Kakantousis <tkak@kth.se <mailto:tkak@kth.se>> wrote:
>>
>>                     Hi all,
>>
>>                     Flink 1.0.3
>>                     Hadoop 2.4.0
>>
>>                     When running a job on a Flink Cluster on Yarn,
>>                     the application output is
>>                     not included in the Yarn log. Instead, it is only
>>                     printed in the stdout from
>>                     where I run my program.  For the jobmanager, I'm
>>                     using the log4j.properties
>>                     file from the flink/conf directory. Yarn log
>>                     aggregation is enabled and the
>>                     YarnJobManager log is printed in the yarn log.
>>                     The application is submitted
>>                     by a Flink Client to the FlinkYarnCluster using a
>>                     PackagedProgram.
>>
>>                     Is this expected behavior and if so, is there a
>>                     way to include the
>>                     application output in the Yarn aggregated log?
>>                     Thanks!
>>
>>                     Cheers,
>>                     Theofilos
>>
>>
>>
>>
>
>


Mime
View raw message