flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Theofilos Kakantousis <t...@kth.se>
Subject Re: Application log on Yarn FlinkCluster
Date Tue, 14 Jun 2016 22:24:15 GMT
Hi Max,

The runBlocking(..) problem was due to a Netty dependency issue on my 
project, it works fine now :)

To pinpoint the logging issue, I just ran a single flink job on yarn as 
per the documentation "./bin/flink run -m yarn-cluster -yn 2 
./examples/streaming/Iteration.jar" and I have the same issue.During the 
job I can see in the containers the taskmanager logs, and a sample 
output from the taskmanager.out is the following:
"cat 
/srv/hadoop/logs/userlogs/application_1465901188070_0037/container_1465901188070_0037_01_000002/taskmanager.out


2> ((49,1),3)
2> ((25,11),4)
2> ((46,44),2
.."

However, the yarn aggregated log contains only the jobmanager output. Is 
this expected or could it indicate a problem with my hadoop logging 
configuration not picking up taskmanager logs?

Cheers,
Theofilos

On 6/13/2016 12:13 PM, Maximilian Michels wrote:
> Hi Theofilos,
>
> Flink doesn't send the local client output to the Yarn cluster. I
> think this will only change once we move the entire execution of the
> Job to the cluster framework. All output of the actual Flink job
> should be within the JobManager or TaskManager logs.
>
> There is something wrong with the network communication if the Client
> doesn't return from `runBlocking(..)`. Would be interesting to take a
> look at the logs to find out why that could be.
>
> Cheers,
> Max
>
>
> On Sat, Jun 11, 2016 at 1:53 PM, Theofilos Kakantousis <tkak@kth.se> wrote:
>> Hi Robert,
>>
>> Thanks for the prompt reply. I'm using the IterateExample from Flink
>> examples. In the yarn log I get entries for the YarnJobManager and
>> ExecutionGraph, but I was wondering if there is a way to push all the
>> logging that the client produces into the yarn log. Including the System.out
>> calls. Is there a way to modify the example to use a logging framework to
>> achieve it?
>>
>> Also when I submit the program using the Client runBlocking method, although
>> I see in the taskmanager and jobmanager log that the application has
>> finished, the runBlocking method does not return. Should I call it in a
>> separate thread?
>>
>> Cheers,
>> Theofilos
>>
>> On 2016-06-10 22:12, Robert Metzger wrote:
>>
>> Hi Theofilos,
>>
>> how exactly are you writing the application output?
>> Are you using a logging framework?
>> Are you writing the log statements from the open(), map(), invoke() methods
>> or from some constructors? (I'm asking since different parts are executed on
>> the cluster and locally).
>>
>> On Fri, Jun 10, 2016 at 4:00 PM, Theofilos Kakantousis <tkak@kth.se> wrote:
>>> Hi all,
>>>
>>> Flink 1.0.3
>>> Hadoop 2.4.0
>>>
>>> When running a job on a Flink Cluster on Yarn, the application output is
>>> not included in the Yarn log. Instead, it is only printed in the stdout from
>>> where I run my program.  For the jobmanager, I'm using the log4j.properties
>>> file from the flink/conf directory. Yarn log aggregation is enabled and the
>>> YarnJobManager log is printed in the yarn log. The application is submitted
>>> by a Flink Client to the FlinkYarnCluster using a PackagedProgram.
>>>
>>> Is this expected behavior and if so, is there a way to include the
>>> application output in the Yarn aggregated log? Thanks!
>>>
>>> Cheers,
>>> Theofilos
>>>
>>


Mime
View raw message