Mailing-List: contact user-help@flink.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@flink.apache.org
Subject: Re: Application log on Yarn FlinkCluster
To: user@flink.apache.org
References: <fe66f875-6b56-8391-1057-bf4da2d3ea1f@kth.se>
 <CAGr9p8AaAVRHJe4WdR3tKirzuaaMYqLS0kr3_isnivP1FR++sg@mail.gmail.com>
 <99d797d9-d450-a45b-5765-22624d9fc83f@kth.se>
 <CAGco--ZmAGFvftX2D+S40hQ6m25nTuhzW1HNi9OHCX_w2vSQig@mail.gmail.com>
From: Theofilos Kakantousis <tkak@kth.se>
Message-ID: <c84c097a-b52f-cec2-808e-63fb96e6c80e@kth.se>
Date: Wed, 15 Jun 2016 00:24:15 +0200
User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:45.0) Gecko/20100101
 Thunderbird/45.1.1
MIME-Version: 1.0
In-Reply-To: <CAGco--ZmAGFvftX2D+S40hQ6m25nTuhzW1HNi9OHCX_w2vSQig@mail.gmail.com>
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 7bit
archived-at: Tue, 14 Jun 2016 22:24:32 -0000

Hi Max,

The runBlocking(..) problem was due to a Netty dependency issue on my 
project, it works fine now :)

To pinpoint the logging issue, I just ran a single flink job on yarn as 
per the documentation "./bin/flink run -m yarn-cluster -yn 2 
./examples/streaming/Iteration.jar" and I have the same issue.During the 
job I can see in the containers the taskmanager logs, and a sample 
output from the taskmanager.out is the following:
"cat 
/srv/hadoop/logs/userlogs/application_1465901188070_0037/container_1465901188070_0037_01_000002/taskmanager.out 

2> ((49,1),3)
2> ((25,11),4)
2> ((46,44),2
.."

However, the yarn aggregated log contains only the jobmanager output. Is 
this expected or could it indicate a problem with my hadoop logging 
configuration not picking up taskmanager logs?

Cheers,
Theofilos

On 6/13/2016 12:13 PM, Maximilian Michels wrote:
> Hi Theofilos,
>
> Flink doesn't send the local client output to the Yarn cluster. I
> think this will only change once we move the entire execution of the
> Job to the cluster framework. All output of the actual Flink job
> should be within the JobManager or TaskManager logs.
>
> There is something wrong with the network communication if the Client
> doesn't return from `runBlocking(..)`. Would be interesting to take a
> look at the logs to find out why that could be.
>
> Cheers,
> Max
>
>
> On Sat, Jun 11, 2016 at 1:53 PM, Theofilos Kakantousis <tkak@kth.se> wrote:
>> Hi Robert,
>>
>> Thanks for the prompt reply. I'm using the IterateExample from Flink
>> examples. In the yarn log I get entries for the YarnJobManager and
>> ExecutionGraph, but I was wondering if there is a way to push all the
>> logging that the client produces into the yarn log. Including the System.out
>> calls. Is there a way to modify the example to use a logging framework to
>> achieve it?
>>
>> Also when I submit the program using the Client runBlocking method, although
>> I see in the taskmanager and jobmanager log that the application has
>> finished, the runBlocking method does not return. Should I call it in a
>> separate thread?
>>
>> Cheers,
>> Theofilos
>>
>> On 2016-06-10 22:12, Robert Metzger wrote:
>>
>> Hi Theofilos,
>>
>> how exactly are you writing the application output?
>> Are you using a logging framework?
>> Are you writing the log statements from the open(), map(), invoke() methods
>> or from some constructors? (I'm asking since different parts are executed on
>> the cluster and locally).
>>
>> On Fri, Jun 10, 2016 at 4:00 PM, Theofilos Kakantousis <tkak@kth.se> wrote:
>>> Hi all,
>>>
>>> Flink 1.0.3
>>> Hadoop 2.4.0
>>>
>>> When running a job on a Flink Cluster on Yarn, the application output is
>>> not included in the Yarn log. Instead, it is only printed in the stdout from
>>> where I run my program.  For the jobmanager, I'm using the log4j.properties
>>> file from the flink/conf directory. Yarn log aggregation is enabled and the
>>> YarnJobManager log is printed in the yarn log. The application is submitted
>>> by a Flink Client to the FlinkYarnCluster using a PackagedProgram.
>>>
>>> Is this expected behavior and if so, is there a way to include the
>>> application output in the Yarn aggregated log? Thanks!
>>>
>>> Cheers,
>>> Theofilos
>>>
>>