Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 81A80200B13 for ; Wed, 15 Jun 2016 10:18:32 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id 80078160A4D; Wed, 15 Jun 2016 08:18:32 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 7D5F8160A4C for ; Wed, 15 Jun 2016 10:18:31 +0200 (CEST) Received: (qmail 71919 invoked by uid 500); 15 Jun 2016 08:18:30 -0000 Mailing-List: contact user-help@flink.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@flink.apache.org Delivered-To: mailing list user@flink.apache.org Received: (qmail 71909 invoked by uid 99); 15 Jun 2016 08:18:30 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 15 Jun 2016 08:18:30 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id 16C5F18057B for ; Wed, 15 Jun 2016 08:18:30 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -1.827 X-Spam-Level: X-Spam-Status: No, score=-1.827 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_MED=-2.3, RP_MATCHES_RCVD=-1.426, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd3-us-west.apache.org (amavisd-new); dkim=pass (1024-bit key) header.d=kth.se Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id 56myBqsTolKm for ; Wed, 15 Jun 2016 08:18:27 +0000 (UTC) Received: from smtp-4.sys.kth.se (smtp-4.sys.kth.se [130.237.48.193]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id 5DDA55F1B3 for ; Wed, 15 Jun 2016 08:18:26 +0000 (UTC) Received: from smtp-4.sys.kth.se (localhost.localdomain [127.0.0.1]) by smtp-4.sys.kth.se (Postfix) with ESMTP id 9DBC21D82 for ; Wed, 15 Jun 2016 10:18:19 +0200 (CEST) X-Virus-Scanned: by amavisd-new at kth.se Received: from smtp-4.sys.kth.se ([127.0.0.1]) by smtp-4.sys.kth.se (smtp-4.sys.kth.se [127.0.0.1]) (amavisd-new, port 10024) with LMTP id P25y9enf0_WL for ; Wed, 15 Jun 2016 10:18:18 +0200 (CEST) X-KTH-Auth: tkak [193.10.67.68] DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=kth.se; s=default; t=1465978698; bh=Y+bLwUsW1BJUZvnXrLmLwECZEtecqCIWTGnC8agMH20=; h=Subject:To:References:From:Date:In-Reply-To; b=oE8ucQunYHLnjSzZ6+zyOtAt+UP0CUa/vg1vHB945HalKnMlcODpprM6qUqsFQtb2 7/rnNUpcbSwI8We2fT6ZVhdjmuHS58l3KQPTjOyTzYtsAq4dDd1YiVJcOFkjGF2Tk5 weHDDCg6JlZZNZHQHdAAUD33Zl8Wvt622c6ytRuo= X-KTH-mail-from: tkak@kth.se X-KTH-rcpt-to: user@flink.apache.org Received: from [193.10.67.68] (tkak1.sics.se [193.10.67.68]) by smtp-4.sys.kth.se (Postfix) with ESMTPSA id 98F9B2552 for ; Wed, 15 Jun 2016 10:18:18 +0200 (CEST) Subject: Re: Application log on Yarn FlinkCluster To: user@flink.apache.org References: <99d797d9-d450-a45b-5765-22624d9fc83f@kth.se> From: Theofilos Kakantousis Message-ID: Date: Wed, 15 Jun 2016 10:18:15 +0200 User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:45.0) Gecko/20100101 Thunderbird/45.1.1 MIME-Version: 1.0 In-Reply-To: Content-Type: multipart/alternative; boundary="------------9520EE597CC99B3037CAB479" archived-at: Wed, 15 Jun 2016 08:18:32 -0000 This is a multi-part message in MIME format. --------------9520EE597CC99B3037CAB479 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit Hi, By yarn aggregated log I mean Yarn log aggregation is enabled and the log I'm referring to is the one returned by `yarn logs -applicationId `. When running a Spark job for example on the same setup, the yarn aggregated log contains all the information printed out by the application. Cheers, Theofilos On 6/15/2016 10:14 AM, Maximilian Michels wrote: > Please use the `yarn logs -applicationId ` to retrieve the logs. > If you have enabled log aggregation, this will give you all container > logs concatenated. > > Cheers, > Max > > On Wed, Jun 15, 2016 at 12:24 AM, Theofilos Kakantousis > wrote: > > Hi Max, > > The runBlocking(..) problem was due to a Netty dependency issue on > my project, it works fine now :) > > To pinpoint the logging issue, I just ran a single flink job on > yarn as per the documentation "./bin/flink run -m yarn-cluster -yn > 2 ./examples/streaming/Iteration.jar" and I have the same > issue.During the job I can see in the containers the taskmanager > logs, and a sample output from the taskmanager.out is the following: > "cat > /srv/hadoop/logs/userlogs/application_1465901188070_0037/container_1465901188070_0037_01_000002/taskmanager.out > > 2> ((49,1),3) > 2> ((25,11),4) > 2> ((46,44),2 > .." > > However, the yarn aggregated log contains only the jobmanager > output. Is this expected or could it indicate a problem with my > hadoop logging configuration not picking up taskmanager logs? > > Cheers, > Theofilos > > > On 6/13/2016 12:13 PM, Maximilian Michels wrote: > > Hi Theofilos, > > Flink doesn't send the local client output to the Yarn cluster. I > think this will only change once we move the entire execution > of the > Job to the cluster framework. All output of the actual Flink job > should be within the JobManager or TaskManager logs. > > There is something wrong with the network communication if the > Client > doesn't return from `runBlocking(..)`. Would be interesting to > take a > look at the logs to find out why that could be. > > Cheers, > Max > > > On Sat, Jun 11, 2016 at 1:53 PM, Theofilos Kakantousis > > wrote: > > Hi Robert, > > Thanks for the prompt reply. I'm using the IterateExample > from Flink > examples. In the yarn log I get entries for the > YarnJobManager and > ExecutionGraph, but I was wondering if there is a way to > push all the > logging that the client produces into the yarn log. > Including the System.out > calls. Is there a way to modify the example to use a > logging framework to > achieve it? > > Also when I submit the program using the Client > runBlocking method, although > I see in the taskmanager and jobmanager log that the > application has > finished, the runBlocking method does not return. Should I > call it in a > separate thread? > > Cheers, > Theofilos > > On 2016-06-10 22:12, Robert Metzger wrote: > > Hi Theofilos, > > how exactly are you writing the application output? > Are you using a logging framework? > Are you writing the log statements from the open(), map(), > invoke() methods > or from some constructors? (I'm asking since different > parts are executed on > the cluster and locally). > > On Fri, Jun 10, 2016 at 4:00 PM, Theofilos Kakantousis > > wrote: > > Hi all, > > Flink 1.0.3 > Hadoop 2.4.0 > > When running a job on a Flink Cluster on Yarn, the > application output is > not included in the Yarn log. Instead, it is only > printed in the stdout from > where I run my program. For the jobmanager, I'm using > the log4j.properties > file from the flink/conf directory. Yarn log > aggregation is enabled and the > YarnJobManager log is printed in the yarn log. The > application is submitted > by a Flink Client to the FlinkYarnCluster using a > PackagedProgram. > > Is this expected behavior and if so, is there a way to > include the > application output in the Yarn aggregated log? Thanks! > > Cheers, > Theofilos > > > > --------------9520EE597CC99B3037CAB479 Content-Type: text/html; charset=utf-8 Content-Transfer-Encoding: 8bit
Hi,

By yarn aggregated log I mean Yarn log aggregation is enabled and the log I'm referring to is the one returned by `yarn logs -applicationId <id>`. When running a Spark job for example on the same setup, the yarn aggregated log contains all the information printed out by the application.

Cheers,
Theofilos

On 6/15/2016 10:14 AM, Maximilian Michels wrote:
Please use the `yarn logs -applicationId <id>` to retrieve the logs. If you have enabled log aggregation, this will give you all container logs concatenated.

Cheers,
Max

On Wed, Jun 15, 2016 at 12:24 AM, Theofilos Kakantousis <tkak@kth.se> wrote:
Hi Max,

The runBlocking(..) problem was due to a Netty dependency issue on my project, it works fine now :)

To pinpoint the logging issue, I just ran a single flink job on yarn as per the documentation "./bin/flink run -m yarn-cluster -yn 2 ./examples/streaming/Iteration.jar" and I have the same issue.During the job I can see in the containers the taskmanager logs, and a sample output from the taskmanager.out is the following:
"cat /srv/hadoop/logs/userlogs/application_1465901188070_0037/container_1465901188070_0037_01_000002/taskmanager.out
2> ((49,1),3)
2> ((25,11),4)
2> ((46,44),2
.."

However, the yarn aggregated log contains only the jobmanager output. Is this expected or could it indicate a problem with my hadoop logging configuration not picking up taskmanager logs?

Cheers,
Theofilos


On 6/13/2016 12:13 PM, Maximilian Michels wrote:
Hi Theofilos,

Flink doesn't send the local client output to the Yarn cluster. I
think this will only change once we move the entire execution of the
Job to the cluster framework. All output of the actual Flink job
should be within the JobManager or TaskManager logs.

There is something wrong with the network communication if the Client
doesn't return from `runBlocking(..)`. Would be interesting to take a
look at the logs to find out why that could be.

Cheers,
Max


On Sat, Jun 11, 2016 at 1:53 PM, Theofilos Kakantousis <tkak@kth.se> wrote:
Hi Robert,

Thanks for the prompt reply. I'm using the IterateExample from Flink
examples. In the yarn log I get entries for the YarnJobManager and
ExecutionGraph, but I was wondering if there is a way to push all the
logging that the client produces into the yarn log. Including the System.out
calls. Is there a way to modify the example to use a logging framework to
achieve it?

Also when I submit the program using the Client runBlocking method, although
I see in the taskmanager and jobmanager log that the application has
finished, the runBlocking method does not return. Should I call it in a
separate thread?

Cheers,
Theofilos

On 2016-06-10 22:12, Robert Metzger wrote:

Hi Theofilos,

how exactly are you writing the application output?
Are you using a logging framework?
Are you writing the log statements from the open(), map(), invoke() methods
or from some constructors? (I'm asking since different parts are executed on
the cluster and locally).

On Fri, Jun 10, 2016 at 4:00 PM, Theofilos Kakantousis <tkak@kth.se> wrote:
Hi all,

Flink 1.0.3
Hadoop 2.4.0

When running a job on a Flink Cluster on Yarn, the application output is
not included in the Yarn log. Instead, it is only printed in the stdout from
where I run my program.  For the jobmanager, I'm using the log4j.properties
file from the flink/conf directory. Yarn log aggregation is enabled and the
YarnJobManager log is printed in the yarn log. The application is submitted
by a Flink Client to the FlinkYarnCluster using a PackagedProgram.

Is this expected behavior and if so, is there a way to include the
application output in the Yarn aggregated log? Thanks!

Cheers,
Theofilos





--------------9520EE597CC99B3037CAB479--