Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 6E219200AF5 for ; Thu, 2 Jun 2016 22:39:42 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id 6CB9F160A51; Thu, 2 Jun 2016 20:39:42 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 8E8CF160A3E for ; Thu, 2 Jun 2016 22:39:41 +0200 (CEST) Received: (qmail 7396 invoked by uid 500); 2 Jun 2016 20:39:40 -0000 Mailing-List: contact user-help@flink.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@flink.apache.org Delivered-To: mailing list user@flink.apache.org Received: (qmail 7386 invoked by uid 99); 2 Jun 2016 20:39:40 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Jun 2016 20:39:40 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id 2AF89C06CD for ; Thu, 2 Jun 2016 20:39:40 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.28 X-Spam-Level: * X-Spam-Status: No, score=1.28 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01] autolearn=disabled Authentication-Results: spamd4-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=data-artisans-com.20150623.gappssmtp.com Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id 1MXsLQl83KJ3 for ; Thu, 2 Jun 2016 20:39:38 +0000 (UTC) Received: from mail-wm0-f52.google.com (mail-wm0-f52.google.com [74.125.82.52]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id A000D5F24B for ; Thu, 2 Jun 2016 20:39:37 +0000 (UTC) Received: by mail-wm0-f52.google.com with SMTP id a20so81755192wma.1 for ; Thu, 02 Jun 2016 13:39:37 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=data-artisans-com.20150623.gappssmtp.com; s=20150623; h=from:message-id:mime-version:subject:date:references:to:in-reply-to; bh=wQKmaE9SAUFvT2roxYK2KCapA41MXAwFhNvbQJdpbj0=; b=L2vrhsfchEeM5Qr9NvYM7H0n+7kudqHozHr1lO872B/SOdyCjiUKKgJSFWj6IOvyRF wKd5uUlXBaRqxySIjoxX7NgQFMeVrNUSBzeE0s2QXHFXQpqClmThOMxKdly7Gzc6QWzA EyNk4G9B7JFJlWl2t5GZyrqFHaf8TB1BPpcsyHp7lBEzOPsqGti9S0pIlYTUWKOxHIh7 3a6OJKCI2nBhoelhV+GWQx8bOYQ+NwDAdaslVU49L3Ea/YGm9S3RWMIDOCy+bV7VqBhf bAiRR982jHm6lC7DdglML8Q4JzLC4tv445qphDRnx/kV/ZYAlLHDGAt2PfRjDy8Cw5Ud n9CQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:from:message-id:mime-version:subject:date :references:to:in-reply-to; bh=wQKmaE9SAUFvT2roxYK2KCapA41MXAwFhNvbQJdpbj0=; b=ck9NPIU7WCNypWxAIJddRyS8gFD3PBiVW6eWfZCFHRVvbTMOuinuo6L1EH2gsfXlyS MFxVLxBGGMoAZ3+l7S65VqfnwevRy4NMXpkpKmnI7Va2ge3aFuSG7aYgFfcmyXnF7nht Bi9e68Tkd1PTb5JEvq0cEfNzAmcMoNFZHcF4BQPfHCmUwlNGNMAEhWJhnC/Hut8F4OsI Zz7EO19pcWc2WyucEsiZ105V630NiLCaFAVqE/vmr9vnTE7pR46Vj0tm6l0e0Vh0Yz3C qIHDCYyLGJlgcZsUxKzmhN04ZnD4BDLjFxCgs2nUHgZl6w2lkasJTwc+UZnw4GQ3IcxR sORA== X-Gm-Message-State: ALyK8tJDygkwHI1Vuwm1FicFYE4pIC7GezHunMi/fJ0/J/zNhFzWyyTwj+ms0SE+0OigNRQe X-Received: by 10.194.133.135 with SMTP id pc7mr49881wjb.49.1464899976090; Thu, 02 Jun 2016 13:39:36 -0700 (PDT) Received: from [192.168.0.3] (cable-86-56-24-123.cust.telecolumbus.net. [86.56.24.123]) by smtp.gmail.com with ESMTPSA id d195sm2948320wmd.12.2016.06.02.13.39.35 for (version=TLS1 cipher=ECDHE-RSA-AES128-SHA bits=128/128); Thu, 02 Jun 2016 13:39:35 -0700 (PDT) From: Kostas Kloudas Content-Type: multipart/alternative; boundary="Apple-Mail=_BF86820C-559C-4DF1-B256-BB7BB79863A4" Message-Id: Mime-Version: 1.0 (Mac OS X Mail 9.3 \(3124\)) Subject: Re: Adding Context To Logs Date: Thu, 2 Jun 2016 22:39:34 +0200 References: To: user@flink.apache.org In-Reply-To: X-Mailer: Apple Mail (2.3124) archived-at: Thu, 02 Jun 2016 20:39:42 -0000 --Apple-Mail=_BF86820C-559C-4DF1-B256-BB7BB79863A4 Content-Transfer-Encoding: quoted-printable Content-Type: text/plain; charset=utf-8 Hello Paul, If I understand correctly, your issues seem to be related to: = https://issues.apache.org/jira/browse/FLINK-1502?jql=3Dassignee%20in%20(Ze= ntol)%20AND%20text%20~%20%22metrics%22 = and = https://issues.apache.org/jira/browse/FLINK-2732?jql=3Dassignee%20in%20(Ze= ntol)%20AND%20text%20~%20%22metrics%22 = If this is correct, probably Chesnay (who worked on these issues) is the = right person to talk to. Kostas > On Jun 2, 2016, at 6:53 PM, Hamilton, Paul = wrote: >=20 > Hi, > =20 > We are trying to rework the way our Flink jobs (and really it becomes = how the operators within the jobs) log. Specifically we are trying to = add contextual information to the log messages so that it is easier to = filter and search for a specific jobs logs after they have been send = through to our ELK stack. I am certainly open to suggestions, and = hopefully I am missing something easier, but these were my current = thoughts: > =20 > 1) In the application and then also within individual functions = that log add the =E2=80=9Cjob name=E2=80=9D to the logging MDC in the = open function so that no matter where it is running on it can log the = appropriate job name. The problem here is that I haven=E2=80=99t found = an easy way to get to the job name within the RichFunction. What would = be the best way to get it? > =20 > 2) Pass the job name (and any other contextual things we want to = log) in at construction time when building the flink job to each = function that needs them. Simple, but seems kind of ugly. > =20 > =20 > Thoughts? > =20 > Thanks, > =20 > Paul Hamilton > SAP Hybris --Apple-Mail=_BF86820C-559C-4DF1-B256-BB7BB79863A4 Content-Transfer-Encoding: quoted-printable Content-Type: text/html; charset=utf-8 Hello Paul,

If I understand correctly, your issues seem to be related = to:
https://issues.apache.org/jira/browse/FLINK-1502?jql=3Dassignee= %20in%20(Zentol)%20AND%20text%20~%20%22metrics%22
and
https://issues.apache.org/jira/browse/FLINK-2732?jql=3Dassignee= %20in%20(Zentol)%20AND%20text%20~%20%22metrics%22

If this is correct, = probably Chesnay (who worked on these issues) is the right person to = talk to.

Kostas

On = Jun 2, 2016, at 6:53 PM, Hamilton, Paul <paul.hamilton01@sap.com> wrote:

Hi,
 
We are trying to rework the way = our Flink jobs (and really it becomes how the operators within the jobs) = log.  Specifically we are trying to add contextual information to = the log messages so that it is easier to filter and search for a = specific jobs logs after they have been send through to our ELK = stack.  I am certainly open to suggestions, and hopefully  I = am missing something easier, but these were my current thoughts:
 
1)       In the application and then also = within individual functions that log add the =E2=80=9Cjob name=E2=80=9D = to the logging MDC in the open function so that no matter where  it = is running on it can log the appropriate job name.  The problem = here is that I haven=E2=80=99t found an easy way to get to the job name = within the RichFunction.  What would be the best way to get it?
 
2)       Pass the job name (and any other = contextual things we want to log) in at construction time when building = the flink job to each function that needs them.  Simple, but seems = kind of ugly.
 
 
Thoughts?
 
Thanks,
 
Paul Hamilton
SAP Hybris

= --Apple-Mail=_BF86820C-559C-4DF1-B256-BB7BB79863A4--