From commits-return-92214-archive-asf-public=cust-asf.ponee.io@beam.apache.org Wed Sep 12 17:22:05 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id 9E596180630 for ; Wed, 12 Sep 2018 17:22:04 +0200 (CEST) Received: (qmail 28586 invoked by uid 500); 12 Sep 2018 15:22:03 -0000 Mailing-List: contact commits-help@beam.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@beam.apache.org Delivered-To: mailing list commits@beam.apache.org Received: (qmail 28577 invoked by uid 99); 12 Sep 2018 15:22:03 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 12 Sep 2018 15:22:03 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id 3583F18FE19 for ; Wed, 12 Sep 2018 15:22:03 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -109.501 X-Spam-Level: X-Spam-Status: No, score=-109.501 tagged_above=-999 required=6.31 tests=[ENV_AND_HDR_SPF_MATCH=-0.5, KAM_ASCII_DIVIDERS=0.8, RCVD_IN_DNSWL_MED=-2.3, SPF_PASS=-0.001, USER_IN_DEF_SPF_WL=-7.5, USER_IN_WHITELIST=-100] autolearn=disabled Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id mpRahfoX9dbe for ; Wed, 12 Sep 2018 15:22:02 +0000 (UTC) Received: from mailrelay1-us-west.apache.org (mailrelay1-us-west.apache.org [209.188.14.139]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTP id 020575F4E7 for ; Wed, 12 Sep 2018 15:22:02 +0000 (UTC) Received: from jira-lw-us.apache.org (unknown [207.244.88.139]) by mailrelay1-us-west.apache.org (ASF Mail Server at mailrelay1-us-west.apache.org) with ESMTP id 1F922E2634 for ; Wed, 12 Sep 2018 15:22:01 +0000 (UTC) Received: from jira-lw-us.apache.org (localhost [127.0.0.1]) by jira-lw-us.apache.org (ASF Mail Server at jira-lw-us.apache.org) with ESMTP id 57C9D26EEB for ; Wed, 12 Sep 2018 15:22:00 +0000 (UTC) Date: Wed, 12 Sep 2018 15:22:00 +0000 (UTC) From: "ASF GitHub Bot (JIRA)" To: commits@beam.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Work logged] (BEAM-5246) Beam metrics exported as flink metrics are not correct MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/BEAM-5246?focusedWorklogId=143577&page=com.atlassian.jira.plugin.system.issuetabpanels:worklog-tabpanel#worklog-143577 ] ASF GitHub Bot logged work on BEAM-5246: ---------------------------------------- Author: ASF GitHub Bot Created on: 12/Sep/18 15:21 Start Date: 12/Sep/18 15:21 Worklog Time Spent: 10m Work Description: echauchot removed a comment on issue #6319: [BEAM-5246] Metric container should forward only metrics of the stepName URL: https://github.com/apache/beam/pull/6319#issuecomment-420688584 @JozoVilcek thanks for the contribution ! And welcome abord ! @JozoVilcek @mxm `FlinkMetricsContainer` backs a `MetricsContainer` and there is one `MetricsContainer` per bundle x step. Also `DoFnRunnerWithMetricsUpdate` runs for a particular step to which the `FlinkMetricsContainer` is scoped to. So I think it is makes a lot of sense. LGTM also. ---------------------------------------------------------------- This is an automated message from the Apache Git Service. To respond to the message, please log on GitHub and use the URL above to go to the specific comment. For queries about this service, please contact Infrastructure at: users@infra.apache.org Issue Time Tracking ------------------- Worklog Id: (was: 143577) Time Spent: 1h (was: 50m) > Beam metrics exported as flink metrics are not correct > ------------------------------------------------------ > > Key: BEAM-5246 > URL: https://issues.apache.org/jira/browse/BEAM-5246 > Project: Beam > Issue Type: Bug > Components: runner-flink > Affects Versions: 2.6.0 > Reporter: Jozef Vilcek > Assignee: Jozef Vilcek > Priority: Major > Fix For: 2.7.0 > > Time Spent: 1h > Remaining Estimate: 0h > > In Flink UI and fink native MetricReported, I am seeing too many instances of my Beam metric counter. It looks like the counter is materialised for every operator running within the task, although is is emitter from only one beam step (which should map to one operator?). This produces double counting. > A bit debugging I noticed this is happening for stream jobs. In batch I was not able to reproduce it. Problem might be in FlinkMetricContainer. > [https://github.com/apache/beam/blob/master/runners/flink/src/main/java/org/apache/beam/runners/flink/metrics/FlinkMetricContainer.java#L86] > The update seems to be called from operators after finishing the bundle. Data from accumulator are flushed to `runtimeContext.getMetricGroup()`. The scope of accumulator seems to be different than metricGroup as in there with different call the scope components change, especially for operatorID. It seems like during the run, `metricResult.getStep()` does not match operatorName of metricGroup where metric is being pushed. -- This message was sent by Atlassian JIRA (v7.6.3#76005)