From user-return-4294-archive-asf-public=cust-asf.ponee.io@beam.apache.org Fri Nov 30 17:10:37 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id C76C3180671 for ; Fri, 30 Nov 2018 17:10:36 +0100 (CET) Received: (qmail 96527 invoked by uid 500); 30 Nov 2018 16:10:35 -0000 Mailing-List: contact user-help@beam.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@beam.apache.org Delivered-To: mailing list user@beam.apache.org Received: (qmail 96515 invoked by uid 99); 30 Nov 2018 16:10:35 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 Nov 2018 16:10:35 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id 408E6C035A for ; Fri, 30 Nov 2018 16:10:35 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -0.203 X-Spam-Level: X-Spam-Status: No, score=-0.203 tagged_above=-999 required=6.31 tests=[DKIMWL_WL_MED=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd4-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id h3uNtZ5QCVrm for ; Fri, 30 Nov 2018 16:10:34 +0000 (UTC) Received: from mail-it1-f169.google.com (mail-it1-f169.google.com [209.85.166.169]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id 560B25FAC3 for ; Fri, 30 Nov 2018 16:10:33 +0000 (UTC) Received: by mail-it1-f169.google.com with SMTP id i7so9872980iti.2 for ; Fri, 30 Nov 2018 08:10:33 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:content-transfer-encoding:mime-version:subject:message-id:date :to; bh=yjszpNkKSNO3KIsWNzeG6XEGMhdnPuH1BMap83qO88w=; b=JFsOa6dqsvZbPSeYX5NcjHhmudGLn0cVw7TZ2GdeWD2nHG1MhnE1bE2B8AZbAIS/0a rbkqKxGNvwUVhVqfoDR9Vx+ul5injBkf4gptIdE7eZDM2A1XqUTD84TtPGAd+LvE/Thr yZicUnj5tLanO9AoFayyff1sQRZNHfJW+bL+rLdXhQzvBQjzlF+k4a9688PZ95oZpGXm CZVYs1dgLvnt8LE5f82ANc8ju0QnlDjQ5jHMl6v+10kLy98fnNWCjKAbG3c2+9tTcQMj 1xeIkg0gJjHPKiGjpKYRQJKfbpPbE2JjKuFpxkr1tuKmRP6dZMx8qw4k6TyLYpPiL7Cn 952Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:content-transfer-encoding:mime-version :subject:message-id:date:to; bh=yjszpNkKSNO3KIsWNzeG6XEGMhdnPuH1BMap83qO88w=; b=hWQUc2rGdxwxSMmXDpjFKUszRcYdp29RSBtzGuKiCNZVu4naaumlnDU9wvPRPVJXxP BNYmNh9i024abAMCRf2BY6yDA/ff3A/8Wy5gOTtv5pk0Hg+I/hffgX9aVDaZdfgV88EB Vjoxq9hd5XyrGt7OkC3iku+MdndW8ZOZntUtbABnzgMZJenODDpblNF+7Rj4QJRHeFOZ dxqW6lKDAqy8t/LRUOilT+Mfd5tsJvldJjXPHXoPXOzunSh2U4KOdj/Yth7n1HuND5Hd pVvMvGJYTmeMbS/icMZpC/vzzU0LVGzPBb31TgZUm//UHFo/eXjLEfvHTq2aCWt3Vas7 e3dA== X-Gm-Message-State: AA+aEWbD7lQVuToViRy2Oi2VL7AYBqFbLlYmHn6VUEqIHe0WLgKmfRZ3 DkzBihtNdu/bb5ejTz7sLmRDYGP/ X-Google-Smtp-Source: AFSGD/UnStuE+73UunghfE0p4GxTPWUBR1qo/B4yg/n20zMz0M1tNzLxh20y3P0LVixHMge59UnEzg== X-Received: by 2002:a24:76d0:: with SMTP id z199mr5310806itb.165.1543594231950; Fri, 30 Nov 2018 08:10:31 -0800 (PST) Received: from [10.132.32.147] ([159.140.254.99]) by smtp.gmail.com with ESMTPSA id e5sm1120667ios.50.2018.11.30.08.10.30 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 30 Nov 2018 08:10:31 -0800 (PST) From: Phil Franklin Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable Mime-Version: 1.0 (Mac OS X Mail 11.2 \(3445.5.20\)) Subject: Re: Beam Metrics questions Message-Id: <6904A07F-D119-4B35-A4D7-2915D63D317F@gmail.com> Date: Fri, 30 Nov 2018 10:10:30 -0600 To: user@beam.apache.org X-Mailer: Apple Mail (2.3445.5.20) Hi again, Etienne! You didn=E2=80=99t mention whether spark is = reporting committed values now, but you also didn=E2=80=99t mention = opening a ticket concerning the spark output. Am I right in inferring = that spark does in fact report committed values? Thanks! -Phil On 2018/11/30 13:57:43, Etienne Chauchot wrote:=20 > Hi Phil,>=20 > Thanks for using MetricsPusher and Beam in general ! >=20 > - MetricsHttpSink works that way: it filters out committed metrics = from the json output when committed metrics are not>=20 > supported. I checked, Flink runner still does not support committed = metrics. So there should be no committed metrics>=20 > values in the output json.There might be a bug. I'll open a ticket: = thx for pointing out ! You tested on flink and spark>=20 > right? and both output committed metrics values right?>=20 > - there is no default mechanism to fallback committed metrics values = on attempted ones>=20 > - Apache Flink does no make flink Accumulators available in detached = mode, so indeed, metrics are not available in this>=20 > mode.>=20 > CCing dev list.>=20 > Etienne>=20 > Le lundi 26 novembre 2018 =C3=A0 15:57 -0600, Phil Franklin a =C3=A9crit= :>=20 > > All of the discussion I=E2=80=99ve seen says that Flink and Spark = only provided attempted metric values, but when I use>=20 > > MetricsHttpSink and look at the JSON it has both attempted and = committed values (albeit, both the same for my simple>=20 > > testing). Has the metrics processing been updated recently, and = I=E2=80=99m just missing the change updates? Or are the>=20 > > committed values being defaulted to the attempted values? >=20 > > >=20 > > Also, I=E2=80=99ve seen it mentioned that Flink doesn=E2=80=99t = report metrics when in detached mode. Is this still the case?>=20 > > >=20 > > >=20 > > Thanks for your help!>=20 >=20=