From user-return-24612-archive-asf-public=cust-asf.ponee.io@flink.apache.org Wed Nov 28 16:01:50 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id C032F180658 for ; Wed, 28 Nov 2018 16:01:49 +0100 (CET) Received: (qmail 28527 invoked by uid 500); 28 Nov 2018 15:01:48 -0000 Mailing-List: contact user-help@flink.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Delivered-To: mailing list user@flink.apache.org Received: (qmail 28517 invoked by uid 99); 28 Nov 2018 15:01:48 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 28 Nov 2018 15:01:48 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id 98D82CFAA3 for ; Wed, 28 Nov 2018 15:01:47 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.8 X-Spam-Level: * X-Spam-Status: No, score=1.8 tagged_above=-999 required=6.31 tests=[DKIMWL_WL_MED=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=0.001, RCVD_IN_MSPIKE_WL=0.001, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd1-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id Z7H3LVMgAnXA for ; Wed, 28 Nov 2018 15:01:45 +0000 (UTC) Received: from mail-yw1-f45.google.com (mail-yw1-f45.google.com [209.85.161.45]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id 032E060D24 for ; Wed, 28 Nov 2018 15:01:45 +0000 (UTC) Received: by mail-yw1-f45.google.com with SMTP id f65so10812850ywc.8 for ; Wed, 28 Nov 2018 07:01:44 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=cc83uumR0uEDNwCF5BFliWRYpwhIlBHkUnuIKstJCBE=; b=KwdIVTBvSYQk4d3hncG+JJCON1g4ZqMIdGmjwMudqrkIfVI4QpoSIwNuLo7WKzCJII s93SnFgfMGQV5NIx5hK7Gj1dG9csuuiMhRT2dw2flaVCkWy0BaHoEeb3aKlQfIasqcaY 8oqoKUav2ozoy+rxWV6yLr9M/Emfe7NDrJGQNdA6ziRg32jRCpBeBsBVSrHZD9HW9TAE evvCLgrPkkL2x62oaxfk5qqzawDGroVUCfUYMbaq/cwO4DTTPPFvUYB6fTuPnaMMKHrY rNnoM4Bync3UcLBvtQ/FQ5LFHdKWIfvsAYEkk99pLAYlTSisC5MhGO92XgBP331IuvMS OqYA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=cc83uumR0uEDNwCF5BFliWRYpwhIlBHkUnuIKstJCBE=; b=L81BfF+JVlTgXD1RJajarBNAOq9xgBp3I0P9sc6XMQtLiUxVMQileo9i1JG7w/tLMx XDaCBK1LrsW/z34zhk5/+/3lG1FjVY4pFt3I2ie/SBjnTkLuH1CDsXBGoydBUINJT/DL xhAL7i1WQLK63+jvORlAh5kuRsQjFZbn7yH3VACka366sbwf4lSGTx/k+91FFzhIPwgI Bi6gj6bBId1BY3tL6HK7We9lnkQzmgf8KCL65oeEphPL3fPDLXoX2nIJAzxl6iar+CK3 icMfK0j+tC3YIYMLGSTqJYil8HjGylRVbH8opcv88LAom9duFV//jalEreAVIDy9X2HM Jd+Q== X-Gm-Message-State: AGRZ1gLDEnE38EfKDetrL4hbVkM6ypr35f03wXw5JlIgXYK+wKGoBQw8 Vf6E7dBnTFHhJ8xTHJM0PdCW7rkwDxE4yeOYdy4= X-Google-Smtp-Source: AJdET5emOHzB4zs/RAqlIYimCxX8jzfPvtQJKe2/hAoxg9QF7ipjfY1N6l6NAJWH3IKsIVRkMMYbMbjN/h+ECBc5MEo= X-Received: by 2002:a81:4757:: with SMTP id u84-v6mr39826317ywa.343.1543417298326; Wed, 28 Nov 2018 07:01:38 -0800 (PST) MIME-Version: 1.0 References: <103760d1-c6fc-04f8-fe95-303e244a6c23@apache.org> <4546c7a1-156a-482d-2ca9-65cacfa2f0ba@apache.org> In-Reply-To: <4546c7a1-156a-482d-2ca9-65cacfa2f0ba@apache.org> From: bastien dine Date: Wed, 28 Nov 2018 16:01:26 +0100 Message-ID: Subject: Re: Flink - Metric are not reported To: chesnay@apache.org Cc: user@flink.apache.org Content-Type: multipart/alternative; boundary="000000000000ce9ab0057bbad72b" --000000000000ce9ab0057bbad72b Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Yea, that was I was thinking.. Batch can be quick Can I report metric on "added" action ? (i should override the notifyOnAddedMetric to report ?) ------------------ Bastien DINE Data Architect / Software Engineer / Sysadmin bastiendine.io Le mer. 28 nov. 2018 =C3=A0 15:54, Chesnay Schepler a =C3=A9crit : > How quick does job batch terminate? Metrics are unregistered once the job > ends; if the job duration is shorter than the report interval they may > never be exposed. > > On 28.11.2018 15:18, bastien dine wrote: > > Hello Chesnay, > > Thanks for your response ! > I have logs enable (info), slf4jReporter is working, I can see : > > 15:16:00.112 [Flink-MetricRegistry-thread-1] INFO > org.apache.flink.metrics.slf4j.Slf4jReporter - > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D Starting metrics report > =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D > > -- Counters > ------------------------------------------------------------------- > > -- Gauges > --------------------------------------------------------------------- > .... > > On both jobmanager & taskmanager, > *BUT* i see only system metrics, not my custom one.. > Am i missing something in declaration in my topology ? > > *Note* : I am using DataSet API (so my program is batch, and not > continuous) > > Regards, > Bastien > > ------------------ > > Bastien DINE > Data Architect / Software Engineer / Sysadmin > bastiendine.io > > > Le mar. 27 nov. 2018 =C3=A0 17:07, Chesnay Schepler = a > =C3=A9crit : > >> Please enable WARN logging and check for warnings by the SLF4JReporter >> and/or MetricQueryService. >> >> On 27.11.2018 17:00, bastien dine wrote: >> >> Hello everyone, >> >> Once again I require your help ! >> I am trying to report custom metric (see my code below) >> Yet, I do not see them anywhere.. nor in the metric tab from my tasks, >> nor in the rest API, nor in the declared slf4j reporter.. >> Can someone help me to debug this .. >> >> Here is my RichMap function : >> >> public class MetricGaugeRichMap> extends Ric= hMapFunction { >> >> private transient T valueToExpose; >> private final String metricGroup; >> private final String metricName; >> >> >> public MetricGaugeRichMap(String metricGroup, String metricName) { >> this.metricGroup =3D metricGroup; >> this.metricName =3D metricName; >> } >> >> @Override public void open(Configuration config) { >> getRuntimeContext() >> .getMetricGroup() >> .addGroup(metricGroup) >> .gauge(metricName, (Gauge) () -> valueToExpose); >> } >> >> @Override public E map(E metricTuple) throws Exception { >> valueToExpose =3D metricTuple.getMetricValue(); >> return metricTuple; >> } >> } >> >> calling from topology : >> >> env.fromElements(new MetricTuple<>(metricGroup, metricName, metricValue)= ) >> .map(new MetricGaugeRichMap<>(metricGroup, metricName)) >> .output(new MetricGaugeOutputFormat<>()); // dummy output >> >> ------------------ >> >> >> Bastien DINE >> Data Architect / Software Engineer / Sysadmin >> bastiendine.io >> >> >> > --000000000000ce9ab0057bbad72b Content-Type: text/html; charset="UTF-8" Content-Transfer-Encoding: quoted-printable
Yea, that was I was thinking..=C2=A0
Batch can be quic= k
Can I report metric on "added" action ? (i should ove= rride the notifyOnAddedMetric to report ?)

=
------------------

Bast= ien DINE
Data Architect / Software Engineer / Sysadmin
bastiendine.io
<= /div>


Le=C2=A0mer. 28 nov. 2018 =C3=A0= =C2=A015:54, Chesnay Schepler <che= snay@apache.org> a =C3=A9crit=C2=A0:
=20 =20 =20
How quick does job b= atch terminate? Metrics are unregistered once the job ends; if the job duration is shorter than the report interval they may never be exposed.

On 28.11.2018 15:18, bastien dine wrote:
Hello Chesnay,=C2=A0

Thanks for your response !
I have logs enable (info), slf4jReporter is working, I can see :

15:16:00.112 [Flink-MetricRegistry-thread-1] INFO=C2=A0 org.apache.flink.metrics.slf4j.Slf4jReporter=C2=A0 -
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D=3D Starting metrics report =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D= =3D=3D=3D=3D=3D=3D=3D

-- Counters -------------------------------------------------------------= ------

-- Gauges -------------------------------------------------------------= --------
....

On both jobmanager & taskmanager,=C2=A0
BUT i see only system metrics, not my custom one..
Am i missing something in declaration in my topology ?

Note : I am using DataSet API (so my program is batch, and not continuous)

Regards,
Bastien

------------------

Bastien DINE
Data Architect / Software Engineer / Sysadmin
bastiendine.io


Le=C2=A0mar. 27 nov. 2018 =C3=A0=C2=A017:07, Chesn= ay Schepler <chesn= ay@apache.org> a =C3=A9crit=C2=A0:
Please enable WARN logging and check for warnings by the SLF4JReporter and/or MetricQueryService.

On 27.11.2018 17:00, bastien dine wrote:
Hello everyone,

Once again I require your help !
I am trying to report custom metric (see my code below)
Yet, I do not see them anywhere.. nor in the metric tab from my tasks, nor in the rest API, nor in the declared slf4j reporter..
Can someone help me to debug this ..

Here is my RichMap function :
public c=
lass MetricGaugeRichMap<T, E extends MetricTuple<T>> extends RichMapFunction<E, E>=
 {

    private transient <=
/span>T valueToExpose;
    private final String metricGroup=
;
    private final String metricName<=
/span>;


    public Metri=
cGaugeRichMap(String metricGroup, String metricName) {
        this.metricGroup =3D =
metricGroup;
        this.metricName =3D m=
etricName;
    }

    @Override
    public void open(Configuration confi=
g) {
        getRuntimeContext()
                .getMetricGroup()
                .addGroup(metricGroup)
                .gauge(metricName, (Gauge<T>) () -> va=
lueToExpose);
    }

    @Override
    public E map(E metric=
Tuple) throws Ex=
ception {
        valueToExpos=
e =3D metricTuple.getMetricValue();
        return m=
etricTuple;
    }
}
calling from topology :
env.fromElements(new MetricTuple<&g=
t;(metricGroup, metricName=
, metricValue=
))
    .map(new Met=
ricGaugeRichMap<>(metricGroup, metricName))
    .output(new =
MetricGaugeOutputFormat<>()); // dummy output
------------------

Bastien DINE
Data Architect / Software Engineer / Sysadmin
bastiendine.io



--000000000000ce9ab0057bbad72b--