Mailing-List: contact user-help@flink.apache.org; run by ezmlm
Precedence: bulk
Subject: Re: metrics for Flink sinks
To: user@flink.apache.org
References: <CAMUrgW83F79OTH3YEKoByLM4=-pMwfEdb7QEouwHvq_uiJbu=w@mail.gmail.com>
From: Chesnay Schepler <chesnay@apache.org>
Message-ID: <c5484b4c-6608-f7c3-074c-304cc35caec2@apache.org>
Date: Tue, 29 Aug 2017 21:57:09 +0200
User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:52.0) Gecko/20100101
 Thunderbird/52.2.1
MIME-Version: 1.0
In-Reply-To: <CAMUrgW83F79OTH3YEKoByLM4=-pMwfEdb7QEouwHvq_uiJbu=w@mail.gmail.com>
Content-Type: multipart/alternative;
 boundary="------------93805F126D638414FC1A397A"
Content-Language: en-US
archived-at: Tue, 29 Aug 2017 19:57:18 -0000

This is a multi-part message in MIME format.
--------------93805F126D638414FC1A397A
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 7bit

Hello,

1. Because no one found time to fix it. In contrast to the remaining 
byte/record metrics, input metrics for sources / output metrics for 
sinks have to be implemented for every single implementation with their 
respective semantics. In contrast, the output metrics are gathered in 
the intersection between operators, independent of the actual operator 
implementation. Furthermore, this requires system metrics (i.e. metrics 
that Flink itself creates) to be exposed (and be mutable!) to 
user-defined functions, which is something i /generally /wanted to 
avoid, but it appears to be a big enough pain point to make an exception 
here.

2. Due to the above it is currently not possible without modifications 
of the code to know how many reads/writes were made.

3. Do you mean aggregated metrics? The web UI allows the aggregation of 
record/byte metrics on the task level. Beyond that we defer aggregation 
to actual time-series databases that specialize in these things.

On 28.08.2017 19:08, Martin Eden wrote:
> Hi all,
>
> Just 3 quick questions both related to Flink metrics, especially 
> around sinks:
>
> 1. In the Flink UI Sources always have 0 input records / bytes and 
> Sinks always have 0 output records / bytes? Why is it like that?
>
> 2. What is the best practice for instrumenting off the shelf Flink sinks?
>
> Currently the only metrics available are num records/bytes in and out 
> at the operator and task scope. For the task scope there are extra 
> buffer metrics. However the output metrics are always zero (see 
> question 1). How can one know the actual number of successful writes 
> done by an off the shelf Flink sink? Or the latency of the write 
> operation?
>
> 3. Is it possible to configure Flink to get global job metrics for all 
> subtasks of an operator? Or are there any best practices around that?
>
> Thanks,
> M


--------------93805F126D638414FC1A397A
Content-Type: text/html; charset=utf-8
Content-Transfer-Encoding: 7bit

<html>
  <head>
    <meta http-equiv="Content-Type" content="text/html; charset=utf-8">
  </head>
  <body text="#000000" bgcolor="#FFFFFF">
    <div class="moz-cite-prefix">Hello,<br>
      <br>
      1. Because no one found time to fix it. In contrast to the
      remaining byte/record metrics, input metrics for sources / output
      metrics for sinks have to be implemented for every single
      implementation with their respective semantics. In contrast, the
      output metrics are gathered in the intersection between operators,
      independent of the actual operator implementation. Furthermore,
      this requires system metrics (i.e. metrics that Flink itself
      creates) to be exposed (and be mutable!) to user-defined
      functions, which is something i <i>generally </i>wanted to
      avoid, but it appears to be a big enough pain point to make an
      exception here.<br>
      <br>
      2. Due to the above it is currently not possible without
      modifications of the code to know how many reads/writes were made.<br>
      <br>
      3. Do you mean aggregated metrics? The web UI allows the
      aggregation of record/byte metrics on the task level. Beyond that
      we defer aggregation to actual time-series databases that
      specialize in these things.<br>
      <br>
      On 28.08.2017 19:08, Martin Eden wrote:<br>
    </div>
    <blockquote type="cite"
cite="mid:CAMUrgW83F79OTH3YEKoByLM4=-pMwfEdb7QEouwHvq_uiJbu=w@mail.gmail.com">
      <div dir="ltr">Hi all,
        <div><br>
        </div>
        <div>Just 3 quick questions both related to Flink metrics,
          especially around sinks:</div>
        <div><br>
        </div>
        <div>1. In the Flink UI Sources always have 0 input records /
          bytes and Sinks always have 0 output records / bytes? Why is
          it like that?</div>
        <div><br>
        </div>
        <div>2. What is the best practice for instrumenting off the
          shelf Flink sinks?</div>
        <div><br>
        </div>
        <div>Currently the only metrics available are num records/bytes
          in and out at the operator and task scope. For the task scope
          there are extra buffer metrics. However the output metrics are
          always zero (see question 1). How can one know the actual
          number of successful writes done by an off the shelf Flink
          sink? Or the latency of the write operation?</div>
        <div><br>
        </div>
        <div>3. Is it possible to configure Flink to get global job
          metrics for all subtasks of an operator? Or are there any best
          practices around that?</div>
        <div><br>
        </div>
        <div>Thanks,</div>
        <div>M</div>
      </div>
    </blockquote>
    <p><br>
    </p>
  </body>
</html>

--------------93805F126D638414FC1A397A--