Mailing-List: contact dev-help@flink.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@flink.apache.org
Subject: Re: Question about Flink Metrics
To: dev@flink.apache.org
References: <CAKhqdDz36-Xm+rdUJK1G0zv1FXkSq6LY=yvQMskrn473csWm7Q@mail.gmail.com>
 <8E213873-50D2-4157-B5DE-13E462307BE6@gmail.com>
 <CAKhqdDxFKRigX86-BnPDf-Zq2shwg0sMZ8P0fqmCXom84Cf86w@mail.gmail.com>
From: Chesnay Schepler <chesnay@apache.org>
Message-ID: <255a975f-93ae-0079-9afa-26e40c76b65a@apache.org>
Date: Tue, 26 Sep 2017 15:57:04 +0200
User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:52.0) Gecko/20100101
 Thunderbird/52.2.1
MIME-Version: 1.0
In-Reply-To: <CAKhqdDxFKRigX86-BnPDf-Zq2shwg0sMZ8P0fqmCXom84Cf86w@mail.gmail.com>
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 8bit
Content-Language: en-US
archived-at: Tue, 26 Sep 2017 13:57:10 -0000

Hello,

i see the value in supporting this, and it's also quite easy to do so 
actually.

I've filed FLINK-7692, containing instructions on how to implement this.

@Tony Are you interested in implementing this?


On 26.09.2017 14:10, Tony Wei wrote:
> Hi Hai Zhou,
>
> It's a good idea to implement my own reporter, but I think it is not the
> best solution.
> After all, reporter needs to be set well when starting the cluster. It is
> not efficient to update cluster whenever you have a new metric for a new
> streaming job.
>
> Anyway, it is still a workaround for now. Thank you!
>
> Best Regards,
> Tony Wei
>
>
> 2017-09-26 19:13 GMT+08:00 Hai Zhou <yew1eb@gmail.com>:
>
>> Hi Tony,
>>
>> you can consider implementing a reporter, use a trick to convert the
>> flink's metrics to the structure that suits your needs.
>>
>> This is just my personal practice, hoping to help you.
>>
>> Cheers,
>> Hai Zhou
>>
>>
>> 在 2017年9月26日，17:49，Tony Wei <tony19920430@gmail.com> 写道：
>>
>> Hi,
>>
>> Recently, I am using PrometheusReporter to monitor every metrics from
>> Flink.
>>
>> I found that the metric name in Prometheus will map to the identifier from
>> User Scope and System Scope [1], and the labels will map to Variables [2].
>>
>> To monitor the same metrics from Prometheus, I would like to use labels
>> to differentiate them.
>> Under the job/task/operator scope, it words fine to me. However, its not
>> convenient to me to monitor partitions' states from Kakfa consumer, because
>> I couldn't place partition id like a tag on each metric. All partition
>> states like current commit offset will be a unique metric in Prometheus.
>> It's hard to use visualization tool such as Grafana to monitor them.
>>
>> My question is: Is it possible to add tags on Metric, instead of using
>> `.addGroup()`?
>> If not, will it be a new feature on Flink Metrics in the future? Since I
>> am not sure about how other reporters work, I am afraid that it is not a
>> good design to just fulfill the requirement on particular reporter.
>>
>> Please guide and thanks for your help.
>>
>> Best Regards,
>> Tony Wei
>>
>> [1]: https://ci.apache.org/projects/flink/flink-docs-
>> release-1.3/monitoring/metrics.html#scope
>> [2]: https://ci.apache.org/projects/flink/flink-docs-
>> release-1.3/monitoring/metrics.html#list-of-all-variables
>>
>>
>>