Mailing-List: contact dev-help@flink.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@flink.apache.org
Subject: Re: Adding custom monitoring to Flink
To: dev@flink.apache.org
References: 
 <CAAqc5=H9V4_Kqsw6vCjmO+u8y9cu4G=ZL6QPrNZ7+1TD=yvfxg@mail.gmail.com>
From: Chesnay Schepler <chesnay@apache.org>
Message-ID: <570FF03B.8050208@apache.org>
Date: Thu, 14 Apr 2016 21:32:11 +0200
User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:38.0) Gecko/20100101
 Thunderbird/38.5.0
MIME-Version: 1.0
In-Reply-To: 
 <CAAqc5=H9V4_Kqsw6vCjmO+u8y9cu4G=ZL6QPrNZ7+1TD=yvfxg@mail.gmail.com>
Content-Type: multipart/alternative;
 boundary="------------060800000301050209070603"

--------------060800000301050209070603
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 7bit

I'm currently working on a metric system that
a) exposes several TaskManger metrics
b) allows gathering metrics in various parts of a task, most notably 
user-defined functions.

The first version makes these metrics available via JMX on each 
TaskManager.
While a mechanism to make that pluggable is /planned/ there are no 
details on that yet.

I /guess/ once it is merged you should be able to modify one of the 
classes so that the data is directly
exported to your tool, but i would have to know more about it to make a 
definite assessment.

There are no plans to funnel all those metrics unaggregated through 
Flink's accumulator mechanism;
only a selection that will be aggregated locally and on the JobManager 
to display in the Dashboard.

Out of curiosity, what metrics are you interested in?

On 14.04.2016 20:59, Maxim wrote:
> Hi!
> I'm looking into integrating Flink into our stack and one of the
> requirements is to report metrics to an internal system. The current
> Accumulators are not adequate to provide visibility that we need to run
> such a system in production. We want much more information about the
> internal cluster state and ability to calculate aggregates ourselves. The
> core reporting API accepts a metric name, metric type (gauge, counter,
> timer) and a set of key value pairs that act as dimensions.
>
> The ideal solution for us would report the metrics through such API and
> provide default binding to existing Accumulators, but allow overriding it
> to our internal reporting client.
>
> Is it something that could be added to the Flink or there are other plans
> for monitoring?
>
> Thanks!
>
> Maxim.
>


--------------060800000301050209070603--