apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ganelin, Ilya" <Ilya.Gane...@capitalone.com>
Subject Re: Creating a common metrics server for internal monitoring
Date Wed, 18 Nov 2015 00:12:22 GMT
I totally agree - I think any such functionality should be a configurable
setting, and should not necessarily apply to every single tuple but
perhaps be periodically computed for “test” tuples flowing through the
system. 


On 11/17/15, 4:10 PM, "Pramod Immaneni" <pramod@datatorrent.com> wrote:

>I am wondering if accounting latency at a tuple level will add too much
>overhead. It might be better to do it on a conditional basis.
>
>On Tue, Nov 17, 2015 at 3:48 PM, Ganelin, Ilya
><Ilya.Ganelin@capitalone.com>
>wrote:
>
>> Specifically, I¹m interested in being able to track latencies for
>> individual tuples. For example, as they move through the BufferServer or
>> the CircularBuffer.
>>
>> On 11/17/15, 3:12 PM, "David Yan" <david@datatorrent.com> wrote:
>>
>> >Hi Ilya,
>> >
>> >The AutoMetric interface is mostly designed for application-specific
>> >metrics within an operator -- things like how many frauds are
>>detected, or
>> >the dollar amount processed.
>> >
>> >We do have BufferServer bytes and queue tuple count (a.k.a. queue
>>size) as
>> >system metrics. If you have specific metrics that pertain to
>>BufferServer
>> >and CircularBuffer that you're interested in, please state them.  We
>> >should
>> >probably add them to system metrics since it's probably useful in
>>general.
>> >
>> >David
>> >
>> >On Tue, Nov 17, 2015 at 2:58 PM, Pramod Immaneni
>><pramod@datatorrent.com>
>> >wrote:
>> >
>> >> Some of them are available in system stats like port queue size and
>>are
>> >> exposed via REST and to a stats listener that you can register via
>>the
>> >>API.
>> >> Some of the others you mentioned can be added.
>> >>
>> >> On Tue, Nov 17, 2015 at 2:52 PM, Ganelin, Ilya <
>> >> Ilya.Ganelin@capitalone.com>
>> >> wrote:
>> >>
>> >> > Hi, all ­ presently Apex supports surfacing metrics via the
>> >>AutoMetrics
>> >> > interface, documented here:
>> >> > http://docs.datatorrent.com/autometrics/
>> >> >
>> >> > However, this only provides utility to monitor operators. I would
>> >>like to
>> >> > provide a more general latency suite - fFor example, I would like
>>to
>> >> > instrument metrics around the BufferServer or the CircularQueue.
>>Not
>> >> > everything in Apex is an operator, and there are other components
>>of
>> >>the
>> >> > system around which we may want explicit measurements or metrics.
>>Is
>> >>this
>> >> > something that could exist? I know other projects like Spark have
>> >>similar
>> >> > efforts, although they are equally underdeveloped :)
>> >> > ________________________________________________________
>> >> >
>> >> > The information contained in this e-mail is confidential and/or
>> >> > proprietary to Capital One and/or its affiliates and may only be
>>used
>> >> > solely in performance of work or services for Capital One. The
>> >> information
>> >> > transmitted herewith is intended only for use by the individual or
>> >>entity
>> >> > to which it is addressed. If the reader of this message is not the
>> >> intended
>> >> > recipient, you are hereby notified that any review, retransmission,
>> >> > dissemination, distribution, copying or other use of, or taking of
>>any
>> >> > action in reliance upon this information is strictly prohibited. If
>> >>you
>> >> > have received this communication in error, please contact the
>>sender
>> >>and
>> >> > delete the material from your computer.
>> >> >
>> >>
>>
>> ________________________________________________________
>>
>> The information contained in this e-mail is confidential and/or
>> proprietary to Capital One and/or its affiliates and may only be used
>> solely in performance of work or services for Capital One. The
>>information
>> transmitted herewith is intended only for use by the individual or
>>entity
>> to which it is addressed. If the reader of this message is not the
>>intended
>> recipient, you are hereby notified that any review, retransmission,
>> dissemination, distribution, copying or other use of, or taking of any
>> action in reliance upon this information is strictly prohibited. If you
>> have received this communication in error, please contact the sender and
>> delete the material from your computer.
>>
>>

________________________________________________________

The information contained in this e-mail is confidential and/or proprietary to Capital One
and/or its affiliates and may only be used solely in performance of work or services for Capital
One. The information transmitted herewith is intended only for use by the individual or entity
to which it is addressed. If the reader of this message is not the intended recipient, you
are hereby notified that any review, retransmission, dissemination, distribution, copying
or other use of, or taking of any action in reliance upon this information is strictly prohibited.
If you have received this communication in error, please contact the sender and delete the
material from your computer.
Mime
View raw message