openwhisk-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rodric Rabbah <rod...@gmail.com>
Subject Re: Sending activation metadata to Kafka
Date Tue, 17 Apr 2018 19:02:15 GMT
It would be useful to provide a reference implementation for consuming this data. Can you also
capture the goals in an issue (I scanned the PR quickly but there’s no corresponding issue).

Further now I think there are multiple ways of recording/reporting some of the metrics (log
markers which are largely silenced by default, kamon metrics, and now kafka). Is that right?

I think we’ll need to also document these and a guide for when to use which - I caution
that we are proliferating multiple ways of doing similar things with no consistency or articulated
long term vision. 

-r

> On Apr 17, 2018, at 12:01 PM, Vadim Raskin <raskinvadim@gmail.com> wrote:
> 
> Hi Chetan,
> 
>> Can you share some details on how this is currently being done with
> CouchDB. Do we have any analytics view configured which computes these
> numbers currently?
> 
> To my knowledge we don't have the views that are shared anywhere in open
> repos.
> 
>> May be we also include a basic default implementation out of the box
> which collect aggregated stats using Kamon metrics already being used
> 
> Not a bad idea, I'll consider sharing the peace of code after finishing the
> development (separate from the original PR), it might require some
> post-processing to strip away the ibm specific parts, so it might bide some
> time.
> 
> regards,
> Vadim.
> 
> 
> 
> On Tue, Apr 17, 2018 at 8:29 AM Chetan Mehrotra <chetan.mehrotra@gmail.com>
> wrote:
> 
>> Hi Vadim,
>> 
>> This looks helpful to get better insight in runtime operational stats!
>> 
>>> It has some advantages over to the prior approach (sending them to
>> CouchDB).
>> 
>> Can you share some details on how this is currently being done with
>> CouchDB. Do we have any analytics view configured which computes these
>> numbers currently?
>> 
>>> Now it would be possible to simply connect a custom micro service
>> to Kafka and consume the activations in real-time.
>> 
>> May be we also include a basic default implementation out of the box
>> which collect aggregated stats using Kamon metrics already being used
>> Chetan Mehrotra
>> 
>> 
>> On Mon, Apr 16, 2018 at 9:51 PM, Vadim Raskin <raskinvadim@gmail.com>
>> wrote:
>>> Hi everyone,
>>> 
>>> 
>>> I’ve just opened a PR that enables sending activation metadata to Kafka.
>>> It has some advantages over to the prior approach (sending them to
>>> CouchDB). Now it would be possible to simply connect a custom micro
>> service
>>> to Kafka and consume the activations in real-time. Some of the use cases
>> it
>>> might cover: activation metrics - collect the data and push them into a
>>> custom time-series database; user activity audit; activation analytics -
>>> potentially get some insights with KSQL.
>>> 
>>> 
>>> At the moment I’ve created a new kafka topic called events, which will
>>> include messages from Controllers and Invokers. It encompasses the
>>> following data collected from a single activation:
>>> 
>>> 
>>> concurrentActivations
>>> throttledActivations
>>> statusCode
>>> initTime
>>> waitTime
>>> duration
>>> kind
>>> 
>>> 
>>> Probably some more metadata will affiliate this list soon.
>>> 
>>> 
>>> Just wanted to give a short heads up here. The PR I mentioned:
>>> https://github.com/apache/incubator-openwhisk/pull/3552
>>> 
>>> 
>>> Thank you,
>>> 
>>> 
>>> Vadim.
>> 

Mime
View raw message