openwhisk-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tyson Norris <>
Subject Re: Replace log markers with metrics sent to statsd
Date Wed, 27 Sep 2017 17:44:00 GMT
Thanks Vadmin!

A few comments:

- RE: async logging:  I thought the akka logger was already async? This is different than
the logback async appender. I’m OK with using the async appender, but people should know
that it by default is configured to ignore events once its buffer is full, so its possible
to get into a situation where log messages “go missing” if faced with sufficient traffic.

- RE: kamon: It’s not clear to me that kamon supports timing data for general methods, although
it looks like it keeps timing data for akka message processing.
Currently the log markers in OW keep timing data for arbitrary blocks of code (unrelated to
akka). It looks like to get “time in method” metrics you will still need to measure the
time, and then record that time using kamon.

I know Sandeep is looking into using the timing/tracing aspect as part of this PR

It may be interesting to consider kamon TraceContext for this, but it is also somewhat invasive
- seems like it requires wrapping all Future usage as in:
Tracer.withNewContext("sample-trace") {
    Future {}

Is there a way to use this without this wrapping?

BTW - I agree that any “counter” type metrics should be replaced and kamon seems good
for this, I’m just not sure about the timing markers.

- RE: first steps: Does changing log library or appender config imply the need to change the
current marker/timing impl? I think a few logical steps are:
- replace logging impl (with logback)
- reduce log generation
- replace log marker counter impl (with kamon?)
- replace logging approach (with TraceContext or similar, if an option even exists, with the
goal being to NOT pass whisk Logging around everywhere)

(I’m really not sure about the last one, but it would be nice to a less invasive way to
emit logs with context)


On Sep 27, 2017, at 7:39 AM, Vadim Raskin <<>>

Hi OpenWhisk community,

while doing our profiling tests on the controller we have found that the
way we log and the logging per se make a huge impact on the controller
thoughput. In a short, the more parallel activations you have, the more
blocking threads waiting for the logger will be there. After doing several
experiments we found a couple of action items:
* replace the logging library (e.g. to logback), use an async appender and
increase the size of the LogAppender buffer.
* reduce the number of logs that we generate.

Since the first action item will only provide a partial relief and postpone
the point when we start observing the degradation, reducing the number of
logs seems to be a more plausible way to go.

One of the first points I’d like to address are the metrics that we append
to log messages in a form of markers.

The initial idea is to send them directly to statsd or some other backend.
For example Kamon library is a good candidate:

Here is the issue I’ve opened some time ago.

Do you have any suggestion/concerns regarding that?

Thank you.

regards, Vadim.

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message