openwhisk-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vadim Raskin <raskinva...@gmail.com>
Subject Replace log markers with metrics sent to statsd
Date Wed, 27 Sep 2017 14:39:12 GMT
Hi OpenWhisk community,

while doing our profiling tests on the controller we have found that the
way we log and the logging per se make a huge impact on the controller
thoughput. In a short, the more parallel activations you have, the more
blocking threads waiting for the logger will be there. After doing several
experiments we found a couple of action items:
* replace the logging library (e.g. to logback), use an async appender and
increase the size of the LogAppender buffer.
* reduce the number of logs that we generate.

Since the first action item will only provide a partial relief and postpone
the point when we start observing the degradation, reducing the number of
logs seems to be a more plausible way to go.

One of the first points I’d like to address are the metrics that we append
to log messages in a form of markers.

The initial idea is to send them directly to statsd or some other backend.
For example Kamon library is a good candidate: http://kamon.io

Here is the issue I’ve opened some time ago.
https://github.com/apache/incubator-openwhisk/issues/2780

Do you have any suggestion/concerns regarding that?

Thank you.

regards, Vadim.

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message