openwhisk-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "David P Grove" <gro...@us.ibm.com>
Subject Re: Re: Backpressure for slow activation storage in Invoker
Date Fri, 21 Jun 2019 17:29:27 GMT



Rodric Rabbah <rodric@gmail.com> wrote on 06/20/2019 09:37:38 PM:
>
> Overflowing to Kafka (option b) is better. Actually I would dump all
> the activations there and have a separate process to drain that
> Kafka topic to the datastore or logstore.

I agree. Spilling to Kafka is desirable to avoid OOMs in the invoker.

> There is another approach of routing the logs directly to a logstore
> without going through the invoker at all. IBM may have experimented
> with this maybe someone else can comment on that.

In the Kubernetes world (especially with the KubernetesContainerFactory),
this is the only really good way of doing it.

To really do this well, our actions should be required to implement
structured logging.  If every log line had the activationId and namespace
info in it, then the logs could stream from the container through an
efficient OpenWhisk-specific logging agent (I had prototyped an agent using
fluent bit last year) to the platform logging service.

If you don't have structured logging from the actions, you can try to
kludge this flow together in various ways but it gets messier.

--dave

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message