openwhisk-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tyson Norris <tnor...@adobe.com.INVALID>
Subject Re: Re: Backpressure for slow activation storage in Invoker
Date Fri, 21 Jun 2019 18:14:24 GMT
The logs issue is mostly separate from the activation records. 
RE activation records:
Can we handle these in same way as user events? Maybe exactly like user events, as in use
a single service to process both topics.


RE logging:
We deal with logs this way (collect from container via fluent), but the problems we've seen
are:
- forcing structure isn't easy to enforce consistently on all runtimes (we get around this
by mostly only supporting nodejs, java may also not be too bad, if you support a few common
logging libs)
- there are separate issues with concurrency support - for nodejs we use cls-hooked to manage
this context; using environment variables instead of a different main signature is also a
problem for concurrency, and also can be worked around with cls-hooked. Not sure other operators
care about this so maybe it is not an issue except to be documented per runtime.
- many tests currently assume that the logs are immediately available, and this cannot be
the case with decoupled log collection (I've started work to add some retries/delay tolerance,
but it's incomplete)

Thanks
Tyson


´╗┐On 6/21/19, 10:29 AM, "David P Grove" <groved@us.ibm.com> wrote:

    
    
    
    Rodric Rabbah <rodric@gmail.com> wrote on 06/20/2019 09:37:38 PM:
    >
    > Overflowing to Kafka (option b) is better. Actually I would dump all
    > the activations there and have a separate process to drain that
    > Kafka topic to the datastore or logstore.
    
    I agree. Spilling to Kafka is desirable to avoid OOMs in the invoker.
    
    > There is another approach of routing the logs directly to a logstore
    > without going through the invoker at all. IBM may have experimented
    > with this maybe someone else can comment on that.
    
    In the Kubernetes world (especially with the KubernetesContainerFactory),
    this is the only really good way of doing it.
    
    To really do this well, our actions should be required to implement
    structured logging.  If every log line had the activationId and namespace
    info in it, then the logs could stream from the container through an
    efficient OpenWhisk-specific logging agent (I had prototyped an agent using
    fluent bit last year) to the platform logging service.
    
    If you don't have structured logging from the actions, you can try to
    kludge this flow together in various ways but it gets messier.
    
    --dave
    

Mime
View raw message