openwhisk-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "James W Dubee" <jwdu...@us.ibm.com>
Subject Activation Store Service Provider Interface
Date Wed, 16 May 2018 20:46:27 GMT


Fellow developers,

Currently the activation store is tightly coupled with the artifact store
utilized by an OpenWhisk deployment. Meaning the controller must use the
same data store, CouchDB for example, utilized by an OpenWhisk instance to
store and retrieve activation details. However, an invoker can use a
customized log store implementation to store user logs and activation
records in a data store that is independent of the primary database. Such
differences in functionality between the controller and invoker with
regards to activation storage and retrieval is problematic.


Activation records are special as they hold meaningful details about the
execution behavior of actions, triggers, rules, and sequences. Ideally
users should be able to run customized queries on these activation records
in order to find out the execution time of a group of entities, if a group
of entities were executed successfully, etc. By default, CouchDb does not
allow such customized queries to be performed. To provide user defined
queries on activations, we have already provided a log store that can be
customized per-deployment via a Service Provider Interface (SPI). Currently
we have log store implementations for Elasticsearch and Splunk. Both of
these services allow user queries.


Unfortunately these log store implementations apply mostly to the invoker.
For instance, the controller can only fetch logs from the log store using
the activation logs API. While the rest of the activation APIs must utilize
the artifact store. Activation records must be in the artifact store in
order for most of the controller activation APIs to return activation
information, even though those same activation records may have also been
saved in a different backing store by the log store. Consequently,
activation records and logs may be duplicated in the log store and artifact
store. Another problem revolves around the controller writing user logs.
User logs generated by the controller for triggers and sequences are only
written to the artifact store even if a non-default log store is being
used. Therefore, users cannot run customized queries on user logs generated
by the controller.

To eliminate the duplication of activations, and controller user logs being
inaccessible to the log store, an activation store SPI can be provided.
When configured appropriately the activation store would be able to utilize
the same data store used by the log store. This would eliminate duplication
of stored activation information in separate databases, and allow the log
store access to user logs generated by the controller.


Work for providing the activation store that is compatible with the
artifact store has already started. The related PR can be found here:
https://github.com/apache/incubator-openwhisk/pull/3619. Providing an
activation store for Elasticsearch will follow.


Regards,
James Dubee

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message