From dev-return-1902-archive-asf-public=cust-asf.ponee.io@openwhisk.apache.org Wed May 16 22:46:37 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id 41F4B180669 for ; Wed, 16 May 2018 22:46:37 +0200 (CEST) Received: (qmail 87007 invoked by uid 500); 16 May 2018 20:46:36 -0000 Mailing-List: contact dev-help@openwhisk.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@openwhisk.apache.org Delivered-To: mailing list dev@openwhisk.apache.org Received: (qmail 86982 invoked by uid 99); 16 May 2018 20:46:35 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 16 May 2018 20:46:35 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id EAC8BCB2E2 for ; Wed, 16 May 2018 20:46:34 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.299 X-Spam-Level: * X-Spam-Status: No, score=1.299 tagged_above=-999 required=6.31 tests=[HTML_MESSAGE=2, RCVD_IN_DNSWL_LOW=-0.7, SPF_PASS=-0.001] autolearn=disabled Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id TFAM2wCAGIkI for ; Wed, 16 May 2018 20:46:33 +0000 (UTC) Received: from mx0a-001b2d01.pphosted.com (mx0a-001b2d01.pphosted.com [148.163.156.1]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id 6FCC95F47E for ; Wed, 16 May 2018 20:46:32 +0000 (UTC) Received: from pps.filterd (m0098410.ppops.net [127.0.0.1]) by mx0a-001b2d01.pphosted.com (8.16.0.22/8.16.0.22) with SMTP id w4GKigFf063178 for ; Wed, 16 May 2018 16:46:30 -0400 Received: from smtp.notes.na.collabserv.com (smtp.notes.na.collabserv.com [192.155.248.67]) by mx0a-001b2d01.pphosted.com with ESMTP id 2j0qg9m6bp-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Wed, 16 May 2018 16:46:30 -0400 Received: from localhost by smtp.notes.na.collabserv.com with smtp.notes.na.collabserv.com ESMTP for from ; Wed, 16 May 2018 20:46:29 -0000 Received: from us1a3-smtp04.a3.dal06.isc4sb.com (10.106.154.237) by smtp.notes.na.collabserv.com (10.106.227.16) with smtp.notes.na.collabserv.com ESMTP; Wed, 16 May 2018 20:46:28 -0000 Received: from us1a3-mail112.a3.dal06.isc4sb.com ([10.146.6.42]) by us1a3-smtp04.a3.dal06.isc4sb.com with ESMTP id 2018051620462805-1140922 ; Wed, 16 May 2018 20:46:28 +0000 Subject: Activation Store Service Provider Interface To: dev@openwhisk.apache.org From: "James W Dubee" Date: Wed, 16 May 2018 16:46:27 -0400 X-KeepSent: C3C9C5E8:17B5B679-0025828F:0071C1A5; type=4; name=$KeepSent X-Mailer: IBM Notes Release 9.0.1EXT SHF886 February 20, 2018 X-LLNOutbound: False X-Disclaimed: 26567 X-TNEFEvaluated: 1 x-cbid: 18051620-0327-0000-0000-000005A70F48 X-IBM-SpamModules-Scores: BY=0; FL=0; FP=0; FZ=0; HX=0; KW=0; PH=0; SC=0.439231; ST=0; TS=0; UL=0; ISC=; MB=0.092329 X-IBM-SpamModules-Versions: BY=3.00009036; HX=3.00000241; KW=3.00000007; PH=3.00000004; SC=3.00000260; SDB=6.01033288; UDB=6.00528326; IPR=6.00812419; BA=6.00005971; NDR=6.00000001; ZLA=6.00000005; ZF=6.00000009; ZB=6.00000000; ZP=6.00000000; ZH=6.00000000; ZU=6.00000002; MB=3.00021149; XFM=3.00000015; UTC=2018-05-16 20:46:29 X-IBM-AV-DETECTION: SAVI=unsuspicious REMOTE=unsuspicious XFE=unused X-IBM-AV-VERSION: SAVI=2018-05-16 20:17:48 - 6.00008463 x-cbparentid: 18051620-0328-0000-0000-000041DA1022 Message-Id: Content-type: multipart/alternative; boundary="0__=8FBB081CDFE247358f9e8a93df938690918c8FBB081CDFE24735" Content-Disposition: inline MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:,, definitions=2018-05-16_10:,, signatures=0 X-Proofpoint-Spam-Reason: safe --0__=8FBB081CDFE247358f9e8a93df938690918c8FBB081CDFE24735 Content-type: text/plain; charset=US-ASCII Content-Transfer-Encoding: Quoted-printable Fellow developers, Currently the activation store is tightly coupled with the artifact store utilized by an OpenWhisk deployment. Meaning the controller must use the same data store, CouchDB for example, utilized by an OpenWhisk instance to store and retrieve activation details. However, an invoker can use a customized log store implementation to store user logs and activation records in a data store that is independent of the primary database. Such differences in functionality between the controller and invoker with regards to activation storage and retrieval is problematic. Activation records are special as they hold meaningful details about the execution behavior of actions, triggers, rules, and sequences. Ideally users should be able to run customized queries on these activation records in order to find out the execution time of a group of entities, if a group of entities were executed successfully, etc. By default, CouchDb does not allow such customized queries to be performed. To provide user defined queries on activations, we have already provided a log store that can be customized per-deployment via a Service Provider Interface (SPI). Currently we have log store implementations for Elasticsearch and Splunk. Both of these services allow user queries. Unfortunately these log store implementations apply mostly to the invoker. For instance, the controller can only fetch logs from the log store using the activation logs API. While the rest of the activation APIs must utilize the artifact store. Activation records must be in the artifact store in order for most of the controller activation APIs to return activation information, even though those same activation records may have also been saved in a different backing store by the log store. Consequently, activation records and logs may be duplicated in the log store and artifact store. Another problem revolves around the controller writing user logs. User logs generated by the controller for triggers and sequences are only written to the artifact store even if a non-default log store is being used. Therefore, users cannot run customized queries on user logs generated by the controller. To eliminate the duplication of activations, and controller user logs being inaccessible to the log store, an activation store SPI can be provided. When configured appropriately the activation store would be able to utilize the same data store used by the log store. This would eliminate duplication of stored activation information in separate databases, and allow the log store access to user logs generated by the controller. Work for providing the activation store that is compatible with the artifact store has already started. The related PR can be found here: https://github.com/apache/incubator-openwhisk/pull/3619. Providing an activation store for Elasticsearch will follow. Regards, James Dubee --0__=8FBB081CDFE247358f9e8a93df938690918c8FBB081CDFE24735--