From dev-return-1831-archive-asf-public=cust-asf.ponee.io@openwhisk.apache.org Wed Apr 18 07:21:01 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id 5BCA818064E for ; Wed, 18 Apr 2018 07:21:01 +0200 (CEST) Received: (qmail 17575 invoked by uid 500); 18 Apr 2018 05:21:00 -0000 Mailing-List: contact dev-help@openwhisk.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@openwhisk.apache.org Delivered-To: mailing list dev@openwhisk.apache.org Received: (qmail 17561 invoked by uid 99); 18 Apr 2018 05:20:59 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 18 Apr 2018 05:20:59 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id 05C1EC00DA for ; Wed, 18 Apr 2018 05:20:59 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.879 X-Spam-Level: * X-Spam-Status: No, score=1.879 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H3=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd4-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id 2rqxi86G0hTy for ; Wed, 18 Apr 2018 05:20:55 +0000 (UTC) Received: from mail-lf0-f42.google.com (mail-lf0-f42.google.com [209.85.215.42]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id C554E5F4DC for ; Wed, 18 Apr 2018 05:20:54 +0000 (UTC) Received: by mail-lf0-f42.google.com with SMTP id p142-v6so700236lfd.6 for ; Tue, 17 Apr 2018 22:20:54 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to; bh=GH3YP3wlIqtj8UB2pSVgRR/60o7booyFbLn+NEfdLw0=; b=KGAAXkM29vYU2r576cTFl/ykTrbgSiqIE0DGuzBO8d36L+FSEelhfe2nJ73cuCGLF/ IB8w+FhDuqbm2x59uu2vACZfLUGTM/VbenGMSx7pET4Ns1GBxA+cRmeFi88LQvKlpcT0 zc0SidxOnxl+cbCLzehIxprX+nvspZatYUZLi/rLulLTUie0wwfGuI5e4BA4skQXGSYd ZeE5LCPr3Iw/PDxHHoAWGwrg1RNqCUseVSVHJqreza0ZnJ2J1hmkt3ltZ0sGf/aqGGOZ nhRVLH3s8kvJovJ90PsCzwsFXq4eLYRanJu56fJ6W7ANe8y91q2MVHqW4LCpsKWHoheX hltQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=GH3YP3wlIqtj8UB2pSVgRR/60o7booyFbLn+NEfdLw0=; b=LIG0Thk05DalS6Ey6XUdhYYrqthO0ofSCs+rNibwJ8/twD7S6wjaUviWuvIiXyQxNO S3+42u2rwGMQM+00RbZfE7pvaGX1W+rLdIsVFtW+X1h/597py/9ORgbIMDi0GPO9k/ni B0CGpdSaSpxYUY4Olv+6Bi+erpnY8Tr5Gpo0i8wE8fnSkevsqi4eDixuiO4F2zIiKjCU pnu21eFfdWDXouj+/iWYsMJLq7AnLTnT3vyRxCkgACmcdqlF26+d1A68oSq41YB5AaI9 qXqP7j4aVFVPWCFYWreF/HkLO0lG5CADg16hPBBwaO91Mabi++cfGlXzTLQ2mdCb5h1A 5wCQ== X-Gm-Message-State: ALQs6tAL8JLepOxQal602zfgGjLPW8oihJnBdnkqMMDeynJvZWOzomHC k0nnkNW5dspuh4bSKw+9o0C3sDfGB3wy1r/Qc2I= X-Google-Smtp-Source: AIpwx4/uiv+XaRBJsalmpmfaeqgQ0gjjBPWP+zYTStP8wkPL2pVSlBhGznpkmxfM2Vl2LscH0ZByD5H+W5AzF66+cdg= X-Received: by 10.46.148.202 with SMTP id r10mr378225ljh.22.1524028853219; Tue, 17 Apr 2018 22:20:53 -0700 (PDT) MIME-Version: 1.0 References: <095EB8C5-BA41-46B0-8852-483B9782DC46@gmail.com> In-Reply-To: From: Dascalita Dragos Date: Wed, 18 Apr 2018 05:20:42 +0000 Message-ID: Subject: Re: Sending activation metadata to Kafka To: dev@openwhisk.apache.org Content-Type: multipart/alternative; boundary="f4f5e80c4da06c8eaf056a189e6d" --f4f5e80c4da06c8eaf056a189e6d Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Continuing what Rodric was saying I'd also like to bring into this picture Distributed Tracing from this older PR [1]. A holistic approach of "what to use, when, and for what purpose" would be useful. [1] - https://github.com/apache/incubator-openwhisk/pull/2282 On Tue, Apr 17, 2018 at 2:59 PM Tyson Norris wrote: > I took a brief look at the PR - it looks like =E2=80=9Cthe prior approach= =E2=80=9D of > sending to couchdb is still enabled, is that correct? > > If so, it may be worthwhile to make the reference impl store to couchdb, > and remove the activation persistence from controller/invoker? > > This would also imply that the =E2=80=9Cpolling=E2=80=9D that is controll= er would also > need to be replaced? > Thanks > Tyson > > On Apr 17, 2018, at 12:02 PM, Rodric Rabbah rodric@gmail.com>> wrote: > > It would be useful to provide a reference implementation for consuming > this data. Can you also capture the goals in an issue (I scanned the PR > quickly but there=E2=80=99s no corresponding issue). > > Further now I think there are multiple ways of recording/reporting some o= f > the metrics (log markers which are largely silenced by default, kamon > metrics, and now kafka). Is that right? > > I think we=E2=80=99ll need to also document these and a guide for when to= use > which - I caution that we are proliferating multiple ways of doing simila= r > things with no consistency or articulated long term vision. > > -r > > On Apr 17, 2018, at 12:01 PM, Vadim Raskin raskinvadim@gmail.com>> wrote: > > Hi Chetan, > > Can you share some details on how this is currently being done with > CouchDB. Do we have any analytics view configured which computes these > numbers currently? > > To my knowledge we don't have the views that are shared anywhere in open > repos. > > May be we also include a basic default implementation out of the box > which collect aggregated stats using Kamon metrics already being used > > Not a bad idea, I'll consider sharing the peace of code after finishing t= he > development (separate from the original PR), it might require some > post-processing to strip away the ibm specific parts, so it might bide so= me > time. > > regards, > Vadim. > > > > On Tue, Apr 17, 2018 at 8:29 AM Chetan Mehrotra > > wrote: > > Hi Vadim, > > This looks helpful to get better insight in runtime operational stats! > > It has some advantages over to the prior approach (sending them to > CouchDB). > > Can you share some details on how this is currently being done with > CouchDB. Do we have any analytics view configured which computes these > numbers currently? > > Now it would be possible to simply connect a custom micro service > to Kafka and consume the activations in real-time. > > May be we also include a basic default implementation out of the box > which collect aggregated stats using Kamon metrics already being used > Chetan Mehrotra > > > On Mon, Apr 16, 2018 at 9:51 PM, Vadim Raskin > > wrote: > Hi everyone, > > > I=E2=80=99ve just opened a PR that enables sending activation metadata to= Kafka. > It has some advantages over to the prior approach (sending them to > CouchDB). Now it would be possible to simply connect a custom micro > service > to Kafka and consume the activations in real-time. Some of the use cases > it > might cover: activation metrics - collect the data and push them into a > custom time-series database; user activity audit; activation analytics - > potentially get some insights with KSQL. > > > At the moment I=E2=80=99ve created a new kafka topic called events, which= will > include messages from Controllers and Invokers. It encompasses the > following data collected from a single activation: > > > concurrentActivations > throttledActivations > statusCode > initTime > waitTime > duration > kind > > > Probably some more metadata will affiliate this list soon. > > > Just wanted to give a short heads up here. The PR I mentioned: > > https://na01.safelinks.protection.outlook.com/?url=3Dhttps%3A%2F%2Fgithub= .com%2Fapache%2Fincubator-openwhisk%2Fpull%2F3552&data=3D02%7C01%7Ctnorris%= 40adobe.com%7Cc908a394ad184dffaefe08d5a495c08c%7Cfa7b1b5a7b34438794aed2c178= decee1%7C0%7C0%7C636595885445950512&sdata=3DseMpqJ1pUTYUt2a4XDUcBPTleDNCjxr= uf3HSjx%2BhDk4%3D&reserved=3D0 > > > Thank you, > > > Vadim. > > --f4f5e80c4da06c8eaf056a189e6d--