openwhisk-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chetan Mehrotra <chetan.mehro...@gmail.com>
Subject Re: Passing TransactionId as part of action invocation
Date Mon, 19 Aug 2019 14:01:57 GMT
> What if we make annotations a dictionary?

I would have preferred that as current structure makes it harder to
process them. However changing them now would cause compatibility
issue with various tooling out there which may be interpreting the
annotation per current design

Chetan Mehrotra

On Mon, Aug 19, 2019 at 6:59 AM Rodric Rabbah <rodric@gmail.com> wrote:
>
> What if we make annotations a dictionary?
>
> -r
>
> On Aug 19, 2019, at 9:54 AM, Chetan Mehrotra <chetan.mehrotra@gmail.com> wrote:
>
> >> I second the idea of recording the request/transaction id as part of the activation
record. We don’t currently do that,  but we do have a caused-by annotation for composite
activations. I suggest another annotation which is the transaction id.
> >
> > There are few other cases where I felt a need to record some "meta"
> > ids in activation
> >
> > 1. While integrating with Lambda or other such compute stack I would
> > like to save the responseId from the backend system in activation so
> > as to enable fetching logs later based on the responseId from previous
> > call.
> > 2. In Mesos or k8s setup one may want to record the Mesos taskId, k8s
> > podId for correlating with some system logs later
> >
> > Should we consider it as yet another annotation or add new sub
> > document `meta` and make it a property there?
> >
> > If we make it an annotation then it becomes tricky to index this
> > property in db due to the document structure. For couchdb you can do
> > it by extracting the key when generating the view but for other db's
> > we typically have to add a computed field at time of persistence.
> >
> > Another option would be to introduce a new sub structure say `meta`
> > where we store such ids as flat keys
> >
> > "meta" : {
> >            "transactionId" : "xxx",
> >            "podId" : "ow_xxx"
> >        }
> >
> > Such an structure would be easier to index compared to one based on annotations
> >
> > Chetan Mehrotra
> >
> >    {
> >        "activationId": "e40c9340aea340f58c9340aea320f5e9",
> >        "annotations": [
> >            {
> >                "key": "causedBy",
> >                "value": "sequence"
> >            },
> >            {
> >                "key": "kind",
> >                "value": "nodejs:10"
> >            },
> >            {
> >                "key": "timeout",
> >                "value": false
> >            },
> >            {
> >                "key": "limits",
> >                "value": {
> >                    "concurrency": 200,
> >                    "logs": 10,
> >                    "memory": 256,
> >                    "timeout": 60000
> >                }
> >            }
> >        ],
> >        "cause": "f3380aaeca7d4791b80aaeca7d5791f6",
> >        "duration": 5015,
> >        "end": 1562344247086,
> >        "entityType": "activation",
> >        "response": { ... },
> >        "start": 1562344242071,
> >
> >    }
> >
> >> On Mon, Aug 19, 2019 at 5:34 AM Rodric Rabbah <rodric@gmail.com> wrote:
> >>
> >> I second the idea of recording the request/transaction id as part of the activation
record. We don’t currently do that,  but we do have a caused-by annotation for composite
activations. I suggest another annotation which is the transaction id.
> >>
> >> Chetan’s pr is straightforward and I think achieves the goal of linking parents
to children for external tracing.
> >>
> >> -r
> >>
> >>> On Aug 19, 2019, at 5:02 AM, Erez Hadad <EREZH@il.ibm.com> wrote:
> >>>
> >>> Hi folks,
> >>>
> >>> My two cents: recall my "shared context" presentation about using a
> >>> semi-transparent execution context that is shared across multiple
> >>> consequent invocations. Similar to the transaction id being discussed.
> >>> https://cwiki.apache.org/confluence/download/attachments/74689638/Shared%20Context%20in%20OpenWhisk%20Invocations.pptx?api=v2
> >>>
> >>> Specifically, you may want to consider the implementation options in slide
> >>> 11:
> >>> 1. One particular option of interest is to embed the shared context id (
=
> >>> transaction id) in the activation id. When the new activation id is
> >>> generated, the transaction id can be taken from the caller's activation
id
> >>> and embedded in the new activation id. This way the activation id remains
> >>> unique per invocation but the transaction id can be extracted from it if
> >>> needed, manually or via a client library (e.g., extending the OW js
> >>> library). Kind of a minimal change.
> >>> 2. Another possibly valuable option is to have the transaction id used as
> >>> a key for retrieving data (e.g., key/value) from a 3rd-party fast service,
> >>> such as redis. This can be left open for the user of the transaction id
or
> >>> provide a reference implementation in the client library. Such data can
> >>> also be pre-fetched before the invocation starts (e.g., from CouchDB /
> >>> Redis), but again, at the cost of a higher code change.
> >>>
> >>> Regards,
> >>> -- Erez
> >>>
> >>>
> >>>
> >>>
> >>>
> >>>
> >>> From:   Rodric Rabbah <rodric@gmail.com>
> >>> To:     dev@openwhisk.apache.org
> >>> Cc:     tyson.norris@gmail.com
> >>> Date:   17/08/2019 03:36
> >>> Subject:        [EXTERNAL] Re: Passing TransactionId as part of action
> >>> invocation
> >>>
> >>>
> >>>
> >>> Of course if the transaction id is the same as the activation id (of the
> >>> composite action) the solutions are comparable.
> >>>
> >>> The downside of using transaction id is that we have no APIs to query by
> >>> it today, and it?s not recorded in the activation metadata. So while it?s
> >>> useful for external tracing (no objections to doing it), I think from an
> >>> openwhisk API point of view we still have a gap.
> >>>
> >>> -r
> >>>
> >>> On Aug 16, 2019, at 4:58 PM, Chetan Mehrotra <chetan.mehrotra@gmail.com>
> >>> wrote:
> >>>
> >>>>> I think if OW SDK, and sequences/compositions, propagate X-Request-Id
> >>>>> header (using the existing transaction id/X-Request-Id), the parent
is
> >>> not
> >>>>> needed?
> >>>>
> >>>> Thats what I counting on as we just need to correlate calls made for
a
> >>>> given flow corelated with time to get a sequence of flow.
> >>>>
> >>>>> - expose the transaction id to runtime container
> >>>>> - propagate the transaction id in requests initiated from runtime
> >>>>> container/controller
> >>>>
> >>>> Makes sense. Would open a ticket capturing these changes then
> >>>> Chetan Mehrotra
> >>>>
> >>>>> On Fri, Aug 16, 2019 at 7:44 AM Tyson Norris <tysonnorris@gmail.com>
> >>> wrote:
> >>>>>
> >>>>> I think if OW SDK, and sequences/compositions, propagate X-Request-Id
> >>>>> header (using the existing transaction id/X-Request-Id), the parent
is
> >>> not
> >>>>> needed? i.e. there may be 2 parts to this effort:
> >>>>> - expose the transaction id to runtime container
> >>>>> - propagate the transaction id in requests initiated from runtime
> >>>>> container/controller
> >>>>>
> >>>>>
> >>>>>
> >>>>>> On Thu, Aug 15, 2019 at 10:38 AM Rodric Rabbah <rodric@gmail.com>
> >>> wrote:
> >>>>>>
> >>>>>> In general yes but I think generally do you need the transaction
id or
> >>> the
> >>>>>> parent id for an activation?
> >>>>>>
> >>>>>> This issue is relevant -
> >>> https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_openwhisk_issues_3083&d=DwIFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Oo9B0p_tCCWIIum5GpjjqA&m=RVvqHH3EKTdA9cNFz3n2dPeXV_wFUOU01I5vxvAqt8Q&s=eZzttWaoeE0WFe_MndFO1O6K1wA9hZbc-BKbRZa6kDM&e=
> >>> .
> >>>>>> I also recall in the early days of the composer, we wanted a
way to
> >>> query
> >>>>>> parent/child activations but this requires new couch views and
we
> >>> didn't
> >>>>>> pursue it.
> >>>>>>
> >>>>>>
> >>>>>>
> >>>>>> On Thu, Aug 15, 2019 at 1:20 PM Chetan Mehrotra
> >>> <chetan.mehrotra@gmail.com
> >>>>>>>
> >>>>>> wrote:
> >>>>>>
> >>>>>>> Currently we pass the `activation_id` as part of `/run`
call to any
> >>>>>>> action runtime [1]. Would it be fine to also pass the `TransactionId`
> >>>>>>> such that it can be accessed by action code?
> >>>>>>>
> >>>>>>> One usecase of this would be to enable tracing a sequence/composition
> >>>>>>> by linking all activations which are part of same transaction
in
> >>>>>>> epsagon [2]
> >>>>>>>
> >>>>>>> Chetan Mehrotra
> >>>>>>> [1]
> >>>>>>>
> >>>>>>
> >>> https://urldefense.proofpoint.com/v2/url?u=https-3A__github.com_apache_openwhisk_blob_master_docs_actions-2Dnew.md-23activation&d=DwIFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Oo9B0p_tCCWIIum5GpjjqA&m=RVvqHH3EKTdA9cNFz3n2dPeXV_wFUOU01I5vxvAqt8Q&s=-vjUf19hoPEdgOTjnbs7jjHFqsJhWIRJjQSQYIme8xw&e=
> >>>
> >>>>>>> [2]
> >>>>>>>
> >>>>>>
> >>> https://urldefense.proofpoint.com/v2/url?u=https-3A__epsagon.com_blog_epsagon-2Dmakes-2Dtroubleshooting-2Dapache-2Dopenwhisk-2Da-2Dsnap_&d=DwIFaQ&c=jf_iaSHvJObTbx-siA1ZOg&r=Oo9B0p_tCCWIIum5GpjjqA&m=RVvqHH3EKTdA9cNFz3n2dPeXV_wFUOU01I5vxvAqt8Q&s=IkGq6vR80pwlGlaXAkXHi_rV14IXnX0hc3smCWO8BM8&e=
> >>>
> >>>>>>>
> >>>>>>
> >>>
> >>>
> >>>
> >>>

Mime
View raw message