From dev-return-1530-archive-asf-public=cust-asf.ponee.io@openwhisk.apache.org Mon Feb 26 18:06:29 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id 26C3918064A for ; Mon, 26 Feb 2018 18:06:28 +0100 (CET) Received: (qmail 57455 invoked by uid 500); 26 Feb 2018 17:06:28 -0000 Mailing-List: contact dev-help@openwhisk.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@openwhisk.apache.org Delivered-To: mailing list dev@openwhisk.apache.org Received: (qmail 57437 invoked by uid 99); 26 Feb 2018 17:06:27 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 26 Feb 2018 17:06:27 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id 06C15C0040 for ; Mon, 26 Feb 2018 17:06:27 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 2.148 X-Spam-Level: ** X-Spam-Status: No, score=2.148 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, FREEMAIL_ENVFROM_END_DIGIT=0.25, HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd4-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id jehaXoK-5qA0 for ; Mon, 26 Feb 2018 17:06:23 +0000 (UTC) Received: from mail-ua0-f171.google.com (mail-ua0-f171.google.com [209.85.217.171]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id 2D1E25FB5F for ; Mon, 26 Feb 2018 17:06:23 +0000 (UTC) Received: by mail-ua0-f171.google.com with SMTP id b23so8578579uak.9 for ; Mon, 26 Feb 2018 09:06:23 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to; bh=DF4NErtZJshIrIyelVDUtKDh46L9g/WXC69v02J3FsQ=; b=PJhm127b2FX/XTMcf7AsPr/Erp66q3xPDsWIf5/dRv6zI0dOyc4RnO2LuTaxZoPRWO E0rG80PMBZm70VQtkspDIHEJ+hU11ue/yjE+WfRYVHNShWUG3ECIK+Sek0ATaC9oqnQu L4dz909g938mdadZor02bocV7M6mg3HXjhxsB+DkVne6GfrlGr/RkH2MMfNeRcQ5y2cf pJPu+8QNc8bzckXPYfxJPprb+o3EeHZcLdvM/1IXMlDMVDpHk4aNia8oLv23Bv5RP+I4 3DopJ5LCjSc95fTBBPfZwEQWW9mtILTW5UBv3WpJ6dXDiZgMvxVdy21EWvnlRjSJqZ4p OA1w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=DF4NErtZJshIrIyelVDUtKDh46L9g/WXC69v02J3FsQ=; b=CY9PgbZqgoLexjRC3JJgI84kYjUh+PzQkXvno+akmUgQ9qF+Icbn4Rupb3hESME3XJ AKs6jyDdvIMhutMYMdtfSMMWMd+toZCU/ong6cJgDnDXk/oKKqyG2mnLMykPwUPpCAGW NoXXYQ0lIwveMO2mbj7gyeq2w7/NO1ELdlv9iODCNbtCUwCLE98P2gTgJ92PIr07H9ZY pLn0eKylygbMOAvnoJE0if3W2KZyIydUhcVgESmBqpemtEk4CPYNw/4sew5a3G16Qpoz 1DUTl9Fk0FrHtKy95rNdsgTcn2KivP9WszeXFSQwrP7wAE1XyUcH0Y8zIzUBsC3efTA3 tUpA== X-Gm-Message-State: APf1xPDk10KyaTuKwcFggZjiaTHiPR0CiRg5MBqC1SV6tex1ol4G6lbI /xv+NyQ49Qh/ZzROkRD4ILRryACJr+z+pp+qysA= X-Google-Smtp-Source: AG47ELvGgu/J0yB92CrKHS/tHqoJD8ZErB/JkwnQOOFfGP6Vj7smL3jEkggPtbt4lKqFgU5h6sltxXkeGydH6yJpMAo= X-Received: by 10.176.89.232 with SMTP id k37mr7941363uad.160.1519664782457; Mon, 26 Feb 2018 09:06:22 -0800 (PST) MIME-Version: 1.0 References: <1519498684.3209770.1282095880.5988968F@webmail.messagingengine.com> <1519627132.1511133.1283252944.7A667938@webmail.messagingengine.com> In-Reply-To: <1519627132.1511133.1283252944.7A667938@webmail.messagingengine.com> From: Carlos Santana Date: Mon, 26 Feb 2018 17:06:11 +0000 Message-ID: Subject: Re: Implementation plan for Go actions To: dev@openwhisk.apache.org Content-Type: multipart/alternative; boundary="001a11473968893fb5056620877f" --001a11473968893fb5056620877f Content-Type: text/plain; charset="UTF-8" Hi Michele >There is only one "annoyance" here: I noticed the dockerskeleton always expects a zip file including an "exec" dockerskeleton suppports adding an exec straight, it will take the file do a chmod +x and executed. So not sure what you are referring here as this is possible, maybe I miss understood >It is noticeably slow. Go is a bit faster but the problem with Go is downloading dependencies (that can be a lot...). This is the major problem for Swift also, SPM (Swift Package Manager) will take the longest time git cloning entires repos to pull down dependencies >1. create an apache/incubator-openwhisk-runtime-go repo, so I can fork it and add my code in order to follow your build infrastructure and code guidelines. Then I will submit a PR to it. Yes this was the intention, to create the apache repo, then you can fork it and submit a PR >2. ....To make it super easy you may want a simple function somewhere to import. .... just to add some code to the existing https://github.com/apache/incubator-openwhisk-client-go. This is super goes in line with what we do for the javascript library npm, we use the same package in action code, and also outside client side to the whisk api. >3. Yes no issues in changing the wsk cli super +1 > so please let some real expert on swift give a look at what I did and fix it. I'm not an expert :-), but I can help here either myself, or I can reach out to swift experts that I know first hand in IBM from Kitura team On Mon, Feb 26, 2018 at 10:55 AM Michele Sciabarra wrote: > > On Mon, Feb 26, 2018, at 4:39 AM, Carlos Santana wrote: > > Great progress Michele +1 > > > > I agree with the order of your steps. > > > > Step #1 > > Let's nock step 1 first. > > I will summarize it as having first class support for Go at the same > level > > of JavaScript, Python, Java, Swift, PHP > > Meaning from end user support from CLI, docs, runtime image. > > wsk action create goaction goaction.go > > wsk action create goaction goexec --kind go:1.10 > > wsk action create goaction goaction.zip --kind go:1.10 > > extend the runtime image with anything you want to add to the image > > From example: > > FROM openwhisk/action-go-v1.10 > > apt-get install -y imagemagick > > docker build . -t csantanapr/action-go-v1.10 > > wsk action create goaction goaction.go --docker > csantanapr/action-go-v1.10 > > I guess will have a FDK (Functions Developer Kit) :-), making it super > easy > > to import it and just write a simple go main method/function the FDK > > handles the json input, output, etc.. > > > > > > I can help by requesting a new runtime repo > > "apache/incubator-openwhisk-runtime-go" > > I can setup a boilerplate repo content with some structure for gradle, > > build, travis, scala tests > > Then you can submit PR to the repo with go specifics. > > I guess since your go guru you can handle the few changes to the wsk cli > > Do you have a repo with some code already I can take a look? > I have all the code done so far in > https://github.com/sciabarracom/openwhisk-runtime-go. > However since I followed my own way, please let do this way: > > 1. create an apache/incubator-openwhisk-runtime-go repo, so I can fork it > and add my code in order to follow your build infrastructure and code > guidelines. Then I will submit a PR to it. > > 2. Regarding the Go FDK, so far you can easily implement a piped executor > using standard Go libraries. To make it super easy you may want a simple > function somewhere to import. I think instead of creating a new sdk the > best way is just to add some code to the existing > https://github.com/apache/incubator-openwhisk-client-go. > I can submit a PR for it. > > 3. Yes no issues in changing the wsk cli (I already gave a look at it). > Will submit PR. > > > > > > Step #2 > > For swift, I can help since I'm working on swift4, it looks like we can > > have a new proxy instead of using the python one. > > We do some pre-compilation/magic today with buildandrecord.py [1] to > > compile as much as possible up front and creating a shell script that has > > the exact lines for compiling and linking. > > Do you have some code on swift consuming from pipe instead of ReadLine I > > can take a look? > > Yes this is the code I am using: > > https://github.com/sciabarracom/openwhisk-runtime-go/blob/pipe-loop/hello/swift/Sources/Hello/main.swift > > Written this way, it works under my new runtime. Note the ugly hack to > call the flush in a system dependent way. Unfortunately I have not found a > portable way to do that, and without flushing the stdout the runtime hangs. > > But all my knowledge of Swifts comes from "it looks like a bit like Scala" > (that I know) and "Google says...", so please let some real expert on swift > give a look at what I did and fix it. > > I think it is possible to have a backward compatible swift runtime maybe > just running it under the new Go runtime, but before I think I do better to > have stable code for the Go-only runtime. > > > > > Step #3 > > After doing go and swift, we can see what are the common parts we can > > extract to be common in the docker runtime repo. > > And work on backwards compatible single image, and handle the new pipe > with > > some annotation, or maybe a new image, we can discuss more deeply once we > > get there. > Yup. Once solved Swift it should be easier... > > > > > > > [1] > > > https://github.com/apache/incubator-openwhisk-runtime-swift/blob/master/core/swift41Action/buildandrecord.py > > > > > > > > > > > > > > On Sat, Feb 24, 2018 at 1:58 PM Michele Sciabarra < > openwhisk@sciabarra.com> > > wrote: > > > > > Hello, whiskers > > > > > > I am at the risk of expanding too much the scope of what I am trying to > > > do, so I want to try to focus and work step by step. > > > > > > So I am writing down my plan for review. > > > > > > # Step one: implement a good support for GoLang, with piped standard > input > > > and output, and log in stderr. > > > > > > Because there is not a past support for Go, except the generic docker > > > support, I think I have some freedom here. > > > > > > The support will then work this way (please review carefully): > > > > > > ## /init > > > > > > The runtime expects posted in /init the usual > > > > > > { > > > "binary": ... > > > "code": ... > > > "main": .. > > > } > > > > > > where "code" is either a zip file (with binary: true) or a source code > > > (with binary: false or absent). > > > > > > If it source code, it expects to be a go source. > > > It places in /action/main/exec.go and compiles it with go build -o > > > /action/exec /action/main/exec.go, getting a /action/exec to execute. > > > > > > If is is a zip file, it will unzip it in /action. If there is a > > > /action/exec file in the zip it will run it. > > > > > > If there is not, it will look if there is a "main" parameter, getting > the > > > . > > > > > > It will compile a /action/main/.go. If the is > > > missing, it will default to /action/main/exec.go. > > > > > > Note the zip can include other source files that will be stored in > > > /action/. So I will have to use GOPATH=/action to find them. > > > > > > ## /run > > > > > > The /run behaviour is: > > > > > > At every request, if no action process is running,it will start one and > > > will pass the first line as an argument and also in standard output. > > > > > > If the executable does not terminate, it will keep feeding more input > in > > > standard input, expecting output in standard output (line by line) and > logs > > > in stderr. > > > > > > The server will have to collect stderr logs and write them in docker > > > stdout, and terminate the output with the markers when an action > completes: > > > > > > XXX_THE_END_OF_A_WHISK_ACTIVATION_XXX > > > XXX_THE_END_OF_A_WHISK_ACTIVATION_XXX > > > > > > It the executable DOES terminate, it will execute again at the next > /run > > > request > > > > > > ## Step 2: need to implement the same logic for swift actions (we > have to > > > keep the existing compilation infrastructure) > > > > > > Issues to solve here are the compilation and the "warm" startup > problem. I > > > have some ideas but... I do better to discuss them once the Go support > is > > > finished. > > > > > > ## Step 3: will discuss with the list if it is worth to extend the > support > > > to the generic dockerskeleton, that is based on the python proxy. > > > > > > There are compatibility issues, because the current docker support logs > > > everything in stdout and the last line is the answer, while for using > the > > > pipe you need to do separate log. > > > > > > How does it sound? > > > Suggested or required changes? > > > Am I missing something? > > > Approved? > > > > > > > > > -- > > > Michele Sciabarra > > > openwhisk@sciabarra.com > > > > --001a11473968893fb5056620877f--