openwhisk-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michele Sciabarra <mich...@sciabarra.com>
Subject Re: Implementation plan for Go actions
Date Mon, 26 Feb 2018 12:50:25 GMT
I agree on focusing on binary support for now. There is only one "annoyance" here: I noticed
the dockerskeleton always expects a zip file including an "exec". I would like to be able
to support also binaries pushed straight, and I was trying to do that using magic file detection
(there is a Go library for that).

To be honest, as I told already on Slack, I am a bit skeptic on source support tot court as
it is implemented for example in the Swift image. It is noticeably slow. Go is a bit faster
but the problem with Go is downloading dependencies (that can be a lot...).

You will have a long startup time when you scale, and having a long startup time when you
need quickly more instances is definitely a killer problem for  scaling support.

As I mentioned, for binaries like swift and go, and also for Java,  I was thinking we should
have something similar to what Kubernetes allows to do: an image for building, producing an
executable, to be cached somewhere, to be reused when you launch more images. 

Just my 2cents of ideas....

-- 
  Michele Sciabarra
  michele@sciabarra.com

On Mon, Feb 26, 2018, at 1:24 PM, Rodric Rabbah wrote:
> Supporting go as source is nice for parity but you can also stage the 
> changes focusing on the binary case first, get the /run details nailed, 
> and working through logs and full experience first. Go cross-compilation 
> works well and if you’re writing in go, beyond the convenience of 
> quickly getting started with a source snippet, I would speculate the 
> common use case is for the binary path.
> 
> -r
> 
> > On Feb 25, 2018, at 10:39 PM, Carlos Santana <csantana23@gmail.com> wrote:
> > 
> > Great progress Michele +1
> > 
> > I agree with the order of your steps.
> > 
> > Step #1
> > Let's nock step 1 first.
> > I will summarize it as having first class support for Go at the same level
> > of JavaScript, Python, Java, Swift, PHP
> > Meaning from end user support from CLI, docs, runtime image.
> > wsk action create goaction goaction.go
> > wsk action create goaction goexec --kind go:1.10
> > wsk action create goaction goaction.zip --kind go:1.10
> > extend the runtime image with anything you want to add to the image
> > From example:
> > FROM openwhisk/action-go-v1.10
> > apt-get install -y imagemagick
> > docker build . -t csantanapr/action-go-v1.10
> > wsk action create goaction goaction.go --docker csantanapr/action-go-v1.10
> > I guess will have a FDK (Functions Developer Kit) :-), making it super easy
> > to import it and just write a simple go main method/function the FDK
> > handles the json input, output, etc..
> > 
> > 
> > I can help by requesting a new runtime repo
> > "apache/incubator-openwhisk-runtime-go"
> > I can setup a boilerplate repo content with some structure for gradle,
> > build, travis, scala tests
> > Then you can submit PR to the repo with go specifics.
> > I guess since your go guru you can handle the few changes to the wsk cli
> > Do you have a repo with some code already I can take a look?
> > 
> > Step #2
> > For swift, I can help since I'm working on swift4, it looks like we can
> > have a new proxy instead of using the python one.
> > We do some pre-compilation/magic today with buildandrecord.py [1] to
> > compile as much as possible up front and creating a shell script that has
> > the exact lines for compiling and linking.
> > Do you have some code on swift consuming from pipe instead of ReadLine I
> > can take a look?
> > 
> > Step #3
> > After doing go and swift, we can see what are the common parts we can
> > extract to be common in the docker runtime repo.
> > And work on backwards compatible single image, and handle the new pipe with
> > some annotation, or maybe a new image, we can discuss more deeply once we
> > get there.
> > 
> > 
> > [1]
> > https://github.com/apache/incubator-openwhisk-runtime-swift/blob/master/core/swift41Action/buildandrecord.py
> > 
> > 
> > 
> > 
> > 
> > 
> > On Sat, Feb 24, 2018 at 1:58 PM Michele Sciabarra <openwhisk@sciabarra.com>
> > wrote:
> > 
> >> Hello, whiskers
> >> 
> >> I am at the risk of expanding too much the scope of what I am trying to
> >> do, so I want to try to focus and work step by step.
> >> 
> >> So  I am writing down my plan for review.
> >> 
> >> # Step one: implement a good support for GoLang, with piped standard input
> >> and output, and log in stderr.
> >> 
> >> Because there is not a past support for Go, except the generic docker
> >> support, I think I have some freedom here.
> >> 
> >> The support will then work this way (please review carefully):
> >> 
> >> ## /init
> >> 
> >> The runtime expects posted in /init the usual
> >> 
> >> {
> >> "binary": ...
> >> "code": ...
> >> "main": ..
> >> }
> >> 
> >> where "code" is either a zip file (with binary: true) or a source code
> >> (with binary: false or absent).
> >> 
> >> If it source code, it expects to be a go source.
> >> It places in /action/main/exec.go and compiles it with go build -o
> >> /action/exec /action/main/exec.go, getting a /action/exec to execute.
> >> 
> >> If is is a zip file, it will unzip it in /action. If there is a
> >> /action/exec file in the zip it will run it.
> >> 
> >> If there is not,  it will look if there is a "main" parameter, getting the
> >> <main-value>.
> >> 
> >> It will compile a /action/main/<main-value>.go. If the <main-value>
is
> >> missing, it will default to /action/main/exec.go.
> >> 
> >> Note the zip can include other source files that will be stored in
> >> /action/<folders>. So I will have to use GOPATH=/action to find them.
> >> 
> >> ## /run
> >> 
> >> The /run behaviour is:
> >> 
> >> At every request, if no action process is running,it will start one and
> >> will pass the first line as an argument and also in standard output.
> >> 
> >> If the executable does not terminate, it will keep feeding more input in
> >> standard input, expecting output in standard output (line by line) and logs
> >> in stderr.
> >> 
> >> The server will have to collect stderr logs and write them in docker
> >> stdout, and terminate the output with the markers when an action completes:
> >> 
> >> XXX_THE_END_OF_A_WHISK_ACTIVATION_XXX
> >> XXX_THE_END_OF_A_WHISK_ACTIVATION_XXX
> >> 
> >> It the executable DOES terminate, it will execute again at the next /run
> >> request
> >> 
> >> ## Step 2: need to implement the same  logic for swift actions (we have to
> >> keep the existing compilation infrastructure)
> >> 
> >> Issues to solve here are the compilation and the "warm" startup problem. I
> >> have some ideas but... I do better to discuss them once the Go support is
> >> finished.
> >> 
> >> ## Step 3: will discuss with the list if it is worth to extend the support
> >> to the generic dockerskeleton, that is based on the python proxy.
> >> 
> >> There are compatibility issues, because the current docker support logs
> >> everything in stdout and the last line is the answer, while for using the
> >> pipe you need to do separate log.
> >> 
> >> How does it sound?
> >> Suggested or required changes?
> >> Am I missing something?
> >> Approved?
> >> 
> >> 
> >> --
> >>  Michele Sciabarra
> >>  openwhisk@sciabarra.com
> >> 

Mime
View raw message