openwhisk-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chetan Mehrotra <chetan.mehro...@gmail.com>
Subject Possible enhancements around attachment handling
Date Sun, 25 Mar 2018 19:02:19 GMT
Last week I had a Slack call with Rodric around AttachmentStore PR and
following possible enhancements around attachment handling in
OpenWhisk were discussed. These enhancements would enable efficient
handling of code content specially for large actions

A - Attachment Inlining
------------------------------

Currently code related to action can be stored in following form

1. Plaintext string - Code is stored as is in plain text form in
`code` attribute

2. Base64 encoded zip content - Code is stored as base64 encoded
zipped content in `code` attribute with `binary` set to true

3. CouchDB attachment - Code content is stored as raw bytes via
CouchDB attachment support [1]

In all cases when Invoker invokes the action the code content is
passed as string as part of initializer json. Also when action is
created the code is passed inline as part of json. Going forward
(specially useful with switch to external AttachmentStore) the code
content would always be stored as attachment (PR #2847). For such
cases it would better to have a way where smaller code can be stored
inlined (as being done for non java cases currently) to avoid extra
calls to AttachmentStore.

This inlining logic would be an internal detail of AttachmentStore and
would be based on configurable maxInlineSize configuration
irrespective of action kind.

B - Streaming Action Code - Action Creation
----------------------------------------------------------

Currently the action is provided inline string attribute in json
payload POSTed to namespaces/<package>/actions/<actionName> endpoint.
Due to this whole code content is processed in memory as byte array.
Compared to this the ArtifactStore abstraction supports streams for
creating and reading attachments.

For supporting large action code it would better if we enable
streaming of action code from client to ArtifactStore without storing
it as byte array within heap. This can be done by supporting
multi-part upload [2] for action endpoint and then just passing the
byteSource to ArtifactStore

C - Streaming Action Code - Action Execution
-------------------------------------------------------------

This compliments the #B option. Currently action code is passed as
part of json in action initializer call. Going forward with support of
AttachmenetStore we can make use of constructs like signed url [3]
(suppored in most stores like S3, IBM COS and Azure Blob Storage).
Here instead of passing the action code in json the Invoker would
obtained a signed url which is then passed to container and internally
container directly streams the url content.

Kindly share your thoughts/comment around this topic

Chetan Mehrotra
[1] http://docs.couchdb.org/en/2.0.0/api/document/attachments.html
[2] https://doc.akka.io/docs/akka-http/current/routing-dsl/directives/file-upload-directives/fileUpload.html
[3] https://docs.aws.amazon.com/AmazonS3/latest/dev/ShareObjectPreSignedURL.html

Mime
View raw message