openwhisk-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chetan Mehrotra <chetan.mehro...@gmail.com>
Subject Re: Possible enhancements around attachment handling
Date Thu, 29 Mar 2018 15:49:14 GMT
> This should be absolutely doable without a change in the runtimes at all, as they "shouldn't"
(to be verified) need a change to support TCP streaming.

I think some of the runtimes are currently reading whole request in
memory. For e.g. in Java [1] it constructs a json instance. So would
need some logic to stream the content to filesystem (possiby via
multipart support). For nodejs some sort of multi part handling or
stream handling would be required.

Chetan Mehrotra
[1] https://github.com/apache/incubator-openwhisk-runtime-java/blob/master/core/javaAction/proxy/src/main/java/openwhisk/java/action/Proxy.java#L80


On Thu, Mar 29, 2018 at 4:40 PM, Markus Thoemmes
<markus.thoemmes@de.ibm.com> wrote:
> Heya,
>
>>> So for #C is there a way to proxy the stream without changing the runtime http
api where binary is passed embedded in the json object on /init?
>
>> They would need to be modified. A less intrusive approach would have
>> been to pass signed url and use client like wget to fetch the binary.
>> But as David mentioned some setups may lock down access to external
>> services. For such cases we would need to implement the support in
>> client runtimes as part of init protocol
>
>> Chetan Mehrotra
>
>
> I believe they don't need to be changed at all. TCP as the transport protocol is streamed
by default (stuff is broken down into smallish packets). Say for example our payload we need
to pass is: {"main": "foo", "code": "Some veeeeery long string here"}. Let's also assume our
attachement is available as Source[ByteString]. What we can do, is generate the first bit
until the part we *need* to stream, in this case: {"main": "foo", "code": ". Now we consume
the chunks of the attachement Source and put them on the downstream. In the end, we put an
element like "} on the stream, to close the JSON object.
>
> Pseudocode: Source.single(ByteString("""{"main": "foo", "code":""")).concat(attachementSource).to(httpConnectionToContainer).run()
>
> This should be absolutely doable without a change in the runtimes at all, as they "shouldn't"
(to be verified) need a change to support TCP streaming.
>
> Does that make sense?
>
> Cheers,
> Markus
>

Mime
View raw message