openwhisk-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Tyson Norris <tnor...@adobe.com.INVALID>
Subject Re: Concurrency in invoker http client
Date Mon, 25 Jun 2018 17:29:27 GMT
Thanks Markus - one other question: 

Assuming retry is the current missing piece to using PoolingRestClient (or akka http directly),
I’m also wondering if “retry” is the proper approach here?
It may be worthwhile to initiate a port connection (with its own timeout/retry behavior) before
the /init so that we can distinguish between “container startup is slow” and “bad behavior
in action container after startup”?

Also, I’m wondering if there are cases where rampant retry causes unintended side affects,
etc - this could be worse with concurrency enabled, but I don’t know if this should be considered
a real problem. 

FWIW We avoid this (http request to container that is not yet listening) in mesos by not returning
the container till the mesos health check passes (which currently just check the port connection),
so this would be a similar setup at the invoker layer.

Thanks
Tyson

> On Jun 25, 2018, at 10:08 AM, Markus Thoemmes <markus.thoemmes@de.ibm.com> wrote:
> 
> Hi Tyson,
> 
> Ha, I was thinking about moving back to akka the other day. A few comments:
> 
> 1. Travis build environments have 1.5 CPU cores which might explain the "strange" behavior
you get from the apache client? Maybe it adjusts its thread pool based on the number of cores
available?
> 2. To implement retries based on akka-http, have a look at what we used to use for invoker
communication: https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Fincubator-openwhisk%2Fcommit%2F31946029cad740a00c6e6f367637a1bcfea5dd18%23diff-5c6f165d3e8395b6fe915ef0d24e5d1f&data=02%7C01%7Ctnorris%40adobe.com%7Ca07af0966c50438a270908d5dabe4b3a%7Cfa7b1b5a7b34438794aed2c178decee1%7C0%7C1%7C636655433199601572&sdata=ZHjbf0ukNQaEFq2i58f5hxMt3zRa3JCHdHR0MRAn8Uo%3D&reserved=0
(NewHttpUtils.scala to be precise).
> 3. I guess you're going to create a new PoolingRestClient per container? I could imagine
it is problematic if new containers come and go with a global connection pool. Just something
to be aware of.
> 
> Oh, another thing to be aware of: We **used** to have akka-http there and never tried
again after the revert. We're certainly on a much newer version now but we had issues of indefinitly
hanging connections when we first implemented it. Running some high-load scenarios before
pushing this into master will be needed.
> 
> I don't want to put you off the task though, give it a shot, I'd love to have this back
:). Thanks for attacking!
> 
> Cheers,
> -m
> 

Mime
View raw message