openwhisk-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Markus Thömmes <markusthoem...@apache.org>
Subject Re: Kafka and Proposal on a future architecture of OpenWhisk
Date Thu, 23 Aug 2018 20:05:53 GMT
Hi Tyson,

Am Do., 23. Aug. 2018 um 21:28 Uhr schrieb Tyson Norris
<tnorris@adobe.com.invalid>:

>     >
>     > And each ContainerRouter has a queue consumer that presumably pulls
> from
>     > the queue constantly? Or is consumption based on something else? If
> all
>     > ContainerRouters are consuming at the same rate, then while this does
>     > distribute the load across ContainerRouters, it doesn't really
> guarantee
>     > any similar state (number of containers, active connections, etc) at
> each
>     > ContainerRouter, I think. Maybe I am missing something here?
>     >
>
>
>     The idea is that ContainerRouters do **not** pull from the queue
>     constantly. They pull work for actions that they have idle containers
> for.
>
> Router is not pulling at queue for "specific actions", just for any action
> that might replace idle containers - right? This is complicated with
> concurrency though since while a container is not idle (paused +
> removable), it may be useable, but only if the action received is the same
> as one existing warm container, and that container has concurrency slots
> available for additional activations. It may be helpful to diagram some of
> this stealing queue flow a bit more, I'm not seeing how it will work out
> other than creating more containers than is absolutely required, which may
> be ok, not sure.
>

Yes, I will diagram things out soonish, I'm a little bit narrow on time
currently.

The idea is that indeed the Router pulls for *specific* actions. This is a
problem when using Kafka, but might be solvable when we don't require
Kafka. I have to test this for feasibility though.


>
>     Similar state in terms of number of containers is done via the
>     ContainerManager. Active connections should roughly even out with the
> queue
>     being pulled on idle.
>
> Yeah carefully defining "idle" may be tricky, if we want to achieve
> absolute minimum containers in use for a specific action at any time.
>
>
>     >
>     >     The edge-case here is for very slow load. It's minimizing the
> amount of
>     >     Containers needed. Another example:
>     >     Say you have 3 Routers. A request for action X comes in, goes to
>     > Router1.
>     >     It requests a container, puts the work on the queue, nobody
> steals it,
>     > as
>     >     soon as the Container gets ready, the work is taken from the
> queue and
>     >     executed. All nice and dandy.
>     >
>     >     Important remark: The Router that requested more Containers is
> not
>     >     necessarily the one that's getting the Containers. We need to
> make
>     > sure to
>     >     evenly distribute Containers across the system.
>     >
>     >     So back to our example: What happens if requests for action X
> are made
>     > one
>     >     after the other? Well, the layer above the Routers (something
> needs to
>     >     loadbalance them, be it DNS or another type of routing layer)
> isn't
>     > aware
>     >     of the locality of the Container that we created to execute
> action X.
>     > As it
>     >     schedules fairly randomly (round-robin in a multi-tenant system
> is
>     >     essentially random) the action will hit each Router once very
> soon. As
>     >     we're only generating one request after the other, arguably we
> only
>     > want to
>     >     create only one container.
>     >
>     >     That's why in this example the 2 remaining Routers with no
> container
>     > get a
>     >     reference to Router1.
>     >
>     >     In the case you mentioned:
>     >     > it seems like sending to another Router which has the
> container, but
>     > may
>     >     not be able to use it immediately, may cause failures in some
> cases.
>     >
>     >     I don't recall if it's in the document or in the discussion on
> the
>     >     dev-list: The router would respond to the proxied request with a
> 503
>     >     immediatly. That tells the proxying router: Oh, apparently we
> need more
>     >     resources. So it requests another container etc etc.
>     >
>     >     Does that clarify that specific edge-case?
>     >
>     > Yes, but I would not call this an edge-case -  I think it is more of
> a
>     > ramp up to maximum container reuse, and will probably dramatically
> impacted
>     > by containers that do NOT support concurrency (will get a 503 when a
> single
>     > activation is in flight, vs high concurrency container, which would
> cause
>     > 503 only once max concurrency reached).
>     > If each ContainerRouter is as likely to receive the original
> request, and
>     > each is also as likely to receive the queued item from the stealing
> queue,
>     > then there will be a lot of cross traffic during the ramp up from 1
>     > container to <Router count> containers. E.g.
>     >
>     > From client:
>     > Request1 -> Router 1 -> queue (no containers)
>     > Request2 -> Router 2 -> queue (no containers)
>     > Request3 -> Router 3 -> queue (no containers)
>     > From queue:
>     > Request1 -> Router1  -> create and use container
>     > Reuqest2 -> Router2 -> Router1 -> 503 -> create container
>     > Request3 -> Router3 -> Router1 -> 503 -> Router2 -> 503 ->
create
> container
>     >
>     > In other words - the 503 may help when there is one container
> existing,
>     > and it is deemed to be busy, but what if there are 10 containers
> existing
>     > (on different Routers other than where the request was pulled from
> the
>     > stealing queue) - do you make HTTP requests to all 10 Routers to see
> if
>     > they are busy before creating a new container?
>     >
>
>     Good point, haven't really thought about that to be frank. Gut feeling
> is
>     that we should only have 1 direct reference per Router/Action to
> another
>     Router. If that yields a 503, just generate new resources immediately.
> That
>     might overshoot what we really need, but might just be good enough?
> Maybe
>     I'm overcomplicating here...
>
>     Alternative to the whole work-stealing algorithm (which could unify
> this
>     ramp-up phase and work-stealing itself nicely):
>     What if it is possible to detect that when an event is pushed to the
>     work-stealing queue, there are no active consumers for that event right
>     now. If that is possible, we could use that to signal that more
> Containers
>     are needed, because apparently nobody has idling resources for a
> specific
>     container.
>
> I think this is only possible in the case where 0 containers (for that
> action) existed before; in other cases, without having Router state
> visibility, it will be impossible to detect Routers that have capacity to
> service that event.
>

Why? If Routers have a container that has capacity, they will be listening
for more work (per my answer above).


>
>     We would then also use this mechanism to generate Containers in
> general.
>     The ContainerRouters would **never** request new containers. They would
>     instead put their work on the stealing queue to see let someone else
> work
>     on it. If the system detects that nobody is waiting, it requests a new
>     Container (maybe the ContainerManager could be used to detect that
> signal?)
>
>     For our "edge-case" that'd mean: No references to other Routers are
> handed
>     out at all. If a request arrives at a Router that has no Container to
> serve
>     it, it just puts it on the queue. If there's a consumer for it, great,
>     done. If not, we know  we need more resources.
>
>     This boils down to needing an efficient mechanism to signal free
> capacities
>     though. Something to think deeper about, thanks for bringing it up!
>
> Yes, I think "capacity" is more accurate way to think of it, than "idle",
> and yes I think a 503 should generate container immediately, but I think we
> need to supply more data to intelligently limit the 503 workflows.
>
> What about this:
> - ContainerRouters report capacity ("possible capacity", or "busy") to
> ContainerManager periodically (not accurate at point in time, but close
> enough in some cases)
> - When no capacity exists in ContainRouter's existing warm containers,
> ContainRouter requests container from ContainerManager
> - ContainerManager responds with either ContainerRouter (if one exists
> with "possible capacity"), or Container (if none exists with "possible
> capacity")
>

I think these 3 points are addressed by being able to pull for specific
actions and/or being able to detect that pull signal. In fact, it's the
same mechanism: The ContainerManager essentially is your work-stealing
backend.


> - In case Container is returned, if C > 1, its capacity state should start
> as "possible capacity"; otherwise, "busy" (we know it will be immediately
> used)
> - In case ContainerRouter is returned, attempt (once) proxy to that
> ContainerRouter
> - On a proxied request to ContainerRouter CR2 either service the request,
> OR immediately create a container (in CR2), OR we may limit the number of
> same action containers in a ContainerRouter so return 503 at which point
> immediately create a container (in CR1).
>
> This would:
> - encourage multiple containers for same action to be managed in a subset
> of routers (better chance of reusing containers)

- not restrict the number of routers used for a specific action when under
> load (e.g. say that each router can handle active requests for n
> containers, meaning n connection pools).
> - allow designating a ContainerRouter capacity config for both a)
> activations on warm containers as configured per action and b) overall
> number of containers (connection pools) as configured at ContainerRouter
> (to smooth hot spots across ContainerRouters).
>
> I think the biggest problem would be cases where containers hover around
> "busy" state (C requests), causing requests to CM, and inaccurate data in
> CM (causing extra containers to be created), but it may be OK for most load
> patterns, nothing will work perfectly for all.
>
> Reporting capacity can either be metrics based or a health ping similar to
> what exists today (but with more details of pool states).
>
> I will try to diagram this, it's getting complicated...
>

Yes, I will as well.


>
> Thanks
> Tyson
>
>
>     >
>     >     Memo to self: I need to incooperate all the stuff from these
>     > discussions
>     >     into the document.
>     >
>     >
>     >     >
>     >     >
>     >     >     The work-stealing queue though is used to rebalance work
> in case
>     > one
>     >     > of the
>     >     >     Routers get overloaded.
>     >     >
>     >     > Got it.
>     >     >
>     >     > Thanks
>     >     > Tyson
>     >     >
>     >     >
>     >     >
>     >
>     >
>     >
>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message