openwhisk-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sven Lange-Last" <sven.lange-l...@de.ibm.com>
Subject Re: Don't differentiate between blackbox and whitebox images
Date Fri, 30 Nov 2018 08:06:44 GMT
I agree with Rodric. In the past, our assumption was that the risk 
associated with blackbox invocations is higher than with managed 
invocations. That's why we used two separate invoker pools such that the 
possible problems with blackbox invocations won't affect any managed 
invokers and thus, managed invocations.

We need to re-evaluate whether we still see the same risk as in the past 
and whether we see ways to address possible problems.

For those who are willing to accept certain risks, Rodric's proposal of 
providing flexibility via configuration settings may be the right 
approach. Apparently, the requirements depend on the type of OW 
environment you run. For a large production environment supporting a large 
number of different tenants requirements are different than for your local 
development environment.

I think we have (at least) these risks:

=> Docker images consume too much disk space so that no more new images 
can be pulled / no new containers can be created.

A limited set of Docker images is required to support managed invocations. 
The situation is different for blackbox invocations. Potentially each 
blackbox invocation may lead to a pull of an additional Docker image so 
that the disk volume storing images and containers may fill up. At the 
moment, there are no restrictions on the number and size of blackbox 
images on an invoker. So invokers running blackbox actions have a higher 
risk for outages.

OW does not have any image cleanup strategy and a limited container 
cleanup strategy. Kubernetes has a garbage collection approach [1]. We may 
add a cleanup strategy to OW as well or users need their own cleanup 
approach outside of OW.

=> Managed runtime images have more control.

All managed runtimes containers run OW code up to the point where the 
/init REST endpoint can be used to inject end user action code. So we can 
prepare the environment and accept the initial socket connection from the 
invoker. Our timeout concept bases these socket connections.

Blackbox containers have more control and can decide to not even open a 
listening socket. It's harder for the invoker to deal with such a 
non-cooperating container and prevent the container from consuming 
resources.

=> Blackbox images can bring much more user code than managed actions.

It's easier to hide malicious code or tools in a blackbox image.

=> Blackbox image pulls keep container infrastructure busy.

As already mentioned, each blackbox invocation may lead to a pull of an 
additional Docker image. These pulls consume CPU and network resources 
impacting other invocations. The pull mechanism could even be used to 
attack Docker Hub.


Any other thoughts or feedback on the risks I described?


Regards, Sven.


[1] 
https://kubernetes.io/docs/concepts/cluster-administration/kubelet-garbage-collection/

Rodric Rabbah <rodric@gmail.com> wrote on 2018/11/28 13:09:12:

> From: Rodric Rabbah <rodric@gmail.com>
> To: dev@openwhisk.apache.org
> Date: 2018/11/28 13:09
> Subject: Re: Don't differentiate between blackbox and whitebox images
> 
> I think it would be a mistake to blindly merge two container pools -
> docker actions as run today are pulled as needed and can take a long
> time, they’re subject to different kinds of attacks, and can affect 
> performance of other tenants (higher noisy neighbors). 
> 
> If you want to allow more overlap, it should be at least done as a 
> configuration parameter which allows different pools to continue to 
> exist (note that for small N as in local dev they’re allowed to 
> overlap today). Or, used for overflow only as a way of providing 
> more elasticity.
> 
> -r
> 
> > On Nov 28, 2018, at 5:56 AM, Christian Bickel <cbickel@apache.org> 
wrote:
> > 
> > Hi,
> > 
> > in the past we divided the invokers into blackbox and whitebox 
invokers.
> > This has the effect that one of these two types could be overloaded 
while
> > the other type still has free capacity.
> > 
> > Does anyone see issues on using every invoker for everything?
> > Or does anyone see any action items that need to be addressed before 
we
> > invoke every action-type on every invoker?
> > 
> > If not I'll open a PR to not differentiate between these types 
anymore.
> > 
> > Greetings
> > Christian Bickel



Mime
View raw message