From dev-return-2855-apmail-openwhisk-dev-archive=openwhisk.apache.org@openwhisk.apache.org Fri Nov 30 08:06:54 2018 Return-Path: X-Original-To: apmail-openwhisk-dev-archive@minotaur.apache.org Delivered-To: apmail-openwhisk-dev-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id BC02A180D5 for ; Fri, 30 Nov 2018 08:06:54 +0000 (UTC) Received: (qmail 24385 invoked by uid 500); 30 Nov 2018 08:06:54 -0000 Delivered-To: apmail-openwhisk-dev-archive@openwhisk.apache.org Received: (qmail 24297 invoked by uid 500); 30 Nov 2018 08:06:53 -0000 Mailing-List: contact dev-help@openwhisk.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@openwhisk.apache.org Delivered-To: mailing list dev@openwhisk.apache.org Received: (qmail 24286 invoked by uid 99); 30 Nov 2018 08:06:52 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 Nov 2018 08:06:52 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id E1195D14DD for ; Fri, 30 Nov 2018 08:06:51 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -0.701 X-Spam-Level: X-Spam-Status: No, score=-0.701 tagged_above=-999 required=6.31 tests=[RCVD_IN_DNSWL_LOW=-0.7, SPF_PASS=-0.001] autolearn=disabled Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id O_SUppAmOz8T for ; Fri, 30 Nov 2018 08:06:49 +0000 (UTC) Received: from mx0a-001b2d01.pphosted.com (mx0b-001b2d01.pphosted.com [148.163.158.5]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id 0D7BA60DBB for ; Fri, 30 Nov 2018 08:06:48 +0000 (UTC) Received: from pps.filterd (m0098414.ppops.net [127.0.0.1]) by mx0b-001b2d01.pphosted.com (8.16.0.22/8.16.0.22) with SMTP id wAU7xCHm037154 for ; Fri, 30 Nov 2018 03:06:47 -0500 Received: from smtp.notes.na.collabserv.com (smtp.notes.na.collabserv.com [192.155.248.90]) by mx0b-001b2d01.pphosted.com with ESMTP id 2p30c42su5-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Fri, 30 Nov 2018 03:06:47 -0500 Received: from localhost by smtp.notes.na.collabserv.com with smtp.notes.na.collabserv.com ESMTP for from ; Fri, 30 Nov 2018 08:06:47 -0000 Received: from us1a3-smtp06.a3.dal06.isc4sb.com (10.146.103.243) by smtp.notes.na.collabserv.com (10.106.227.141) with smtp.notes.na.collabserv.com ESMTP; Fri, 30 Nov 2018 08:06:44 -0000 Received: from us1a3-mail83.a3.dal06.isc4sb.com ([10.146.21.231]) by us1a3-smtp06.a3.dal06.isc4sb.com with ESMTP id 2018113008064363-331768 ; Fri, 30 Nov 2018 08:06:43 +0000 In-Reply-To: <6CF3F35A-AD3A-4338-97FA-BBDD08D0BD33@gmail.com> To: dev@openwhisk.apache.org Subject: Re: Don't differentiate between blackbox and whitebox images From: "Sven Lange-Last" Date: Fri, 30 Nov 2018 09:06:44 +0100 References: <6CF3F35A-AD3A-4338-97FA-BBDD08D0BD33@gmail.com> X-KeepSent: 46057784:2C1615DF-C1258355:002C6ECA; type=4; name=$KeepSent X-Mailer: IBM Notes Release 9.0.1EXT SHF886 February 20, 2018 X-LLNOutbound: False X-Disclaimed: 31307 X-TNEFEvaluated: 1 x-cbid: 18113008-9717-0000-0000-00000A2D56A4 X-IBM-SpamModules-Scores: BY=0; FL=0; FP=0; FZ=0; HX=0; KW=0; PH=0; SC=0.388783; ST=0; TS=0; UL=0; ISC=; MB=0.320625 X-IBM-SpamModules-Versions: BY=3.00010146; HX=3.00000242; KW=3.00000007; PH=3.00000004; SC=3.00000270; SDB=6.01124838; UDB=6.00584044; IPR=6.00904966; BA=6.00006167; NDR=6.00000001; ZLA=6.00000005; ZF=6.00000009; ZB=6.00000000; ZP=6.00000000; ZH=6.00000000; ZU=6.00000002; MB=3.00024394; XFM=3.00000015; UTC=2018-11-30 08:06:46 X-IBM-AV-DETECTION: SAVI=unsuspicious REMOTE=unsuspicious XFE=unused X-IBM-AV-VERSION: SAVI=2018-11-30 02:04:29 - 6.00009283 x-cbparentid: 18113008-9718-0000-0000-00002F015C09 Message-Id: Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: Quoted-printable MIME-Version: 1.0 X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:,, definitions=2018-11-30_04:,, signatures=0 X-Proofpoint-Spam-Reason: safe I agree with Rodric. In the past, our assumption was that the risk =0D associated with blackbox invocations is higher than with managed =0D invocations. That's why we used two separate invoker pools such that the =0D possible problems with blackbox invocations won't affect any managed =0D invokers and thus, managed invocations.=0D =0D We need to re-evaluate whether we still see the same risk as in the past =0D and whether we see ways to address possible problems.=0D =0D For those who are willing to accept certain risks, Rodric's proposal of =0D providing flexibility via configuration settings may be the right =0D approach. Apparently, the requirements depend on the type of OW =0D environment you run. For a large production environment supporting a large = =0D number of different tenants requirements are different than for your local = =0D development environment.=0D =0D I think we have (at least) these risks:=0D =0D =3D> Docker images consume too much disk space so that no more new images = =0D can be pulled / no new containers can be created.=0D =0D A limited set of Docker images is required to support managed invocations. = =0D The situation is different for blackbox invocations. Potentially each =0D blackbox invocation may lead to a pull of an additional Docker image so =0D that the disk volume storing images and containers may fill up. At the =0D moment, there are no restrictions on the number and size of blackbox =0D images on an invoker. So invokers running blackbox actions have a higher =0D risk for outages.=0D =0D OW does not have any image cleanup strategy and a limited container =0D cleanup strategy. Kubernetes has a garbage collection approach [1]. We may = =0D add a cleanup strategy to OW as well or users need their own cleanup =0D approach outside of OW.=0D =0D =3D> Managed runtime images have more control.=0D =0D All managed runtimes containers run OW code up to the point where the =0D /init REST endpoint can be used to inject end user action code. So we can = =0D prepare the environment and accept the initial socket connection from the = =0D invoker. Our timeout concept bases these socket connections.=0D =0D Blackbox containers have more control and can decide to not even open a =0D listening socket. It's harder for the invoker to deal with such a =0D non-cooperating container and prevent the container from consuming =0D resources.=0D =0D =3D> Blackbox images can bring much more user code than managed actions.=0D =0D It's easier to hide malicious code or tools in a blackbox image.=0D =0D =3D> Blackbox image pulls keep container infrastructure busy.=0D =0D As already mentioned, each blackbox invocation may lead to a pull of an =0D additional Docker image. These pulls consume CPU and network resources =0D impacting other invocations. The pull mechanism could even be used to =0D attack Docker Hub.=0D =0D =0D Any other thoughts or feedback on the risks I described?=0D =0D =0D Regards, Sven.=0D =0D =0D [1] =0D https://kubernetes.io/docs/concepts/cluster-administration/kubelet-garbage-= collection/=0D =0D Rodric Rabbah wrote on 2018/11/28 13:09:12:=0D =0D > From: Rodric Rabbah =0D > To: dev@openwhisk.apache.org=0D > Date: 2018/11/28 13:09=0D > Subject: Re: Don't differentiate between blackbox and whitebox images=0D > =0D > I think it would be a mistake to blindly merge two container pools -=0D > docker actions as run today are pulled as needed and can take a long=0D > time, they=E2=80=99re subject to different kinds of attacks, and can affe= ct =0D > performance of other tenants (higher noisy neighbors). =0D > =0D > If you want to allow more overlap, it should be at least done as a =0D > configuration parameter which allows different pools to continue to =0D > exist (note that for small N as in local dev they=E2=80=99re allowed to = =0D > overlap today). Or, used for overflow only as a way of providing =0D > more elasticity.=0D > =0D > -r=0D > =0D > > On Nov 28, 2018, at 5:56 AM, Christian Bickel =0D wrote:=0D > > =0D > > Hi,=0D > > =0D > > in the past we divided the invokers into blackbox and whitebox =0D invokers.=0D > > This has the effect that one of these two types could be overloaded =0D while=0D > > the other type still has free capacity.=0D > > =0D > > Does anyone see issues on using every invoker for everything?=0D > > Or does anyone see any action items that need to be addressed before =0D we=0D > > invoke every action-type on every invoker?=0D > > =0D > > If not I'll open a PR to not differentiate between these types =0D anymore.=0D > > =0D > > Greetings=0D > > Christian Bickel=0D =0D =0D