From dev-return-1776-archive-asf-public=cust-asf.ponee.io@openwhisk.apache.org Fri Mar 30 19:02:30 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id 8ED18180647 for ; Fri, 30 Mar 2018 19:02:29 +0200 (CEST) Received: (qmail 28522 invoked by uid 500); 30 Mar 2018 17:02:28 -0000 Mailing-List: contact dev-help@openwhisk.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@openwhisk.apache.org Delivered-To: mailing list dev@openwhisk.apache.org Received: (qmail 28502 invoked by uid 99); 30 Mar 2018 17:02:27 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd2-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 30 Mar 2018 17:02:27 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd2-us-west.apache.org (ASF Mail Server at spamd2-us-west.apache.org) with ESMTP id 71DCE1A0A38 for ; Fri, 30 Mar 2018 17:02:27 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd2-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.279 X-Spam-Level: * X-Spam-Status: No, score=1.279 tagged_above=-999 required=6.31 tests=[HTML_MESSAGE=2, RCVD_IN_DNSWL_LOW=-0.7, RCVD_IN_MSPIKE_H4=-0.01, RCVD_IN_MSPIKE_WL=-0.01, SPF_PASS=-0.001] autolearn=disabled Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd2-us-west.apache.org [10.40.0.9]) (amavisd-new, port 10024) with ESMTP id Ch4dKpQQHVMh for ; Fri, 30 Mar 2018 17:02:26 +0000 (UTC) Received: from mx0a-001b2d01.pphosted.com (mx0b-001b2d01.pphosted.com [148.163.158.5]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id 9B7E75F27B for ; Fri, 30 Mar 2018 17:02:25 +0000 (UTC) Received: from pps.filterd (m0098419.ppops.net [127.0.0.1]) by mx0b-001b2d01.pphosted.com (8.16.0.22/8.16.0.22) with SMTP id w2UGtgCt109601 for ; Fri, 30 Mar 2018 13:02:24 -0400 Received: from smtp.notes.na.collabserv.com (smtp.notes.na.collabserv.com [192.155.248.93]) by mx0b-001b2d01.pphosted.com with ESMTP id 2h1s32gfwd-1 (version=TLSv1.2 cipher=ECDHE-RSA-AES256-GCM-SHA384 bits=256 verify=NOT) for ; Fri, 30 Mar 2018 13:02:24 -0400 Received: from localhost by smtp.notes.na.collabserv.com with smtp.notes.na.collabserv.com ESMTP for from ; Fri, 30 Mar 2018 17:02:23 -0000 Received: from us1a3-smtp04.a3.dal06.isc4sb.com (10.106.154.237) by smtp.notes.na.collabserv.com (10.106.227.39) with smtp.notes.na.collabserv.com ESMTP; Fri, 30 Mar 2018 17:02:21 -0000 Received: from us1a3-mail91.a3.dal06.isc4sb.com ([10.146.21.47]) by us1a3-smtp04.a3.dal06.isc4sb.com with ESMTP id 2018033017022158-600665 ; Fri, 30 Mar 2018 17:02:21 +0000 MIME-Version: 1.0 In-Reply-To: Subject: Re: Invoker HA on Mesos To: dev@openwhisk.apache.org From: "David P Grove" Date: Fri, 30 Mar 2018 11:58:05 -0500 References: X-KeepSent: 6DC3C101:E2FB16D6-85258260:0058F395; type=4; name=$KeepSent X-Mailer: IBM Notes Release 9.0.1EXT SHF886 February 20, 2018 X-LLNOutbound: False X-Disclaimed: 14663 X-TNEFEvaluated: 1 Content-type: multipart/alternative; Boundary="0__=0ABB08F3DFCB75058f9e8a93df938690918c0ABB08F3DFCB7505" Content-Disposition: inline x-cbid: 18033017-1799-0000-0000-000005E1AD44 X-IBM-SpamModules-Scores: BY=0; FL=0; FP=0; FZ=0; HX=0; KW=0; PH=0; SC=0.417846; ST=0; TS=0; UL=0; ISC=; MB=0.393951 X-IBM-SpamModules-Versions: BY=3.00008772; HX=3.00000241; KW=3.00000007; PH=3.00000004; SC=3.00000255; SDB=6.01010669; UDB=6.00514957; IPR=6.00789947; BA=6.00005891; NDR=6.00000001; ZLA=6.00000005; ZF=6.00000009; ZB=6.00000000; ZP=6.00000000; ZH=6.00000000; ZU=6.00000002; MB=3.00020334; XFM=3.00000015; UTC=2018-03-30 17:02:22 X-IBM-AV-DETECTION: SAVI=unsuspicious REMOTE=unsuspicious XFE=unused X-IBM-AV-VERSION: SAVI=2018-03-30 15:02:13 - 6.00008264 x-cbparentid: 18033017-1800-0000-0000-00003C68B8D1 Message-Id: X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10432:,, definitions=2018-03-30_07:,, signatures=0 X-Proofpoint-Spam-Reason: safe --0__=0ABB08F3DFCB75058f9e8a93df938690918c0ABB08F3DFCB7505 Content-Transfer-Encoding: quoted-printable Content-type: text/plain; charset=US-ASCII Tyson Norris wrote on 03/27/2018 06:25:59 PM: > > Do you have an example of the labels working? I guess the labels are > changed over time through the lifecycle of the container? > Apologies for brutally chopping the email chain; my mail client made a horrible hash of it. Right now, all we are doing with Kube labels is to label each action container with its owning invoker on startup. This lets us delete orphaned containers if the invoker crashes and needs to be restarted. The labeling happens at [1] and the removal of orphans using the labels at [2]. I think the Kube-native version of part of what you are doing with the DistributedData for Mesos would be to add and remove additional labels to give us the option of attaching a new invoker instance to orphaned containers instead of just destroying them. Interacting with the Kubernetes API server to do a labeling operation takes around 10ms, so we couldn't do this on a truly hot path. But we could probably afford to update container labels in parallel with pause/unpause operations, which could enable re-attachment to any paused containers. --dave [1] https://github.com/apache/incubator-openwhisk/blob/0b20df0f725a671f8e51c9e8= 793116476fd22f76/core/invoker/src/main/scala/whisk/core/containerpool/kuber= netes/KubernetesContainerFactory.scala#L81 [2] https://github.com/apache/incubator-openwhisk/blob/0b20df0f725a671f8e51c9e8= 793116476fd22f76/core/invoker/src/main/scala/whisk/core/containerpool/kuber= netes/KubernetesContainerFactory.scala#L57 --0__=0ABB08F3DFCB75058f9e8a93df938690918c0ABB08F3DFCB7505--