From dev-return-2650-archive-asf-public=cust-asf.ponee.io@openwhisk.apache.org Fri Sep 21 20:34:41 2018 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id 5066C180656 for ; Fri, 21 Sep 2018 20:34:40 +0200 (CEST) Received: (qmail 33335 invoked by uid 500); 21 Sep 2018 18:34:39 -0000 Mailing-List: contact dev-help@openwhisk.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@openwhisk.apache.org Delivered-To: mailing list dev@openwhisk.apache.org Received: (qmail 33323 invoked by uid 99); 21 Sep 2018 18:34:38 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 21 Sep 2018 18:34:38 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id 3D866C67C2 for ; Fri, 21 Sep 2018 18:34:38 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: -0.103 X-Spam-Level: X-Spam-Status: No, score=-0.103 tagged_above=-999 required=6.31 tests=[DKIMWL_WL_MED=-0.001, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd1-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-us.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id thRtsYh8KTI9 for ; Fri, 21 Sep 2018 18:34:34 +0000 (UTC) Received: from mail-qk1-f170.google.com (mail-qk1-f170.google.com [209.85.222.170]) by mx1-lw-us.apache.org (ASF Mail Server at mx1-lw-us.apache.org) with ESMTPS id 621285F51F for ; Fri, 21 Sep 2018 18:34:34 +0000 (UTC) Received: by mail-qk1-f170.google.com with SMTP id w73-v6so3009984qkb.9 for ; Fri, 21 Sep 2018 11:34:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=from:content-transfer-encoding:mime-version:date:subject:message-id :references:in-reply-to:to; bh=oBHpywYyieLJmZS5mGKYzCDjef/Zkx74A3eiBE9l294=; b=PrkJqnI7A+v9U+5hMQR3M664dYRBpIULvBZbDYS+fufPB3UdnEHmGqWloIASJVosO8 Si9fI70rK0RSTWHkSYhncgO84DJCg27RFQC58VcvcctFxZn1GyTGFWYXkHTZIOS46qAp N2plWbN85EVRUv87PHyRDebLix6X1k6NOZt1TP0prZ2oeuuyhiDqdFcmDqRgklLDaFJo t3b0roQUmsAWZc6iPSgxkntgMrSSU3NNZeKM5+xOhq9SHgHR9bXqUzPLnrfA8598Yirb GosX9MA2uAufToLBVUvrXoN5GhFuCPHaI5KWipeJ884WIz610jEwb0a86jRDktDtuKiR ByXQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:from:content-transfer-encoding:mime-version:date :subject:message-id:references:in-reply-to:to; bh=oBHpywYyieLJmZS5mGKYzCDjef/Zkx74A3eiBE9l294=; b=UKTROHk0yvKWyx2CLCHnIQQodUbGedRYJihACZS+3zgN1Z5oFXUQEuUPjnYGXyZVP9 iPqXnw2NtJHBORTPFEOs2TXJmTIl38skp/dI9ftlrmhQa/sh1pHz2STsF1UR03Km+wyc E9IFxhvdHvnjkwcfHctFLiAl156FMXCZG1ZHOFojdXvAT7iP+gG9VtIZgoQSQrAcLJNG 4k7GQPMlOrCxuh/cziwIouw4KQIyAPFYHXwrVhlfcRD7ro+dxA3dR0hZgOCJL6/tPd2e R7aE1fOWVJzhOuv9d0fjHixamoN69V8VCFwEJCVDVBoik4joqcmhuzk8ifkNJZF5J1Vn H+Kw== X-Gm-Message-State: APzg51CTsoyjIEo7p4aytjbSCKHw2IllCCZo6ZmJrEGobYGT6EzvfCTr fCAvcFHy4QNJUVfoMQ+asLyU+NrA X-Google-Smtp-Source: ANB0VdZh62UYJ/brS6zAWs4QzV0x8ni8UBvj/2Xvb5NjckXspXWiAZInIF6Kin26mNlacf3u8vLusQ== X-Received: by 2002:a37:ddcf:: with SMTP id u76-v6mr11842025qku.184.1537554873643; Fri, 21 Sep 2018 11:34:33 -0700 (PDT) Received: from ?IPv6:2600:1017:b812:5171:2971:ea14:71d7:2ff4? ([2600:1017:b812:5171:2971:ea14:71d7:2ff4]) by smtp.gmail.com with ESMTPSA id n25-v6sm18926978qtb.32.2018.09.21.11.34.32 for (version=TLS1_2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128/128); Fri, 21 Sep 2018 11:34:32 -0700 (PDT) From: Rodric Rabbah Content-Type: text/plain; charset=us-ascii Content-Transfer-Encoding: quoted-printable Mime-Version: 1.0 (1.0) Date: Fri, 21 Sep 2018 14:34:32 -0400 Subject: Re: Proposal to Remove Artifact Store Polling for Blocking Invocations Message-Id: References: In-Reply-To: To: dev@openwhisk.apache.org X-Mailer: iPhone Mail (15G77) Thanks James for the explanation and patches. It sounds like there should be= two separate PRs, one to address the bug and the other to remove polling. W= hat do you think? -r > On Sep 21, 2018, at 1:09 PM, James W Dubee wrote: >=20 >=20 >=20 >=20 >=20 > Hello OpenWhisk developers, >=20 > When a blocking action is invoked, the controller waits for that action's > response from the invoker and also polls the artifact store for the same > response. Usually blocking invocation responses are obtained from the > invoker. However, there are instances when the invocation response is > retrieved from the artifact store instead. =46rom observation, the most > likely scenario for a blocking activation to be retrieve from the artifact= > store is when an action generates a response that exceeds the maximum > allowed Kafka message size for the "completed" topic. However, this > situation should not occur as large action responses are meant to be > truncated by the invoker to the allowed maximum Kafka message size for the= > corresponding topic. >=20 > Currently artifact store polling for activation records is masking a bug > involving large action responses. While OpenWhisk provides a configuration= > value, whisk.activation.payload.max, for what one would assume would allow= > for adjustments to be made to the maximum activation record size, this > configuration value only adjusts the Kafka topic that is used to schedule > actions for invocation. Instead the Kafka topic used to communicate the > completion of an action always uses the default value for > KAFKA_MESSAGE_MAX_BYTES, which is ~1MB. Additionally, the invoker truncate= s > action responses to the whisk.activation.payload.max value even though > whisk.activation.payload.max is not being applied properly to the > "completed" Kafka topic. More over, this truncation does not account for > data added to the action response by the Kafka producer during > serialization, so an action response may fail to be sent to the "completed= " > topic even if its actual action response size adheres to the topic's size > limitations. As a result, any action response plus the size of > serialization done by the Kafka producer that exceeds ~1MB will be > retrieved via artifact store polling. >=20 > Performance degradation appears to occur when an activation recorded is > retrieved via artifact store polling. Artifact store polling occurs every > 15 seconds for a blocking invocation. Since the response of an action that= > generates a payload greater than ~1MB can not be sent through the > "completed" Kafka topic, that action's activation record must be retrieved= > via polling. Even though such an action may complete in milliseconds, the > end user will not get back the activation response for at least 15 seconds= > due to the polling logic in the controller. >=20 > I have submitted a pull request to remove the polling mechanism and also > fix the large action response bug. The pull request can be found here: > https://github.com/apache/incubator-openwhisk/pull/4033. >=20 > Regards, > James Dubee