From dev-return-2994-archive-asf-public=cust-asf.ponee.io@openwhisk.apache.org Wed Jan 9 15:32:34 2019 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by mx-eu-01.ponee.io (Postfix) with SMTP id E4253180676 for ; Wed, 9 Jan 2019 15:32:33 +0100 (CET) Received: (qmail 48765 invoked by uid 500); 9 Jan 2019 14:32:32 -0000 Mailing-List: contact dev-help@openwhisk.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@openwhisk.apache.org Delivered-To: mailing list dev@openwhisk.apache.org Received: (qmail 48718 invoked by uid 99); 9 Jan 2019 14:32:32 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd3-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 09 Jan 2019 14:32:32 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd3-us-west.apache.org (ASF Mail Server at spamd3-us-west.apache.org) with ESMTP id D1C77180A83 for ; Wed, 9 Jan 2019 14:32:31 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd3-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.907 X-Spam-Level: * X-Spam-Status: No, score=1.907 tagged_above=-999 required=6.31 tests=[DKIMWL_WL_MED=-0.142, DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_ENVFROM_END_DIGIT=0.25, HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd3-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd3-us-west.apache.org [10.40.0.10]) (amavisd-new, port 10024) with ESMTP id B4bMcixvCoi2 for ; Wed, 9 Jan 2019 14:32:30 +0000 (UTC) Received: from mail-ua1-f41.google.com (mail-ua1-f41.google.com [209.85.222.41]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id DF88C6110B for ; Wed, 9 Jan 2019 14:32:29 +0000 (UTC) Received: by mail-ua1-f41.google.com with SMTP id v24so2460206uap.13 for ; Wed, 09 Jan 2019 06:32:29 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to; bh=v80Q0Yny0aFcyJ76IjLwzh0D2Fyy8VupqGoKwkvdOBg=; b=nveAPS9JAEd048y9JuMSnEjLscf1v2TUSMY2gWD4DRKSJi0e5oQFx5XcynZV2vQczs a747QOC2JQcHhubRoTPKXMdBDKSTmMQ8FiJzda86IHPXAWfCpKk5xBtBHTUw8or5nSdJ d8PcSvT1Kp6EhB/sil6KMQZ9+98dzSCGxYyLV6hcG7q5wlaaW+HeX1XTY5cB82MKPrRK tIfqV/9dYEqvqOAByFsGmSsZRKNbwl/WVYJjG7cYHSds3Z62/5QawEtX15Wa0BwFBVW7 pwvpU4ICkCj8TDbfG5YbcgAANE1TdfF3aEHyEjq/QOoXlf79ZZ+Hab6NovhZSZ5OKRWn LdvQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=v80Q0Yny0aFcyJ76IjLwzh0D2Fyy8VupqGoKwkvdOBg=; b=OxpCa2bKE0vPwkqlU8eZeyl56llOAL+ZwGTT9BrnGDXPxi0lD+x0HHG2J628ywur/U r4l/MmkRSAZ/eRmeUNhJ7V7MQVtt6J8yDyw2OuW/FCVPcWV367G5C6zTDtnwdKXbf4YU 0PHtqAleTajXel5+Ct08eJcYg64fXCCjb4a+RXqsizZGx7tr8qIOyc8nnnnm2nOq1HQy zUQ/ciX4XLu5feGBDakICB4aU2Z07Nn8g28q0loPPyzZ1k/3CG4NK+nx8RWcQ+/wWxUt kOZkIwAEip9XwUFuwqLfF/FephYtN/hAv0Tr/0JAoHb1oFNBWykqJq4aQn5jYZCLujTq 6vYA== X-Gm-Message-State: AJcUukcG25aKQRfMJVtMOPwEYLoXgNTm6Eavp7tkMbCcsZs9zmLXva13 hiw28gE9EvoiYfgvi8eaqwiN7Ls+cBEpYQe2rTRXow== X-Google-Smtp-Source: ALg8bN6zVRma3KanIHJkNQOFD0FvnhfmKKyCiaIip/Nl6zPE8DBoWpbMUaOBa3kUSaSV6JHFK+z1cKaPPLD0MNr3OIg= X-Received: by 2002:ab0:7251:: with SMTP id d17mr2359192uap.0.1547044348385; Wed, 09 Jan 2019 06:32:28 -0800 (PST) MIME-Version: 1.0 References: <1547039541.1196760.1629791464.72982C89@webmail.messagingengine.com> In-Reply-To: <1547039541.1196760.1629791464.72982C89@webmail.messagingengine.com> From: Carlos Santana Date: Wed, 9 Jan 2019 09:32:17 -0500 Message-ID: Subject: Re: Persistent Kafka Connections To: dev@openwhisk.apache.org Content-Type: multipart/alternative; boundary="000000000000d65d1d057f075470" --000000000000d65d1d057f075470 Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Is one of those things that works well for a hello world low scale deployment. In practice we saw a lot of actions running in parallel from multiple tenants and all of these even each one doing their caching were opening too many new socket connection to a kafka service and eventually the kafka service started to throttle and rate limit the number of new connections. This is the reason the code it=E2=80=99s there in the repo but deprecated a= nd not recommended. From talking to kafka experts in ibm it was recommended from them that having stateless clients doing a lot of new connection it is a antipattern So they recommended to try to have a low number of connections open and high thruput for each of these connection, this means if you wan to produce from actions at scale and multitenant the actions would need to go thru some type of broker via http and this broker is a stateful service that maintains the socket connection to kafka service. =E2=80=94 Carlos On Wed, Jan 9, 2019 at 8:12 AM Michele Sciabarra wrote: > Hello whiskers, > > I am working on a simple webchat based on OpenWhisk (full disclosure; it > is for the book on OpenWhisk for O'Reilly!) and I am using MessageHub. > > However in the documentation for the messaging package I read that the > /whisk.system/messaging/messageHubProduce is deprecated, and the > recommended way is to use a persistent connection to Kafka. > > This statement is puzzling me. My first idea is "why do not they cache > the connection?", but I checked the code for messageHubProduce.py here: > > > https://github.com/apache/incubator-openwhisk-package-kafka/blob/master/a= ction/messageHubProduce.py > > and looks like the producer is cached, so I wonder if the deprecation of > messageHubProduce is still valid. > > Furthermore, I wrote an action to send messages to MessageHub with the > idea of implementing a cached connection, and apparently works: > > > https://github.com/learning-apache-openwhisk/chapter11-messages/blob/mast= er/src/persistent/main/send.go > > So the question is: is this the correct way to send messages to Kafka > (caching connnections?) Can I recommend this practice in the book? > > > -- > Michele Sciabarra > michele@sciabarra.com > --=20 Carlos Santana --000000000000d65d1d057f075470--