From user-return-12181-archive-asf-public=cust-asf.ponee.io@zookeeper.apache.org Fri Sep 27 19:10:39 2019 Return-Path: X-Original-To: archive-asf-public@cust-asf.ponee.io Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [207.244.88.153]) by mx-eu-01.ponee.io (Postfix) with SMTP id 09FDC180638 for ; Fri, 27 Sep 2019 21:10:38 +0200 (CEST) Received: (qmail 52729 invoked by uid 500); 27 Sep 2019 19:10:38 -0000 Mailing-List: contact user-help@zookeeper.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@zookeeper.apache.org Delivered-To: mailing list user@zookeeper.apache.org Received: (qmail 52713 invoked by uid 99); 27 Sep 2019 19:10:37 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 Sep 2019 19:10:37 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id 09FAAC0542 for ; Fri, 27 Sep 2019 19:10:37 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 2.05 X-Spam-Level: ** X-Spam-Status: No, score=2.05 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, DKIM_VALID_AU=-0.1, DKIM_VALID_EF=-0.1, FREEMAIL_ENVFROM_END_DIGIT=0.25, HTML_MESSAGE=2, RCVD_IN_DNSWL_NONE=-0.0001, SPF_HELO_NONE=0.001, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd1-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-he-de.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id ibl1b0Ns6QyR for ; Fri, 27 Sep 2019 19:10:34 +0000 (UTC) Received-SPF: Pass (mailfrom) identity=mailfrom; client-ip=2a00:1450:4864:20::336; helo=mail-wm1-x336.google.com; envelope-from=shyue2010@gmail.com; receiver= Received: from mail-wm1-x336.google.com (mail-wm1-x336.google.com [IPv6:2a00:1450:4864:20::336]) by mx1-he-de.apache.org (ASF Mail Server at mx1-he-de.apache.org) with ESMTPS id 6A73D7DD5C for ; Fri, 27 Sep 2019 19:10:34 +0000 (UTC) Received: by mail-wm1-x336.google.com with SMTP id 5so7123243wmg.0 for ; Fri, 27 Sep 2019 12:10:34 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to; bh=Dbr9LU8El575u1WeeS5S3H6D5DJmkP+Srxw/soJj7/M=; b=K+cbjOlDSsvLDQ8apJtTzI8SxlbH8wu6/VBnS9GH7+5xwFk6g6Njl/s8tN90bVugis U1gLQ43saAD1m9qNapsHFX7yEMFwKPW1gU8vNYdeyUcBzsXplxRSlN6NxmXuN4/0E1vQ fe5cHLfmaweytJAwfn7cha/rrKvfg2bzFpzxDsXcwe+1lmGrauiNO9O0rGHmAdk0OHlP INTLFfRJTKvQHdcLhksPA2HdvEd5Ozml4iQXCq7yILS0RTQ7z+AkzwWUBwoTjzalN6p5 aSrz5g0X7il15fRV+GyYPxEDaVFv+vKdeiOf/UehovrM6YKASdjBxDzCFs9AZZK0y8mD /+qA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to; bh=Dbr9LU8El575u1WeeS5S3H6D5DJmkP+Srxw/soJj7/M=; b=U6jIdKPVAs26eORQmPUTEwjoIFY/CbH+8kB+MtwNPxqLtIAlNAT6MW4SBcGJ6v69bX VlKGbYhS8Bf+XJxRNDWLbrqMGzDUY899idW+aCOK1NJJgd1Dxf+XVrSIay6dVNRlhF/u PDs2IRUSXx6JFeVSMcBfTFtf0GkurW6FZlgTj+CU/kaHyAA8SAtQmNGnz5V7pM6yReci /9ljRKo5WXV4xqeqo6iddtRMRO+01xJf4e5AtTDYeduq1brWWx2D5pgh7MzykqS4nPIy o2Xvg9WUPYtkmxMZFQqysnSxOczilO7yoz5/AdejLzPWcpRvuh6RBMo1RIfyVa0eHsED Omkg== X-Gm-Message-State: APjAAAUr6H0xgfh4AwblwcI5U0Qi5ykBoA9A/Xwl5m807ebgM9vLCPel zUG94PrVrHpDj1L8kXi/XwoPIRf4EMHnKs8tZmxBfjn7 X-Google-Smtp-Source: APXvYqxG6wHZzqyd4WuJNeebEmeGPKKHdmuQ3mHWIAO+SXX6arU7/0phulAZ5rwtEBonYRdcAvSZV/1CsFcAaiLONGU= X-Received: by 2002:a7b:c00e:: with SMTP id c14mr8084195wmb.158.1569611428366; Fri, 27 Sep 2019 12:10:28 -0700 (PDT) MIME-Version: 1.0 References: <6E602A7A-1AA7-417C-8B4E-F41A7B6D9243@gmail.com> In-Reply-To: From: Yue Shen Date: Fri, 27 Sep 2019 12:10:17 -0700 Message-ID: Subject: Re: How to scale ZooKeeper to support 10K concurrent connections? To: user@zookeeper.apache.org Content-Type: multipart/alternative; boundary="0000000000009f907c05938da38f" --0000000000009f907c05938da38f Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Thank you, Enrico. I will check majordodo. Let you know if we can use it for better architecture. Yue On Fri, Sep 27, 2019 at 11:36 AM Enrico Olivelli wrote: > Yue > As Jorn said you should introduce some kind of scalable middleware. > Zookeeper shouldn't be used in the hotpath. > > If you want something simple and based on components from Zookeeper > ecosystem maybe you can give a look to this simple task broker service, > Majordodo (full disclosure, I am one of the maintainers of the project) > > https://github.com/diennea/majordodo > > Enrico > > > Il ven 27 set 2019, 19:39 J=C3=B6rn Franke ha scri= tto: > > > Put the Solr request on a SQS queue using your 10k instances and have 1= 0 > > or so worker working on the queue to put it in Solr. Having 10k > connections > > just because lambda creates that many instances does not make sense for > no > > database service. > > > > > Am 27.09.2019 um 19:01 schrieb Yue Shen : > > > > > > =EF=BB=BFDear ZooKeeper users, > > > > > > I have a special use case, in which I use AWS lambda service. > > > > > > Inside the lambda service logic, it goes to ZooKeeper to check the > worker > > > for the data, if exists, connect to the worker endpoint and send the > > data. > > > If the worker isn't assigned, the logic will post a new assignment, a= nd > > > wait for it to be assigned to a worker. There is a coordinator to wat= ch > > the > > > new assignment and assign tasks. > > > > > > My problem comes with AWS Lambda service, which can launch tens of > > > thousands of calls. When this happens, I found many calls get timeout= . > > The > > > active connections to ZooKeeper plateau around 6500. > > > > > > BTW, I run ZooKeeper as 3 node ensemble, run on Quorum. > > > > > > How can I scale ZooKeeper to support more concurrent connections? > > > > > > Thank you, > > > Yue > > > --0000000000009f907c05938da38f--