zookeeper-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Enrico Olivelli <eolive...@gmail.com>
Subject Re: How to scale ZooKeeper to support 10K concurrent connections?
Date Fri, 27 Sep 2019 18:10:58 GMT
Yue
As Jorn said you should introduce some kind of scalable middleware.
Zookeeper shouldn't be used in the hotpath.

If you want something simple and based on components from Zookeeper
ecosystem maybe you can give a look to this simple task broker service,
Majordodo (full disclosure, I am one of the maintainers of the project)

https://github.com/diennea/majordodo

Enrico


Il ven 27 set 2019, 19:39 Jörn Franke <jornfranke@gmail.com> ha scritto:

> Put the Solr request on a SQS queue using your 10k instances and have 10
> or so worker working on the queue to put it in Solr. Having 10k connections
> just because lambda creates that many instances does not make sense for no
> database service.
>
> > Am 27.09.2019 um 19:01 schrieb Yue Shen <shyue2010@gmail.com>:
> >
> > Dear ZooKeeper users,
> >
> > I have a special use case, in which I use AWS lambda service.
> >
> > Inside the lambda service logic, it goes to ZooKeeper to check the worker
> > for the data, if exists,  connect to the worker endpoint and send the
> data.
> > If the worker isn't assigned, the logic will post a new assignment, and
> > wait for it to be assigned to a worker. There is a coordinator to watch
> the
> > new assignment and assign tasks.
> >
> > My problem comes with AWS Lambda service, which can launch tens of
> > thousands of calls. When this happens, I found many calls get timeout.
> The
> > active connections to ZooKeeper plateau around 6500.
> >
> > BTW, I run ZooKeeper as 3 node ensemble, run on Quorum.
> >
> > How can I scale ZooKeeper to support more concurrent connections?
> >
> > Thank you,
> > Yue
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message