Thank you, Enrico.

I will check majordodo. Let you know if we can use it for better
architecture.

Yue

On Fri, Sep 27, 2019 at 11:36 AM Enrico Olivelli <eolivelli@gmail.com>
wrote:

> Yue
> As Jorn said you should introduce some kind of scalable middleware.
> Zookeeper shouldn't be used in the hotpath.
>
> If you want something simple and based on components from Zookeeper
> ecosystem maybe you can give a look to this simple task broker service,
> Majordodo (full disclosure, I am one of the maintainers of the project)
>
> https://github.com/diennea/majordodo
>
> Enrico
>
>
> Il ven 27 set 2019, 19:39 Jörn Franke <jornfranke@gmail.com> ha scritto:
>
> > Put the Solr request on a SQS queue using your 10k instances and have 10
> > or so worker working on the queue to put it in Solr. Having 10k
> connections
> > just because lambda creates that many instances does not make sense for
> no
> > database service.
> >
> > > Am 27.09.2019 um 19:01 schrieb Yue Shen <shyue2010@gmail.com>:
> > >
> > > Dear ZooKeeper users,
> > >
> > > I have a special use case, in which I use AWS lambda service.
> > >
> > > Inside the lambda service logic, it goes to ZooKeeper to check the
> worker
> > > for the data, if exists,  connect to the worker endpoint and send the
> > data.
> > > If the worker isn't assigned, the logic will post a new assignment, and
> > > wait for it to be assigned to a worker. There is a coordinator to watch
> > the
> > > new assignment and assign tasks.
> > >
> > > My problem comes with AWS Lambda service, which can launch tens of
> > > thousands of calls. When this happens, I found many calls get timeout.
> > The
> > > active connections to ZooKeeper plateau around 6500.
> > >
> > > BTW, I run ZooKeeper as 3 node ensemble, run on Quorum.
> > >
> > > How can I scale ZooKeeper to support more concurrent connections?
> > >
> > > Thank you,
> > > Yue
> >
>