zookeeper-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Yue Shen <shyue2...@gmail.com>
Subject How to scale ZooKeeper to support 10K concurrent connections?
Date Fri, 27 Sep 2019 17:01:23 GMT
Dear ZooKeeper users,

I have a special use case, in which I use AWS lambda service.

Inside the lambda service logic, it goes to ZooKeeper to check the worker
for the data, if exists,  connect to the worker endpoint and send the data.
If the worker isn't assigned, the logic will post a new assignment, and
wait for it to be assigned to a worker. There is a coordinator to watch the
new assignment and assign tasks.

My problem comes with AWS Lambda service, which can launch tens of
thousands of calls. When this happens, I found many calls get timeout. The
active connections to ZooKeeper plateau around 6500.

BTW, I run ZooKeeper as 3 node ensemble, run on Quorum.

How can I scale ZooKeeper to support more concurrent connections?

Thank you,

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message