accumulo-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Konstantin Pelykh <kpel...@gmail.com>
Subject Re: Any limits on Accumulo namespaces/tables?
Date Thu, 25 Jun 2015 21:39:36 GMT
When I start working on this issue I'll look in detail if it's possible to
modify Scanner to avoid having Zookeeper watches for all tables availbale
in Accumulo.
I'll share my results if I find any bottlenecks or will be able to optimize
scanner to work with many namespaces.

--------
Big Data / Search Consultant
LinkedIn: linkedin.com/in/kpelykh <http://www.linkedin.com/in/kpelykh>
Website: www.kpelykh.com

On Thu, Jun 25, 2015 at 11:38 AM, Christopher <ctubbsii@apache.org> wrote:

> In theory, no, there aren't limits. In practice, you'd be limited by
> the amount of memory you have available in ZooKeeper, and you may have
> trouble (currently) with lots of ZooKeeper watchers which monitor
> configuration changes for tables.
>
> There may be data structures in the API and throughout the code which
> contain a complete list of tables, also, and that could hurt in terms
> of memory.
>
> In short, we don't define a limit, but there may be bottlenecks you
> might encounter. If you do find these, we may be able to make
> improvements, so let us know. If this because a big problem, we may
> impose some upper bound in the future, just so people can reason about
> their environment better, and we could offer more concrete numbers on
> what is feasible.
>
> --
> Christopher L Tubbs II
> http://gravatar.com/ctubbsii
>
>
> On Thu, Jun 25, 2015 at 2:31 PM, Konstantin Pelykh <kpelykh@gmail.com>
> wrote:
> > Hi everyone,
> >
> > Are there any limits on a number of namespaces and tables in Accumulo?
> > For my current application I'm planning to use multiple namespaces to
> > separate data for different clients. Each namespace will have 8 tables.
> > Would it be posible to have 100,500, 1000 namespaces in Accumulo? How
> will
> > such namespace layout impact performance of accumulo scanner? As far as I
> > rememebr every time when scanner is created  it gets the whole list of
> > namespaces/tables from zookeeper and it slows down performace as number
> of
> > namespaces increases.
> >
> > Thanks!
> > Konstantin
> > --------
> > Big Data / Search Consultant
> > LinkedIn: linkedin.com/in/kpelykh
> > Website: www.kpelykh.com
>

Mime
View raw message