cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jack Krupansky <jack.krupan...@gmail.com>
Subject Re: 10000+ CF support from Cassandra
Date Wed, 24 Jun 2015 18:39:00 GMT
I would say that it's mostly a performance issue, tied to memory
management, but the main problem is that a large number of tables invites a
whole host of clluster management difficulties that require... expert
attention, which then means you need an expert to maintain and enhance it.

Cassandra scales in two ways: number of rows and number of nodes, but not
number of tables. Both number of tables and number of columns per row need
to be kept moderate for your cluster to be manageable and perform well.

Adding a tenant ID to your table partition key is the optimal approach to
multi-tenancy at this stage with Cassandra. That, and maybe also assigning
subsets of the tenants to different tables, as well as having separate
clusters if your number of tenants and rows gets too large.

-- Jack Krupansky

On Wed, Jun 24, 2015 at 11:55 AM, Arun Chaitanya <chaitan64arun@gmail.com>
wrote:

> Hi Jack,
>
> When I mean entries, I meant rows. Each column family has about 200
> columns.
>
> > Disabling of slab allocation is an expert-only feature - its use is
> generally an anti-pattern, not recommended.
> I understand this and have seen this recommendation at several places. I
> want to understand the consequences? Is it performance, maintenance or
> scalability, that is at stake.
>
> In our use case we have about 3000 column families (ofcourse modelled in
> RDBMS). If I were to limit to 250 column families, do you advise us to use
> multiple clusters (the problem being cost ineffective)?
>
> If we were to use a single cluster and support 3000 column families, the
> only idea is to group few column families and store them in one column
> family. In this case, grouping is a difficult task, imo. And if we want an
> abstraction of grouping for developer, we need special connector for
> Hadoop/Spark systems. So I do not want to enter this territory.
>
> Sorry for such questions, but I am still wondering if I am the only one
> facing this problem.
>
> Thanks a lot,
> Arun
>
>
>
> On Wed, Jun 24, 2015 at 10:28 PM, Jack Krupansky <jack.krupansky@gmail.com
> > wrote:
>
>> By entries, do you mean rows or columns? Please clarify how many columns
>> each of your tables has, and how many rows you are populating for each
>> table.
>>
>> In case I didn't make it clear earlier, limit yourself to "low hundreds"
>> (like 250) of tables and you should be fine. Thousands of tables is a clear
>> anti-pattern for Cassandra - not recommended. If it works for you, great,
>> but if not, don't say you weren't warned.
>>
>> Disabling of slab allocation is an expert-only feature - its use is
>> generally an anti-pattern, not recommended.
>>
>> -- Jack Krupansky
>>
>> On Sun, Jun 21, 2015 at 10:55 PM, Arun Chaitanya <chaitan64arun@gmail.com
>> > wrote:
>>
>>> Hello All,
>>>
>>> Now we settled on the following approach. I want to know if there are
>>> any problems that you foresee in the production environment.
>>>
>>> Our Approach: Use  Off Heap Memory
>>>
>>> Modifications to default cassandra.yaml and cassandra-env.sh
>>> ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
>>>  * memory_allocator: JEMallocAllocator (https://issues.apache.org/jira/browse/CASSANDRA-7883)
>>>  * memtable_allocation_type: offheap_objects
>>>
>>>  By above two, the slab allocation (https://issues.apache.org/jira/browse/CASSANDRA-5935),
which requires
>>>  1MB heap memory per table, is disabled. The memory for table metadata, caches
and memtable are thus
>>>  allocated natively and does not affect GC performance.
>>>
>>>  * tombstone_failure_threshold: 100000000
>>>
>>>    Without this, C* throws TombstoneOverwhelmingException while in startup.
>>>    This setting looks problematic so I want to know why just creating tables
makes so many tombstones ...
>>>
>>>  * -XX:+UseG1GC
>>>
>>>    It is good for reducing GC time.
>>>    Without this, full GCs > 1s are observed.
>>>
>>> We created 5000 column families with about 1000 entries per column
>>> family. The read/write performance seems to stable.
>>> The problem we saw is with startup time.
>>>
>>>      Cassandra Start Time (s) 20
>>>
>>>
>>>
>>> 349  Average CPU Usage (%) 40
>>>
>>>
>>>
>>> 49.65  GC Actitivy (%) 2.6
>>>
>>>
>>>
>>> 0.6
>>> Thanks a lot in advance.
>>>
>>> On Tue, Jun 2, 2015 at 11:26 AM, graham sanderson <graham@vast.com>
>>> wrote:
>>>
>>>> > I strongly advise against this approach.
>>>>> Jon, I think so too. But so you actually foresee any problems with
>>>>> this approach?
>>>>> I can think of a few. [I want to evaluate if we can live with this
>>>>> problem]
>>>>>
>>>>>
>>>>> Just to be clear, I’m not saying this is a great approach, I AM saying
>>>> that it may be better than having 10000+ CFs, which was the original
>>>> question (it really depends on the use case which wasn’t well defined)…
map
>>>> size limit may be a problem, and then there is the CQL vs thrift question
>>>> which could start a flame war; ideally CQL maps should give you the same
>>>> flexibility as arbitrary thrift columns
>>>>
>>>> On Jun 1, 2015, at 9:44 PM, Jonathan Haddad <jon@jonhaddad.com> wrote:
>>>>
>>>> > Sorry for this naive question but how important is this tuning? Can
>>>> this have a huge impact in production?
>>>>
>>>> Massive.  Here's a graph of when we did some JVM tuning at my previous
>>>> company:
>>>>
>>>>
>>>> http://33.media.tumblr.com/5d0efca7288dc969c1ac4fc3d36e0151/tumblr_inline_mzvj254quj1rd24f4.png
>>>>
>>>> About an order of magnitude difference in performance.
>>>>
>>>> Jon
>>>>
>>>> On Mon, Jun 1, 2015 at 7:20 PM Arun Chaitanya <chaitan64arun@gmail.com>
>>>> wrote:
>>>>
>>>>> Thanks Jon and Jack,
>>>>>
>>>>> > I strongly advise against this approach.
>>>>> Jon, I think so too. But so you actually foresee any problems with
>>>>> this approach?
>>>>> I can think of a few. [I want to evaluate if we can live with this
>>>>> problem]
>>>>>
>>>>>    - No more CQL.
>>>>>    - No data types, everything needs to be a blob.
>>>>>    - Limited clustering Keys and default clustering order.
>>>>>
>>>>> > First off, different workloads need different tuning.
>>>>> Sorry for this naive question but how important is this tuning? Can
>>>>> this have a huge impact in production?
>>>>>
>>>>> > You might want to consider a model where you have an application
>>>>> layer that maps logical tenant tables into partition keys within a single
>>>>> large Casandra table, or at least a relatively small number of  Cassandra
>>>>> tables. It will depend on the typical size of your tenant tables - very
>>>>> small ones would make sense within a single partition, while larger ones
>>>>> should have separate partitions for a tenant's data. The key here is
that
>>>>> tables are expensive, but partitions are cheap and scale very well with
>>>>> Cassandra.
>>>>> We are actually trying similar approach. But we don't want to expose
>>>>> this to application layer. We are attempting to hide this and provide
an
>>>>> API.
>>>>>
>>>>> > Finally, you said "10 clusters", but did you mean 10 nodes? You
>>>>> might want to consider a model where you do indeed have multiple clusters,
>>>>> where each handles a fraction of the tenants, since there is no need
for
>>>>> separate tenants to be on the same cluster.
>>>>> I meant 10 clusters. We want to split our tables across multiple
>>>>> clusters if above approach is not possible. [But it seems to be very
costly]
>>>>>
>>>>> Thanks,
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>>
>>>>> On Fri, May 29, 2015 at 5:49 AM, Jack Krupansky <
>>>>> jack.krupansky@gmail.com> wrote:
>>>>>
>>>>>> How big is each of the tables - are they all fairly small or fairly
>>>>>> large? Small as in no more than thousands of rows or large as in
tens of
>>>>>> millions or hundreds of millions of rows?
>>>>>>
>>>>>> Small tables are are not ideal for a Cassandra cluster since the
rows
>>>>>> would be spread out across the nodes, even though it might make more
sense
>>>>>> for each small table to be on a single node.
>>>>>>
>>>>>> You might want to consider a model where you have an application
>>>>>> layer that maps logical tenant tables into partition keys within
a single
>>>>>> large Casandra table, or at least a relatively small number of Cassandra
>>>>>> tables. It will depend on the typical size of your tenant tables
- very
>>>>>> small ones would make sense within a single partition, while larger
ones
>>>>>> should have separate partitions for a tenant's data. The key here
is that
>>>>>> tables are expensive, but partitions are cheap and scale very well
with
>>>>>> Cassandra.
>>>>>>
>>>>>> Finally, you said "10 clusters", but did you mean 10 nodes? You might
>>>>>> want to consider a model where you do indeed have multiple clusters,
where
>>>>>> each handles a fraction of the tenants, since there is no need for
separate
>>>>>> tenants to be on the same cluster.
>>>>>>
>>>>>>
>>>>>> -- Jack Krupansky
>>>>>>
>>>>>> On Tue, May 26, 2015 at 11:32 PM, Arun Chaitanya <
>>>>>> chaitan64arun@gmail.com> wrote:
>>>>>>
>>>>>>> Good Day Everyone,
>>>>>>>
>>>>>>> I am very happy with the (almost) linear scalability offered
by C*.
>>>>>>> We had a lot of problems with RDBMS.
>>>>>>>
>>>>>>> But, I heard that C* has a limit on number of column families
that
>>>>>>> can be created in a single cluster.
>>>>>>> The reason being each CF stores 1-2 MB on the JVM heap.
>>>>>>>
>>>>>>> In our use case, we have about 10000+ CF and we want to support
>>>>>>> multi-tenancy.
>>>>>>> (i.e 10000 * no of tenants)
>>>>>>>
>>>>>>> We are new to C* and being from RDBMS background, I would like
to
>>>>>>> understand how to tackle this scenario from your advice.
>>>>>>>
>>>>>>> Our plan is to use Off-Heap memtable approach.
>>>>>>> http://www.datastax.com/dev/blog/off-heap-memtables-in-Cassandra-2-1
>>>>>>>
>>>>>>> Each node in the cluster has following configuration
>>>>>>> 16 GB machine (8GB Cassandra JVM + 2GB System + 6GB Off-Heap)
>>>>>>> IMO, this should be able to support 1000 CF with no(very less)
>>>>>>> impact on performance and startup time.
>>>>>>>
>>>>>>> We tackle multi-tenancy using different keyspaces.(Solution I
found
>>>>>>> on the web)
>>>>>>>
>>>>>>> Using this approach we can have 10 clusters doing the job. (We
>>>>>>> actually are worried about the cost)
>>>>>>>
>>>>>>> Can you please help us evaluate this strategy? I want to hear
>>>>>>> communities opinion on this.
>>>>>>>
>>>>>>> My major concerns being,
>>>>>>>
>>>>>>> 1. Is Off-Heap strategy safe and my assumption of 16 GB supporting
>>>>>>> 1000 CF right?
>>>>>>>
>>>>>>> 2. Can we use multiple keyspaces to solve multi-tenancy? IMO,
the
>>>>>>> number of column families increase even when we use multiple
keyspace.
>>>>>>>
>>>>>>> 3. I understand the complexity using multi-cluster for single
>>>>>>> application. The code base will get tightly coupled with infrastructure.
Is
>>>>>>> this the right approach?
>>>>>>>
>>>>>>> Any suggestion is appreciated.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Arun
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>

Mime
View raw message