cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Justin Lin <linjianfeng...@gmail.com>
Subject Re: Too many keyspaces causes cql connection to time out ?
Date Tue, 24 May 2016 17:54:00 GMT
so i guess i have to 1) increase the heap size or 2) reduce the number of
keyspaces/column families.

Thanks for you confirmation.

On Tue, May 24, 2016 at 10:08 AM, Eric Stevens <mightye@gmail.com> wrote:

> Large numbers of tables is generally recommended against.  Each table has
> a fixed on-heap memory overhead, and by your description it sounds like you
> might have as many as 12,000 total tables when you start running into
> trouble.
>
> With such a small heap to begin with, you've probably used up most of the
> available heap just with managing the tables.  This is supported by the big
> STW pauses you're observing in Cassandra's logs.
>
> On Tue, May 24, 2016 at 11:04 AM Justin Lin <linjianfengqrh@gmail.com>
> wrote:
>
>> We are exploring cassandra's limit by creating a lot of keyspaces with
>> moderate number of column families (roughly 40 - 50) per keyspace and we
>> have a problem after we reach certain amount of keyspaces, that cqlsh
>> starts to time out when connecting to cassandra.
>>
>> This is our cassandra setup. We have one cassandra running locally and we
>> assign 4GB memory to jvm, set the memtable_allocation_type to be onheap and
>> use default memtable_heap_space_in_mb, which i believe is 2GB. The
>> cassandra version is 2.1.9.
>>
>> So after we create more than 250 keyspaces, cqlsh starts to times out
>> when connecting to cassandra in most case. (Sometimes it still can connect
>> to cassandra). And from cassandra log, we can see it takes roughly 3
>> seconds to do gc when there is an incoming connection. And the gc is the
>> only difference between the timeout connection and the successful
>> connection. So we suspect this Stop-The-World GC might block the connection
>> until it times out. This is the log that i think is relevant.
>>
>> INFO 20160524-060930.028882 ::  Initializing
>> sandbox_20160524_t06_09_18.table1
>>
>> INFO 20160524-060933.908008 ::  G1 Young Generation GC in 551ms.  G1 Eden
>> Space: 222298112 -> 0; G1 Old Gen: 2811821584 -> 3034119696;
>>
>> INFO 20160524-060933.908043 ::  G1 Old Generation GC in 2631ms.  G1 Old
>> Gen: 3034119696 -> 2290099032;
>>
>> We suspect the issue might relate to the reported bug as well:
>> https://issues.apache.org/jira/browse/CASSANDRA-9291
>> but not really sure about it.
>>
>> Sorry for the setup, so my question is
>> 1) is the connection timeout related to gc or the tomestone in
>> system.schame_keyspaces table?
>> 2) how can we fix this problem?
>> 3) I did some tests by dropping a bunch of keyspaces after timing out and
>> it seems to fix the problem as i never got another time out. Is this the
>> only way to fix it?
>>
>> Thanks a lot for your help.
>>
>> --
>> come on
>>
>


-- 
come on

Mime
View raw message