incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Naresh Yadav <nyadav....@gmail.com>
Subject Re: Cassandra HANGS after some writes
Date Tue, 13 Aug 2013 11:28:53 GMT
Thanks Alain, will avoid caps....i am newbie to cassandra, just started
using 2 weeks back..

Here are JConsole screenshots just 5mins after cassandra freezed :

http://i.imgur.com/3oUBjKU.png
http://i.imgur.com/2O4PrKb.png
http://i.imgur.com/zxhFzr1.png               4:05 is time cassandra freezed
thats why decline in no of threads
http://i.imgur.com/ScgAciv.png
Uploaded complete system.log of cassandra till freeze :
http://www.scribd.com/doc/159949231/Cassandrasystem-log

Observation : As in my usecase i am storing 1lakh
combinations(527insert,506update,954select) each parallel by 100 threads in
batch of 1000...
Sometimes it works till 1000 batch then hangs but sometimes it completes
10000 then hangs and once even worked for more than lakh....
Same hardware Same settings of cassandra i see random behaviour of
performance..

Thanks
Naresh

On Tue, Aug 13, 2013 at 3:48 PM, Alain RODRIGUEZ <arodrime@gmail.com> wrote:

> Hi Naresh.
>
> First thing, there is no need of caps in here. People reading this ML is
> here to help when they have time and skills enough to do so. So please,
> chill out and do not use caps to show how much desperate you are.
>
> Concerning your problem, the only abnormal thing I was able to find in
> your logs is
>
>
>    1. ERROR [NonPeriodicTasks:1] 2013-08-13 01:52:42,106
>    SSTableDeletingTask.java (line 72) Unable to delete
>    \var\lib\cassandra\data\system\schema_columnfamilies\system-schema_columnfamilies-ic-241-Data.db
>    (it will be removed on server restart; we'll also retry after GC)
>
>
> I don't think this should keep C* hanging.
>
> Do you have something on kernel logs ?
>
> Do you have monitor any metrics like disk throughput / heap used / cpu
> load / iowait which are known as being bottlenecks / pertinent metrics ?
>
> Alain
>
>
> 2013/8/13 Naresh Yadav <nyadav.ait@gmail.com>
>
>>
>> Hi All,
>>
>> I have single node cassandra using  CQL using datastax java driver 1.0.1
>> and cassandra verison 1.2.6.
>>
>> *Infrastructure :* 16GB machine with 8GB heap given to cassandra, i7
>> processor.. DEFAULT cassandra.yaml no change done by me.
>> -Xms1G^
>>  -Xmx12G^         no other change in cassandra.bat
>>
>> *Problem :
>> _________
>>  *cassandra Freezes after some writes and i see no action on cassandra
>> console for an hour...all Native_Transport threads are also killed....my
>> program keeps running NO ERROR comes.......when i connect with cql that
>> works........ In start it creates 16 NativeTransport threads and after
>> 10-15 minutes Total threads goes to 128...Just before it hangs, With
>> JCONSOLE when i see Native_Transport threads then i find most of them in
>> state as :
>>
>> http://pastebin.com/DeShpHtP
>>
>> *Load on cassandra : *
>> ___________________
>> i am running a usecase which stores Combinations(my project terminology)
>> in cassandra....Currently testing storing 2.5 lakh combinations with 100
>> parallel threads..each thread storing one combination...real case i need to
>> support of many CRORES but that would need different hardware and multi
>> node cluster...
>>
>> In Storing ONE combination takes around 2sec and involves :
>>
>> 527 INSERT INTO queries ....
>>
>> 506 UPDATE queries ....
>>
>> 954 SELECT queries ....
>>
>> 100 parallel threads parallely storing 100 combinations....
>> *MY CASSANDRA LOGS :*
>>
>> http://pastebin.com/CnNvA9x3
>>
>> Please look last 100-200 lines of log because that is time it freezed
>>
>>
>> PLEASE HELP ME OUT, I AM NOT ABLE TO PROCEED FROM 1 week...
>>
>
>

Mime
View raw message