incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alexis Rodríguez <arodrig...@inconcertcc.com>
Subject Re: Cassandra HANGS after some writes
Date Tue, 13 Aug 2013 13:51:43 GMT
Naresh, are you deploying cassandra in windows?

If that is the case you may need to change the data and commitlog
directories in cassandra.yaml. Also you should check the log directories.

See the section 2.1  http://wiki.apache.org/cassandra/GettingStarted


On Tue, Aug 13, 2013 at 8:28 AM, Naresh Yadav <nyadav.ait@gmail.com> wrote:

>
> Thanks Alain, will avoid caps....i am newbie to cassandra, just started
> using 2 weeks back..
>
> Here are JConsole screenshots just 5mins after cassandra freezed :
>
> http://i.imgur.com/3oUBjKU.png
> http://i.imgur.com/2O4PrKb.png
> http://i.imgur.com/zxhFzr1.png               4:05 is time cassandra
> freezed thats why decline in no of threads
> http://i.imgur.com/ScgAciv.png
> Uploaded complete system.log of cassandra till freeze :
> http://www.scribd.com/doc/159949231/Cassandrasystem-log
>
> Observation : As in my usecase i am storing 1lakh
> combinations(527insert,506update,954select) each parallel by 100 threads in
> batch of 1000...
> Sometimes it works till 1000 batch then hangs but sometimes it completes
> 10000 then hangs and once even worked for more than lakh....
> Same hardware Same settings of cassandra i see random behaviour of
> performance..
>
> Thanks
> Naresh
>
>
> On Tue, Aug 13, 2013 at 3:48 PM, Alain RODRIGUEZ <arodrime@gmail.com>wrote:
>
>> Hi Naresh.
>>
>> First thing, there is no need of caps in here. People reading this ML is
>> here to help when they have time and skills enough to do so. So please,
>> chill out and do not use caps to show how much desperate you are.
>>
>> Concerning your problem, the only abnormal thing I was able to find in
>> your logs is
>>
>>
>>    1. ERROR [NonPeriodicTasks:1] 2013-08-13 01:52:42,106
>>    SSTableDeletingTask.java (line 72) Unable to delete
>>    \var\lib\cassandra\data\system\schema_columnfamilies\system-schema_columnfamilies-ic-241-Data.db
>>    (it will be removed on server restart; we'll also retry after GC)
>>
>>
>> I don't think this should keep C* hanging.
>>
>> Do you have something on kernel logs ?
>>
>> Do you have monitor any metrics like disk throughput / heap used / cpu
>> load / iowait which are known as being bottlenecks / pertinent metrics ?
>>
>> Alain
>>
>>
>> 2013/8/13 Naresh Yadav <nyadav.ait@gmail.com>
>>
>>>
>>> Hi All,
>>>
>>> I have single node cassandra using  CQL using datastax java driver 1.0.1
>>> and cassandra verison 1.2.6.
>>>
>>> *Infrastructure :* 16GB machine with 8GB heap given to cassandra, i7
>>> processor.. DEFAULT cassandra.yaml no change done by me.
>>> -Xms1G^
>>>  -Xmx12G^         no other change in cassandra.bat
>>>
>>> *Problem :
>>> _________
>>>  *cassandra Freezes after some writes and i see no action on cassandra
>>> console for an hour...all Native_Transport threads are also killed....my
>>> program keeps running NO ERROR comes.......when i connect with cql that
>>> works........ In start it creates 16 NativeTransport threads and after
>>> 10-15 minutes Total threads goes to 128...Just before it hangs, With
>>> JCONSOLE when i see Native_Transport threads then i find most of them in
>>> state as :
>>>
>>> http://pastebin.com/DeShpHtP
>>>
>>> *Load on cassandra : *
>>> ___________________
>>> i am running a usecase which stores Combinations(my project terminology)
>>> in cassandra....Currently testing storing 2.5 lakh combinations with 100
>>> parallel threads..each thread storing one combination...real case i need to
>>> support of many CRORES but that would need different hardware and multi
>>> node cluster...
>>>
>>> In Storing ONE combination takes around 2sec and involves :
>>>
>>> 527 INSERT INTO queries ....
>>>
>>> 506 UPDATE queries ....
>>>
>>> 954 SELECT queries ....
>>>
>>> 100 parallel threads parallely storing 100 combinations....
>>> *MY CASSANDRA LOGS :*
>>>
>>> http://pastebin.com/CnNvA9x3
>>>
>>> Please look last 100-200 lines of log because that is time it freezed
>>>
>>>
>>> PLEASE HELP ME OUT, I AM NOT ABLE TO PROCEED FROM 1 week...
>>>
>>
>>
>
>
>

Mime
View raw message