incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Naresh Yadav <nyadav....@gmail.com>
Subject Re: Cassandra HANGS after some writes
Date Tue, 13 Aug 2013 14:21:42 GMT
Hi Alex,

Yes i am testing in development environment of Windows 7 64bit.
I left default yaml then cassandra created "var" folder and created data,
log, cache folders in it.....I tried commit log on different harddisk but
this problem not solved with that....I guess this problem is somewhat
related to deadlock in Native Transport threads...thats why cassandra is
hanging indefinitly..

Naresh

On Tue, Aug 13, 2013 at 7:21 PM, Alexis Rodríguez <
arodriguez@inconcertcc.com> wrote:

> Naresh, are you deploying cassandra in windows?
>
> If that is the case you may need to change the data and commitlog
> directories in cassandra.yaml. Also you should check the log directories.
>
> See the section 2.1  http://wiki.apache.org/cassandra/GettingStarted
>
>
> On Tue, Aug 13, 2013 at 8:28 AM, Naresh Yadav <nyadav.ait@gmail.com>wrote:
>
>>
>> Thanks Alain, will avoid caps....i am newbie to cassandra, just started
>> using 2 weeks back..
>>
>> Here are JConsole screenshots just 5mins after cassandra freezed :
>>
>> http://i.imgur.com/3oUBjKU.png
>> http://i.imgur.com/2O4PrKb.png
>> http://i.imgur.com/zxhFzr1.png               4:05 is time cassandra
>> freezed thats why decline in no of threads
>> http://i.imgur.com/ScgAciv.png
>> Uploaded complete system.log of cassandra till freeze :
>> http://www.scribd.com/doc/159949231/Cassandrasystem-log
>>
>> Observation : As in my usecase i am storing 1lakh
>> combinations(527insert,506update,954select) each parallel by 100 threads in
>> batch of 1000...
>> Sometimes it works till 1000 batch then hangs but sometimes it completes
>> 10000 then hangs and once even worked for more than lakh....
>> Same hardware Same settings of cassandra i see random behaviour of
>> performance..
>>
>> Thanks
>> Naresh
>>
>>
>> On Tue, Aug 13, 2013 at 3:48 PM, Alain RODRIGUEZ <arodrime@gmail.com>wrote:
>>
>>> Hi Naresh.
>>>
>>> First thing, there is no need of caps in here. People reading this ML is
>>> here to help when they have time and skills enough to do so. So please,
>>> chill out and do not use caps to show how much desperate you are.
>>>
>>> Concerning your problem, the only abnormal thing I was able to find in
>>> your logs is
>>>
>>>
>>>    1. ERROR [NonPeriodicTasks:1] 2013-08-13 01:52:42,106
>>>    SSTableDeletingTask.java (line 72) Unable to delete
>>>    \var\lib\cassandra\data\system\schema_columnfamilies\system-schema_columnfamilies-ic-241-Data.db
>>>    (it will be removed on server restart; we'll also retry after GC)
>>>
>>>
>>> I don't think this should keep C* hanging.
>>>
>>> Do you have something on kernel logs ?
>>>
>>> Do you have monitor any metrics like disk throughput / heap used / cpu
>>> load / iowait which are known as being bottlenecks / pertinent metrics ?
>>>
>>> Alain
>>>
>>>
>>> 2013/8/13 Naresh Yadav <nyadav.ait@gmail.com>
>>>
>>>>
>>>> Hi All,
>>>>
>>>> I have single node cassandra using  CQL using datastax java driver
>>>> 1.0.1 and cassandra verison 1.2.6.
>>>>
>>>> *Infrastructure :* 16GB machine with 8GB heap given to cassandra, i7
>>>> processor.. DEFAULT cassandra.yaml no change done by me.
>>>> -Xms1G^
>>>>  -Xmx12G^         no other change in cassandra.bat
>>>>
>>>> *Problem :
>>>> _________
>>>>  *cassandra Freezes after some writes and i see no action on cassandra
>>>> console for an hour...all Native_Transport threads are also killed....my
>>>> program keeps running NO ERROR comes.......when i connect with cql that
>>>> works........ In start it creates 16 NativeTransport threads and after
>>>> 10-15 minutes Total threads goes to 128...Just before it hangs, With
>>>> JCONSOLE when i see Native_Transport threads then i find most of them in
>>>> state as :
>>>>
>>>> http://pastebin.com/DeShpHtP
>>>>
>>>> *Load on cassandra : *
>>>> ___________________
>>>> i am running a usecase which stores Combinations(my project
>>>> terminology) in cassandra....Currently testing storing 2.5 lakh
>>>> combinations with 100 parallel threads..each thread storing one
>>>> combination...real case i need to support of many CRORES but that would
>>>> need different hardware and multi node cluster...
>>>>
>>>> In Storing ONE combination takes around 2sec and involves :
>>>>
>>>> 527 INSERT INTO queries ....
>>>>
>>>> 506 UPDATE queries ....
>>>>
>>>> 954 SELECT queries ....
>>>>
>>>> 100 parallel threads parallely storing 100 combinations....
>>>> *MY CASSANDRA LOGS :*
>>>>
>>>> http://pastebin.com/CnNvA9x3
>>>>
>>>> Please look last 100-200 lines of log because that is time it freezed
>>>>
>>>>
>>>> PLEASE HELP ME OUT, I AM NOT ABLE TO PROCEED FROM 1 week...
>>>>
>>>
>>>
>>
>>
>>
>

Mime
View raw message