incubator-cassandra-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Todd Burruss <bburr...@expedia.com>
Subject Re: Assertion
Date Mon, 26 Sep 2011 22:29:18 GMT
I picked up the 1.0.0-rc1 build and testing now .. But I bet you are
correct

On 9/24/11 6:25 PM, "Jonathan Ellis" <jbellis@gmail.com> wrote:

>I bet this is https://issues.apache.org/jira/browse/CASSANDRA-3253.
>
>On Fri, Sep 23, 2011 at 6:00 PM, Todd Burruss <bburruss@expedia.com>
>wrote:
>> My last test, I cleaned everything, started from scratch and I can still
>> repo.  The AssertionError you mention below could be unrelated to my
>> writes not working.
>>
>> This time there are _no_ exceptions or errors reported from any
>>cassandra
>> node.  My app runs for a while before writes start failing.  I can kill
>>my
>> app and will happen immediately upon restarting it.
>>
>> I didn't dig as hard last time, but now I can see that nodetool tpstats
>>on
>> 3 consecutive machines in the ring have their MutationStage "stuck" with
>> piled up requests.  Completed is not going up and pending is not going
>> down.  I have concurrent_writes = 32
>>
>> Here is a sample from one of the machines:
>>
>> Pool Name     Active   Pending     Completed   Blocked  All time blocked
>> MutationStage                    32     1492    1416296   0       0
>>
>>
>>
>>
>>
>> On 9/23/11 3:11 PM, "Brandon Williams" <driftx@gmail.com> wrote:
>>
>>>On 9/23/11, Todd Burruss <bburruss@expedia.com> wrote:
>>>> INFO [HintedHandoff:4] 2011-09-22 22:59:27,626
>>>>HintedHandOffManager.java
>>>> (line 259) Started hinted handoff for token:
>>>> 150124573641590498586782915043427152112 with IP: /10.185.35.39
>>>> ERROR [HintedHandoff:4] 2011-09-22 22:59:27,648
>>>>AbstractCassandraDaemon.java
>>>> (line 133) Fatal exception in thread Thread[HintedHandoff:4,5,main]
>>>> java.lang.AssertionError
>>>>         at
>>>>
>>>>org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(Hin
>>>>te
>>>>dHandOffManager.java:282)
>>>>         at
>>>>
>>>>org.apache.cassandra.db.HintedHandOffManager.access$100(HintedHandOffMa
>>>>na
>>>>ger.java:81)
>>>>         at
>>>>
>>>>org.apache.cassandra.db.HintedHandOffManager$2.runMayThrow(HintedHandOf
>>>>fM
>>>>anager.java:333)
>>>>         at
>>>> 
>>>>org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
>>>>         at
>>>>
>>>>java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecut
>>>>or
>>>>.java:886)
>>>>         at
>>>>
>>>>java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.j
>>>>av
>>>>a:908)
>>>>         at java.lang.Thread.run(Thread.java:662)
>>>
>>>
>>>This sounds like you have old hints 1.0 doesn't understand:
>>>
>>>                assert versionColumn != null;
>>>
>>>-Brandon
>>
>>
>
>
>
>-- 
>Jonathan Ellis
>Project Chair, Apache Cassandra
>co-founder of DataStax, the source for professional Cassandra support
>http://www.datastax.com


Mime
View raw message