incubator-cassandra-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jonathan Ellis <jbel...@gmail.com>
Subject Re: Assertion
Date Sun, 25 Sep 2011 01:25:50 GMT
I bet this is https://issues.apache.org/jira/browse/CASSANDRA-3253.

On Fri, Sep 23, 2011 at 6:00 PM, Todd Burruss <bburruss@expedia.com> wrote:
> My last test, I cleaned everything, started from scratch and I can still
> repo.  The AssertionError you mention below could be unrelated to my
> writes not working.
>
> This time there are _no_ exceptions or errors reported from any cassandra
> node.  My app runs for a while before writes start failing.  I can kill my
> app and will happen immediately upon restarting it.
>
> I didn't dig as hard last time, but now I can see that nodetool tpstats on
> 3 consecutive machines in the ring have their MutationStage "stuck" with
> piled up requests.  Completed is not going up and pending is not going
> down.  I have concurrent_writes = 32
>
> Here is a sample from one of the machines:
>
> Pool Name     Active   Pending     Completed   Blocked  All time blocked
> MutationStage                    32     1492    1416296   0       0
>
>
>
>
>
> On 9/23/11 3:11 PM, "Brandon Williams" <driftx@gmail.com> wrote:
>
>>On 9/23/11, Todd Burruss <bburruss@expedia.com> wrote:
>>> INFO [HintedHandoff:4] 2011-09-22 22:59:27,626 HintedHandOffManager.java
>>> (line 259) Started hinted handoff for token:
>>> 150124573641590498586782915043427152112 with IP: /10.185.35.39
>>> ERROR [HintedHandoff:4] 2011-09-22 22:59:27,648
>>>AbstractCassandraDaemon.java
>>> (line 133) Fatal exception in thread Thread[HintedHandoff:4,5,main]
>>> java.lang.AssertionError
>>>         at
>>>
>>>org.apache.cassandra.db.HintedHandOffManager.deliverHintsToEndpoint(Hinte
>>>dHandOffManager.java:282)
>>>         at
>>>
>>>org.apache.cassandra.db.HintedHandOffManager.access$100(HintedHandOffMana
>>>ger.java:81)
>>>         at
>>>
>>>org.apache.cassandra.db.HintedHandOffManager$2.runMayThrow(HintedHandOffM
>>>anager.java:333)
>>>         at
>>> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
>>>         at
>>>
>>>java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor
>>>.java:886)
>>>         at
>>>
>>>java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.jav
>>>a:908)
>>>         at java.lang.Thread.run(Thread.java:662)
>>
>>
>>This sounds like you have old hints 1.0 doesn't understand:
>>
>>                assert versionColumn != null;
>>
>>-Brandon
>
>



-- 
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of DataStax, the source for professional Cassandra support
http://www.datastax.com

Mime
View raw message