incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From yang yang <yangyang...@yahoo.com.cn>
Subject weird freeze
Date Wed, 07 Sep 2011 18:01:17 GMT
my cassandra server (from github source head, plus I added some code that just 
calls thrift.CassandraServer.batch_mutate() in the same JVM )   runs fine under 
heavy load for about 5 hours, then it froze, 

when I checked the jstack, all the mutation stages are blocked on 
Table.switchlock.ReadLock().lock(), 
and the corresponding writeLock is being held by Table.maybeSwitchMemlock(), 
which is blocked on CommitLog.instance.getContext(), which is blocked on 
get()'ting from a future.


"COMMIT-LOG-WRITER" prio=10 tid=0x00002aaab4a6c000 nid=0x66df waiting on 
condition [0x00000000428d6000]
   java.lang.Thread.State: WAITING (parking)
        at sun.misc.Unsafe.$$YJP$$park(Native Method)
        - parking to wait for  <0x00000007f72bb5f0> (a 
java.util.concurrent.FutureTask$Sync)
        at sun.misc.Unsafe.park(Unsafe.java)
        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811)

        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:969)

        at 
java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1281)

        at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:218)
        at java.util.concurrent.FutureTask.get(FutureTask.java:83)
        at 
org.apache.cassandra.db.commitlog.CommitLog.getContext(CommitLog.java:388)
        at 
org.apache.cassandra.db.ColumnFamilyStore.maybeSwitchMemtable(ColumnFamilyStore.java:661)

        at 
org.apache.cassandra.db.ColumnFamilyStore.forceFlush(ColumnFamilyStore.java:733)
        at 
org.apache.cassandra.db.commitlog.CommitLog.createNewSegment(CommitLog.java:575)
        at 
org.apache.cassandra.db.commitlog.CommitLog.access$300(CommitLog.java:82)
        at 
org.apache.cassandra.db.commitlog.CommitLog$LogRecordAdder.run(CommitLog.java:598)

        at 
org.apache.cassandra.db.commitlog.PeriodicCommitLogExecutorService$1.runMayThrow(PeriodicCommitLogExecutorService.java:49)

        at 
org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
        at java.lang.Thread.run(Thread.java:662)



I tried to search for the thread that does the getting from future in commitLog, 
but it's not present. anyone has an idea why the CommitLog.instance.getContext() 
is not returning? the getContext() Callable is not yet scheduled by the 
CommitLog executor?


Thanks a lot
Yang
Mime
View raw message