incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Yang <teddyyyy...@gmail.com>
Subject Re: weird freeze
Date Thu, 08 Sep 2011 16:49:47 GMT
ok, found this is likely due to GC... I'm seeing full GC that runs
20seconds without actually removing anything:


35240.526: [Full GC [PSYoungGen: 1760704K->1668729K(1848128K)]
[PSOldGen: 4095999K->4095999K(4096000K)] 5856703K->5764729K(5944128K)
[PSPermGen: 24885K->24871K(25152K)], 20.6780790 secs] [Times:
user=20.60 sys=0.00, real=20.68 secs]


I'm trying to do a MAT anaysis now

On Wed, Sep 7, 2011 at 11:01 AM, yang yang <yangyangyyy@yahoo.com.cn> wrote:
> my cassandra server (from github source head, plus I added some code that just
> calls thrift.CassandraServer.batch_mutate() in the same JVM )   runs fine under
> heavy load for about 5 hours, then it froze,
>
> when I checked the jstack, all the mutation stages are blocked on
> Table.switchlock.ReadLock().lock(),
> and the corresponding writeLock is being held by Table.maybeSwitchMemlock(),
> which is blocked on CommitLog.instance.getContext(), which is blocked on
> get()'ting from a future.
>
>
> "COMMIT-LOG-WRITER" prio=10 tid=0x00002aaab4a6c000 nid=0x66df waiting on
> condition [0x00000000428d6000]
>   java.lang.Thread.State: WAITING (parking)
>        at sun.misc.Unsafe.$$YJP$$park(Native Method)
>        - parking to wait for  <0x00000007f72bb5f0> (a
> java.util.concurrent.FutureTask$Sync)
>        at sun.misc.Unsafe.park(Unsafe.java)
>        at java.util.concurrent.locks.LockSupport.park(LockSupport.java:156)
>        at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:811)
>
>        at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:969)
>
>        at
> java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1281)
>
>        at java.util.concurrent.FutureTask$Sync.innerGet(FutureTask.java:218)
>        at java.util.concurrent.FutureTask.get(FutureTask.java:83)
>        at
> org.apache.cassandra.db.commitlog.CommitLog.getContext(CommitLog.java:388)
>        at
> org.apache.cassandra.db.ColumnFamilyStore.maybeSwitchMemtable(ColumnFamilyStore.java:661)
>
>        at
> org.apache.cassandra.db.ColumnFamilyStore.forceFlush(ColumnFamilyStore.java:733)
>        at
> org.apache.cassandra.db.commitlog.CommitLog.createNewSegment(CommitLog.java:575)
>        at
> org.apache.cassandra.db.commitlog.CommitLog.access$300(CommitLog.java:82)
>        at
> org.apache.cassandra.db.commitlog.CommitLog$LogRecordAdder.run(CommitLog.java:598)
>
>        at
> org.apache.cassandra.db.commitlog.PeriodicCommitLogExecutorService$1.runMayThrow(PeriodicCommitLogExecutorService.java:49)
>
>        at
> org.apache.cassandra.utils.WrappedRunnable.run(WrappedRunnable.java:30)
>        at java.lang.Thread.run(Thread.java:662)
>
>
>
> I tried to search for the thread that does the getting from future in commitLog,
> but it's not present. anyone has an idea why the CommitLog.instance.getContext()
> is not returning? the getContext() Callable is not yet scheduled by the
> CommitLog executor?
>
>
> Thanks a lot
> Yang

Mime
View raw message