cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sammy Yu (JIRA)" <j...@apache.org>
Subject [jira] Issue Comment Edited: (CASSANDRA-405) Race condition with ConcurrentLinkedHashMap
Date Tue, 01 Sep 2009 04:59:32 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-405?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12749758#action_12749758
] 

Sammy Yu edited comment on CASSANDRA-405 at 8/31/09 9:59 PM:
-------------------------------------------------------------

We did multiple stack dumps they are all the same.  There is no progression. 

In this stack trace there are two sets of stack traces
pool-1-thread-{63,62,61,59,58,54,53,51,49,47} and ROW-READ-STAGE{8,7,5,4,3,2,1}:
"ROW-READ-STAGE:8" prio=10 tid=0x00007f1b78b52000 nid=0x1945 runnable [0x0000000046532000]
   java.lang.Thread.State: RUNNABLE
	at com.reardencommerce.kernel.collections.shared.evictable.ConcurrentLinkedHashMap$Node.appendToTail(ConcurrentLinkedHashMap.java:536)
	at com.reardencommerce.kernel.collections.shared.evictable.ConcurrentLinkedHashMap.putIfAbsent(ConcurrentLinkedHashMap.java:281)
	at com.reardencommerce.kernel.collections.shared.evictable.ConcurrentLinkedHashMap.put(ConcurrentLinkedHashMap.java:256)
	at org.apache.cassandra.io.SSTableReader.getPosition(SSTableReader.java:241)
	at org.apache.cassandra.db.filter.SSTableNamesIterator. (SSTableNamesIterator.java:46)
	at org.apache.cassandra.db.filter.NamesQueryFilter.getSSTableColumnIterator(NamesQueryFilter.java:69)
	at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1445)
	at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1379)
	at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1398)
	at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1379)
	at org.apache.cassandra.db.Table.getRow(Table.java:589)
	at org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:65)
	at org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:78)
	at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:44)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
	at java.lang.Thread.run(Thread.java:619)

ROW-READ-STAGE6 is a little different:
"ROW-READ-STAGE:6" prio=10 tid=0x00007f1b78b4e000 nid=0x1943 runnable [0x0000000046330000]
   java.lang.Thread.State: RUNNABLE
	at com.reardencommerce.kernel.collections.shared.evictable.ConcurrentLinkedHashMap$Node.appendToTail(ConcurrentLinkedHashMap.java:540)
	at com.reardencommerce.kernel.collections.shared.evictable.ConcurrentLinkedHashMap.putIfAbsent(ConcurrentLinkedHashMap.java:281)
	at com.reardencommerce.kernel.collections.shared.evictable.ConcurrentLinkedHashMap.put(ConcurrentLinkedHashMap.java:256)
	at org.apache.cassandra.io.SSTableReader.getPosition(SSTableReader.java:241)
	at org.apache.cassandra.db.filter.SSTableNamesIterator. (SSTableNamesIterator.java:46)
	at org.apache.cassandra.db.filter.NamesQueryFilter.getSSTableColumnIterator(NamesQueryFilter.java:69)
	at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1445)
	at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1379)
	at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1398)
	at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1379)
	at org.apache.cassandra.db.Table.getRow(Table.java:589)
	at org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:65)
	at org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:78)
	at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:44)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
	at java.lang.Thread.run(Thread.java:619)

All the other pool-1-threads are in the readLock waiting state due to the writeLock
"pool-1-thread-12425" prio=10 tid=0x00007f1b7857e000 nid=0x3fe6 waiting on condition [0x00007f1892528000]
   java.lang.Thread.State: WAITING (parking)
	at sun.misc.Unsafe.park(Native Method)
	- parking to wait for  0x00007f1b8534e848> (a java.util.concurrent.locks.ReentrantReadWriteLock$FairSync)
	at java.util.concurrent.locks.LockSupport.park(LockSupport.java:158)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:747)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireShared(AbstractQueuedSynchronizer.java:877)
	at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireShared(AbstractQueuedSynchronizer.java:1197)
	at java.util.concurrent.locks.ReentrantReadWriteLock$ReadLock.lock(ReentrantReadWriteLock.java:594)
	at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1412)
	at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1379)
	at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1398)
	at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1379)
	at org.apache.cassandra.db.Table.getRow(Table.java:589)
	at org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:65)
	at org.apache.cassandra.service.StorageProxy.weakReadLocal(StorageProxy.java:609)
	at org.apache.cassandra.service.StorageProxy.readProtocol(StorageProxy.java:320)
	at org.apache.cassandra.service.CassandraServer.readColumnFamily(CassandraServer.java:92)
	at org.apache.cassandra.service.CassandraServer.getSlice(CassandraServer.java:173)
	at org.apache.cassandra.service.CassandraServer.get_slice(CassandraServer.java:213)
	at org.apache.cassandra.service.Cassandra$Processor$get_slice.process(Cassandra.java:551)
	at org.apache.cassandra.service.Cassandra$Processor.process(Cassandra.java:539)
	at org.apache.thrift.server.TThreadPoolServer$WorkerProcess.run(TThreadPoolServer.java:252)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
	at java.lang.Thread.run(Thread.java:619)

      was (Author: sammy.yu):
    In this stack trace there are two sets of threads that are stuck in the iterator
pool-1-thread-{63,62,61,59,58,54,53,51,49,47} and ROW-READ-STAGE{8,7,5,4,3,2,1}:
"ROW-READ-STAGE:8" prio=10 tid=0x00007f1b78b52000 nid=0x1945 runnable [0x0000000046532000]
   java.lang.Thread.State: RUNNABLE
	at com.reardencommerce.kernel.collections.shared.evictable.ConcurrentLinkedHashMap$Node.appendToTail(ConcurrentLinkedHashMap.java:536)
	at com.reardencommerce.kernel.collections.shared.evictable.ConcurrentLinkedHashMap.putIfAbsent(ConcurrentLinkedHashMap.java:281)
	at com.reardencommerce.kernel.collections.shared.evictable.ConcurrentLinkedHashMap.put(ConcurrentLinkedHashMap.java:256)
	at org.apache.cassandra.io.SSTableReader.getPosition(SSTableReader.java:241)
	at org.apache.cassandra.db.filter.SSTableNamesIterator. (SSTableNamesIterator.java:46)
	at org.apache.cassandra.db.filter.NamesQueryFilter.getSSTableColumnIterator(NamesQueryFilter.java:69)
	at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1445)
	at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1379)
	at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1398)
	at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1379)
	at org.apache.cassandra.db.Table.getRow(Table.java:589)
	at org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:65)
	at org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:78)
	at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:44)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
	at java.lang.Thread.run(Thread.java:619)

ROW-READ-STAGE6 is a little different:
"ROW-READ-STAGE:6" prio=10 tid=0x00007f1b78b4e000 nid=0x1943 runnable [0x0000000046330000]
   java.lang.Thread.State: RUNNABLE
	at com.reardencommerce.kernel.collections.shared.evictable.ConcurrentLinkedHashMap$Node.appendToTail(ConcurrentLinkedHashMap.java:540)
	at com.reardencommerce.kernel.collections.shared.evictable.ConcurrentLinkedHashMap.putIfAbsent(ConcurrentLinkedHashMap.java:281)
	at com.reardencommerce.kernel.collections.shared.evictable.ConcurrentLinkedHashMap.put(ConcurrentLinkedHashMap.java:256)
	at org.apache.cassandra.io.SSTableReader.getPosition(SSTableReader.java:241)
	at org.apache.cassandra.db.filter.SSTableNamesIterator. (SSTableNamesIterator.java:46)
	at org.apache.cassandra.db.filter.NamesQueryFilter.getSSTableColumnIterator(NamesQueryFilter.java:69)
	at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1445)
	at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1379)
	at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1398)
	at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1379)
	at org.apache.cassandra.db.Table.getRow(Table.java:589)
	at org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:65)
	at org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:78)
	at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:44)
	at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
	at java.lang.Thread.run(Thread.java:619)
  
> Race condition with ConcurrentLinkedHashMap
> -------------------------------------------
>
>                 Key: CASSANDRA-405
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-405
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>    Affects Versions: 0.4
>            Reporter: Chris Goffinet
>         Attachments: stack.log.gz
>
>
> We are seeing a race condition with ConcurrentLinkedHashMap using appendToTail. We could
remove the ConcurrentLinkedHashMap for now until that's resolved.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message