cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Lu Ming" <xl...@live.com>
Subject MESSAGE-DESERIALIZER-POOL is slow down by SliceFromReadCommand
Date Wed, 23 Jun 2010 15:50:33 GMT
Yesterday and today , I found there are many pending message (>100,000) to be de-serialized
in MESSAGE-DESERIALIZER-POOL.
and I do the thread dump, It seems that some "SliceRead" is NOT executed by StageManager.READ_STAGE
and executed  by MESSAGE-DESERIALIZER-POOL.
It is somewhat strange. Read from file  is much slower than deserialization, so MESSAGE-DESERIALIZER-POOL
slow down.

According to MessageService.receive(), when the MessageType() of a Message is not found in
StageManager, its VerbHandle will be executed by MESSAGE-DESERIALIZER-POOL.
So the question is:   which type of  SliceFromReadCommand will executed by MESSAGE-DESERIALIZER-POOL?



"MESSAGE-DESERIALIZER-POOL:1" - Thread t@23
   java.lang.Thread.State: RUNNABLE
 at java.io.RandomAccessFile.readBytes(Native Method)
 at java.io.RandomAccessFile.read(RandomAccessFile.java:322)
 at org.apache.cassandra.io.util.BufferedRandomAccessFile.fillBuffer(BufferedRandomAccessFile.java:210)
 at org.apache.cassandra.io.util.BufferedRandomAccessFile.seek(BufferedRandomAccessFile.java:247)
 at org.apache.cassandra.io.SSTableReader.getFileDataInput(SSTableReader.java:515)
 at org.apache.cassandra.db.filter.SSTableSliceIterator.<init>(SSTableSliceIterator.java:54)
 at org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:68)
 at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:862)
 at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:773)
 at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:742)
 at org.apache.cassandra.db.Table.getRow(Table.java:381)
 at org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:65)
 at org.apache.cassandra.db.ReadVerbHandler.doVerb(ReadVerbHandler.java:70)
 at org.apache.cassandra.net.MessageDeliveryTask.run(MessageDeliveryTask.java:40)
 at java.util.concurrent.ThreadPoolExecutor$CallerRunsPolicy.rejectedExecution(ThreadPoolExecutor.java:1746)
 at java.util.concurrent.ThreadPoolExecutor.reject(ThreadPoolExecutor.java:767)
 at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:658)
 at org.apache.cassandra.net.MessagingService.receive(MessagingService.java:365)
 at org.apache.cassandra.net.MessageDeserializationTask.run(MessageDeserializationTask.java:55)
 at java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:441)
 at java.util.concurrent.FutureTask$Sync.innerRun(FutureTask.java:303)
 at java.util.concurrent.FutureTask.run(FutureTask.java:138)
 at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
 at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
 at java.lang.Thread.run(Thread.java:619)

Mime
View raw message