incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jonathan Ellis <jbel...@gmail.com>
Subject Re: Read Latency
Date Sun, 17 Oct 2010 01:18:58 GMT
Thanks.  Take 2 attached.

On Sat, Oct 16, 2010 at 3:37 PM, Wayne <wav100@gmail.com> wrote:
> ERROR [pool-1-thread-64] 2010-10-16 20:27:55,396 Cassandra.java (line 1280)
> Internal error processing get_slice
> java.lang.AssertionError
>     at
> org.apache.cassandra.service.ReadResponseResolver.resolve(ReadResponseResolver.java:88)
>     at
> org.apache.cassandra.service.ReadResponseResolver.resolve(ReadResponseResolver.java:44)
>     at
> org.apache.cassandra.service.QuorumResponseHandler.get(QuorumResponseHandler.java:86)
>     at
> org.apache.cassandra.service.StorageProxy.strongRead(StorageProxy.java:489)
>     at
> org.apache.cassandra.service.StorageProxy.readProtocol(StorageProxy.java:353)
>     at
> org.apache.cassandra.thrift.CassandraServer.readColumnFamily(CassandraServer.java:97)
>     at
> org.apache.cassandra.thrift.CassandraServer.getSlice(CassandraServer.java:180)
>     at
> org.apache.cassandra.thrift.CassandraServer.multigetSliceInternal(CassandraServer.java:262)
>     at
> org.apache.cassandra.thrift.CassandraServer.get_slice(CassandraServer.java:223)
>     at
> org.apache.cassandra.thrift.Cassandra$Processor$get_slice.process(Cassandra.java:1272)
>     at
> org.apache.cassandra.thrift.Cassandra$Processor.process(Cassandra.java:1166)
>     at
> org.apache.cassandra.thrift.CustomTThreadPoolServer$WorkerProcess.run(CustomTThreadPoolServer.java:167)
>     at
> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
>     at
> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
>     at java.lang.Thread.run(Thread.java:619)
>
>
> On Sat, Oct 16, 2010 at 9:52 AM, Jonathan Ellis <jbellis@gmail.com> wrote:
>>
>> Stack trace from cassandra log?
>>
>> On Sat, Oct 16, 2010 at 6:50 AM, Wayne <wav100@gmail.com> wrote:
>> > While doing a read I get a TApplicationException: Internal Error
>> > processing
>> > get_slice.
>> >
>> > On Fri, Oct 15, 2010 at 5:49 PM, Jonathan Ellis <jbellis@gmail.com>
>> > wrote:
>> >>
>> >> On Fri, Oct 15, 2010 at 2:21 PM, Wayne <wav100@gmail.com> wrote:
>> >> > The optimization definitely shaved off some time. Now it is running
>> >> > about 3x
>> >> > CFSTATS reported time. Below are the logs.
>> >> >
>> >> > There is a ~300ms time frame after the last ResponseVerbHandler prior
>> >> > to
>> >> > the
>> >> > resolver starting. Based on a quorum read the response resolver
>> >> > should
>> >> > kick
>> >> > after 2 reads come in correct? That would mean it waited 500ms before
>> >> > starting. Where is this time going?
>> >>
>> >> It's going to deserialize the replies to see if it has both the data
>> >> and enough digests to call resolve().  Then resolve deserializes them
>> >> a 2nd time, so there's an easy win there caching the first
>> >> deserialize, in the patch attached.  (Applies on top of the previous
>> >> one, which has been committed to the 0.6 svn branch at
>> >> https://svn.apache.org/repos/asf/cassandra/branches/cassandra-0.6.)
>> >>
>> >> > There is still the 3+ second delay between the last 2 entries. Is
>> >> > this
>> >> > the
>> >> > thrift server?
>> >>
>> >> It's converting the reply from Cassandra's internal representation to
>> >> thrift, and sending it to the client.  I suspect most of the time is
>> >> the actual sending/waiting for the client to read part.  Patch 2 also
>> >> includes a debug statement after the convert-to-thrift stage to
>> >> verify.
>> >>
>> >> --
>> >> Jonathan Ellis
>> >> Project Chair, Apache Cassandra
>> >> co-founder of Riptano, the source for professional Cassandra support
>> >> http://riptano.com
>> >
>> >
>>
>>
>>
>> --
>> Jonathan Ellis
>> Project Chair, Apache Cassandra
>> co-founder of Riptano, the source for professional Cassandra support
>> http://riptano.com
>
>



-- 
Jonathan Ellis
Project Chair, Apache Cassandra
co-founder of Riptano, the source for professional Cassandra support
http://riptano.com

Mime
View raw message