incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Arindam Barua <aba...@247-inc.com>
Subject RE: Read latency issue
Date Fri, 12 Oct 2012 23:48:34 GMT

We instrumented the Cassandra and Hector code adding more logs to check where the time was
being spent.
We found the Cassandra read times to be very low, eg. CassandraServer.getSlice() is only 3ms.

However, on Hector's side, creating a ColumnFamilyTemplate<String, Composite>, and doing
queryColumns() on it takes 90ms. 
Looking at the breakup on the Hector side, it appears ExecutionResult.execute takes ~30ms,
and ColumnFamilyResultWrapper takes ~47ms.
(we are reading around 800 composite columns of 1000 bytes each)

Any idea if this is the expected time to process stuff on Hector/other clients?
Btw, using Hector's SliceQuery() and reading into a List, or Astynax seem to result in similar
times too.

Thanks,
Arindam

-----Original Message-----
From: Arindam Barua [mailto:abarua@247-inc.com] 
Sent: Wednesday, October 03, 2012 10:54 AM
To: user@cassandra.apache.org
Subject: RE: Read latency issue


Thanks for your responses.

Just to be clear our table declaration looks something like this:
CREATE TABLE sessionevents (
  atag text,
  col2 uuid,
  col3 text,
  col4 uuid,
  col5 text,
  col6 text,
  col7 blob,
  col8 text,
  col9 timestamp,
  col10 uuid,
  col11 int,
  col12 uuid,
  PRIMARY KEY (atag, col2, col3, col4)
)

My understanding was that the (full) row key in this case would be the 'atag' values. The
column names would then be composites like (<col2_value>:<col3_value>:<col4_value>:col5),
(<col2_value>: <col3_value>: <col4_value>:col6), (<col2_value>:<col3_value>:<col4_value>:col7)
... (<col2_value>: <col3_value>: <col4_value>:col12). The columns would
be sorted first by col2_values, then by col3 values, etc.

Hence a query like "select * from sessionevents where atag=<foo>", we are specifying
the entire row key, and Cassandra would return all the columns for that row.

>> Using read consistency of ONE reduces the read latency by ~20ms, compared to using
QUORUM.
>It would only have read from the local node. (I think, may be confusing secondary index
reads here).
For read consistency ONE, reading only from one node is my expectation as well, and hence
I'm seeing the reduced read latency compared to read consistency QUORUM. Does that not sound
right?
Btw, with read consistency ONE, we found the reading only happens from one node, but not necessarily
the local node, even if the data is present in the local node. To check this, we turned on
DEBUG logs on all the Cassandra hosts in the ring. We are using replication factor=3 on a
4 node ring, hence mostly the data is present locally. However, we noticed that the coordinator
host on receiving the same request multiple times (i.e with the same row key) , would sometimes
return the data locally, but sometimes would contact another host in the ring to fetch the
data.

Thanks,
Arindam

-----Original Message-----
From: aaron morton [mailto:aaron@thelastpickle.com]
Sent: Wednesday, October 03, 2012 12:32 AM
To: user@cassandra.apache.org
Subject: Re: Read latency issue

> Running a query to like "select * from <table_name> where atag=<foo>", where
'atag' is the first column of the composite key, from either JDBC or Hector (equivalent code),
results in read times of 200-300ms from a remote host on the same network. 

If you send a query to select columns from a row and do not fully specify the row key cassandra
has to do a row scan. 

If you want fast performance specify the full row key. 

> Using read consistency of ONE reduces the read latency by ~20ms, compared to using QUORUM.
It would only have read from the local node. (I think, may be confusing secondary index reads
here).
 
Cheers

-----------------
Aaron Morton
Freelance Developer
@aaronmorton
http://www.thelastpickle.com

On 3/10/2012, at 2:17 AM, Roshni Rajagopal <roshni_rajagopal@hotmail.com> wrote:

> Arindam,
> 
>   Did you also try the cassandra stress tool & compare results?
> 
> I havent done a performance test as yet, the only ones published on 
> the internet are of YCSB on an older version of apache cassandra, and it doesn't seem
to be actively supported or updated http://www.brianfrankcooper.net/pubs/ycsb-v4.pdf.
> 
> The numbers you have sound very low, for a read of a row by key which should have been
the fastest.  I hope someone can help investigate or share numbers from their tests.
> 
>  
> 
> Regards,
> Roshni
>  
> 
> > From: Dean.Hiller@nrel.gov
> > To: user@cassandra.apache.org
> > Date: Tue, 2 Oct 2012 06:41:09 -0600
> > Subject: Re: Read latency issue
> > 
> > Interesting results. With PlayOrm, we did a 6 node test of reading 100 rows from
1,000,000 using PlayOrm Scalable SQL. It only took 60ms. Maybe we have better hardware though???
We are using 7200 RPM drives so nothing fancy on the disk side of things. More nodes puts
at a higher throughput though as reading from more disks will be faster. Anyways, you may
want to play with more nodes and re-run. If you run a test with PlayOrm, I would love to know
the results there as well.
> > 
> > Later,
> > Dean
> > 
> > From: Arindam Barua <abarua@247-inc.com<mailto:abarua@247-inc.com>>
> > Reply-To: 
> > "user@cassandra.apache.org<mailto:user@cassandra.apache.org>" 
> > <user@cassandra.apache.org<mailto:user@cassandra.apache.org>>
> > Date: Monday, October 1, 2012 4:57 PM
> > To: "user@cassandra.apache.org<mailto:user@cassandra.apache.org>" 
> > <user@cassandra.apache.org<mailto:user@cassandra.apache.org>>
> > Subject: Read latency issue
> > 
> > unning a query to like "select * from <table_name> where atag=<foo>",
where 'atag' is the first column of the composite key, from either JDBC or Hector (equivalent
code), results in read times of 200-300ms from a remote host on the same network. The query
returned around 800 results. Running the same query on a Cassandra host results in a read
time of ~110-130 ms.
> > Using read consistency of ONE reduces the read latency by ~20ms, compared to using
QUORUM.
> > 
> > Enabling row cache did not seem to change the performance much. Moreover, the row
cache 'size' according to nodetool was very tiny. Here is a snapshot of the nodetool info
after running few read tests:
> > Key Cache : size 2448 (bytes), capacity 104857584 (bytes), 231 hits,
> > 266 requests, 1.000 recent hit rate, 14400 save period in seconds 
> > Row Cache : size 96 (bytes), capacity 4194304000 (bytes), 9 hits, 13 
> > requests, NaN recent hit rate, 0 save period in seconds
> >






Mime
View raw message