cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ivan Ganza (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-4340) Cassandra upgrade to 1.1.1 resulted in slow query issue
Date Thu, 28 Jun 2012 17:46:56 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-4340?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13403285#comment-13403285
] 

Ivan Ganza commented on CASSANDRA-4340:
---------------------------------------

I was trying to duplicate our problem by fetching the same quote rapidly in succession and
got the following error and stock trace. Could you add it to our Jira ticket? I will keep
looking at this. Row caching was turned off.

java.lang.OutOfMemoryError: Java heap space
        at org.apache.cassandra.io.compress.CompressedRandomAccessReader.<init>(CompressedRandomAccessReader.java:66)
        at org.apache.cassandra.io.compress.CompressedRandomAccessReader.open(CompressedRandomAccessReader.java:46)
        at org.apache.cassandra.io.compress.CompressedRandomAccessReader.open(CompressedRandomAccessReader.java:41)
        at org.apache.cassandra.io.util.CompressedSegmentedFile.getSegment(CompressedSegmentedFile.java:63)
        at org.apache.cassandra.io.sstable.SSTableReader.getFileDataInput(SSTableReader.java:871)
        at org.apache.cassandra.db.columniterator.SSTableSliceIterator.<init>(SSTableSliceIterator.java:48)
        at org.apache.cassandra.db.filter.SliceQueryFilter.getSSTableColumnIterator(SliceQueryFilter.java:66)
        at org.apache.cassandra.db.filter.QueryFilter.getSSTableColumnIterator(QueryFilter.java:78)
        at org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:256)
        at org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:63)
        at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1321)
        at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1183)
        at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1118)
        at org.apache.cassandra.db.Table.getRow(Table.java:374)
        at org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:69)
        at org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:816)
        at org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1250)
        at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:886)
        at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:908)
        at java.lang.Thread.run(Thread.java:662)

                
> Cassandra upgrade to 1.1.1 resulted in slow query issue
> -------------------------------------------------------
>
>                 Key: CASSANDRA-4340
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-4340
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>    Affects Versions: 1.1.1
>         Environment: Ubuntu Linux, Java 7, Hector 1.0-1
>            Reporter: Ivan Ganza
>            Assignee: Pavel Yaskevich
>             Fix For: 1.1.2
>
>         Attachments: CassandraIssue.java
>
>
> We have recently introduced Cassandra at the Globe and Mail here in Toronto, Canada.
 We are processing and storing the North American stock-market feed.  We have found it to
work very quickly and things have been looking very good.
> Recently we upgraded to version 1.1.1 and then we have noticed some issues occurring.
> I will try to describe it for you here.  Basically one operation that we very often perform
and is very critical is the ability to 'get the latest quote'.  This would return to you the
latest Quote adjusted against exchange delay rules.  With Cassandra version 1.0.3 we could
get a Quote in around 2ms.  After update we are looking at time of at least 2-3 seconds.
> The way we query the quote is using a REVERSED SuperSliceQuery  with start=now, end=00:00:00.000
(beginning of day) LIMITED to 1.
> Our investigation leads us to suspect that, since upgrade, Cassandra seems to be reading
the sstable from disk even when we request a small range of day only 5 seconds back.  If you
look at the output below you can see that the query does NOT get slower as the lookback increases
from 5  sec, 60 sec, 15 min, 60 min, and 24 hours.
> We also noticed that the query was very fast for the first five minutes of trading, apparently
until the first sstable was flushed to disk.  After that we go into query times of 1-2 seconds
or so.
> Query time[lookback=5]:[1711ms]
> Query time[lookback=60]:[1592ms]
> Query time[lookback=900]:[1520ms]
> Query time[lookback=3600]:[1294ms]
> Query time[lookback=86400]:[1391ms]
> We would really appreciate input or help on this.
> Cassandra version: 1.1.1
> Hector version: 1.0-1
> ---
> public void testCassandraIssue() {
>             try {
>                   int[] seconds = new int[]{ 5, 60, 60 * 15, 60 * 60, 60 * 60 * 24};
>                   for(int sec : seconds) {
>                         DateTime start = new DateTime();
>                         SuperSliceQuery<String, String, String, String> superSliceQuery
= HFactory.createSuperSliceQuery(keyspaceOperator, StringSerializer.get(), StringSerializer.get(),
StringSerializer.get(), StringSerializer.get());
>                         superSliceQuery.setKey("101390" + "." + testFormatter.print(start));
>                         superSliceQuery.setColumnFamily("Quotes");
>                         superSliceQuery.setRange(superKeyFormatter.print(start),
>                                     superKeyFormatter.print(start.minusSeconds(sec)),
>                                     true,
>                                     1);
>                         long theStart = System.currentTimeMillis();
>                         QueryResult<SuperSlice<String, String, String>> result
= superSliceQuery.execute();
>                         long end = System.currentTimeMillis();
>                         System.out.println("Query time[lookback=" + sec + "]:[" + (end
- theStart) + "ms]");
>                   }
>             } catch(Exception e) {
>                   e.printStackTrace();
>                   fail(e.getMessage());
>             }
>       }
> ---
> create column family Quotes
>     with column_type = Super
>     and  comparator = BytesType
>     and subcomparator = BytesType
>     and keys_cached = 7000
>     and rows_cached = 0
>     and row_cache_save_period = 0
>     and key_cache_save_period = 3600
>     and memtable_throughput = 255
>     and memtable_operations = 0.29
>     AND compression_options={sstable_compression:SnappyCompressor, chunk_length_kb:64};

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message