cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jacek Furmankiewicz (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-7088) Massive performance degradation for TRUNCATE when migrating to 2.0
Date Thu, 24 Apr 2014 22:11:18 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-7088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13980384#comment-13980384
] 

Jacek Furmankiewicz commented on CASSANDRA-7088:
------------------------------------------------

We also see a lot of exceptions like this at a time when our test suite seems to freeze waiting
for Cassandra:

k 16ms for 3394 cells
ERROR [ReadStage:82] 2014-04-24 17:09:31,843 CassandraDaemon.java (line 198) Exception in
thread Thread[ReadStage:82,5,main]
java.lang.RuntimeException: java.lang.NumberFormatException: Zero length BigInteger
	at org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1920)
	at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
	at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
	at java.lang.Thread.run(Thread.java:744)
Caused by: java.lang.NumberFormatException: Zero length BigInteger
	at java.math.BigInteger.<init>(BigInteger.java:190)
	at org.apache.cassandra.serializers.IntegerSerializer.deserialize(IntegerSerializer.java:32)
	at org.apache.cassandra.serializers.IntegerSerializer.deserialize(IntegerSerializer.java:26)
	at org.apache.cassandra.db.marshal.AbstractType.getString(AbstractType.java:156)
	at org.apache.cassandra.db.marshal.AbstractCompositeType.getString(AbstractCompositeType.java:242)
	at org.apache.cassandra.db.filter.SliceQueryFilter.collectReducedColumns(SliceQueryFilter.java:219)
	at org.apache.cassandra.db.filter.QueryFilter.collateColumns(QueryFilter.java:122)
	at org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:80)
	at org.apache.cassandra.db.filter.QueryFilter.collateOnDiskAtom(QueryFilter.java:72)
	at org.apache.cassandra.db.CollationController.collectAllData(CollationController.java:297)
	at org.apache.cassandra.db.CollationController.getTopLevelColumns(CollationController.java:53)
	at org.apache.cassandra.db.ColumnFamilyStore.getTopLevelColumns(ColumnFamilyStore.java:1540)
	at org.apache.cassandra.db.ColumnFamilyStore.getColumnFamily(ColumnFamilyStore.java:1369)
	at org.apache.cassandra.db.Keyspace.getRow(Keyspace.java:327)
	at org.apache.cassandra.db.SliceFromReadCommand.getRow(SliceFromReadCommand.java:65)
	at org.apache.cassandra.service.StorageProxy$LocalReadRunnable.runMayThrow(StorageProxy.java:1352)
	at org.apache.cassandra.service.StorageProxy$DroppableRunnable.run(StorageProxy.java:1916)
	... 3 more


> Massive performance degradation for TRUNCATE when migrating to 2.0
> ------------------------------------------------------------------
>
>                 Key: CASSANDRA-7088
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-7088
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Core
>         Environment: Linux Mint 16
>            Reporter: Jacek Furmankiewicz
>         Attachments: cassandra.yaml
>
>
> We attempted to migrate our developers to Cassandra 2.0.7 from 1.2.
> Everything worked perfectly, but we have experienced a massive drop in developer velocity.
> We run integration tests with Cucumber BDD and 1000 BDDs went from 7 minutes (Cassandra
1.2) to 15 minutes (2.0.7),
> This is when we run Cassandra of the ramdisk (/dev/shm) to make it run faster on dev
boxes.
> When we tried pointed to actual drives  the difference was dramatic: the entire suite
took over 70 minutes (!) vs 15 in Cassandra 1.2.
> After investigation, we found that most of the time is spent in the truncation logic
between every scenario, where we truncate all the column families and start with a clean DB
for the next test case.
> This used to be super fast in 1.2, is now very slow in 2.0.
> It may not seem important, but upgrading to 2.0 has basically cut down developer velocity
by 100%, just by more than doubling the time it takes to run our BDD suite.
> We truncate the CFs using the Ruby driver:
>   $cassandra.column_families.each do |column_family|
>         name = column_family[0].to_s
>         $cassandra.truncate! name
>   end
> I am attaching our cassandra.yaml. Please note we already switched off auto_compaction
before truncate, just as we did in 1.2 for dev boxes, Made no difference.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message