cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sylvain Lebresne (JIRA)" <>
Subject [jira] [Commented] (CASSANDRA-9258) Range movement causes CPU & performance impact
Date Thu, 17 Dec 2015 10:16:46 GMT


Sylvain Lebresne commented on CASSANDRA-9258:

bq. Sylvain, are you happy with the provided data or do you insist on writing a jmh bench
for regression testing?

I think we still should write a jmh regression test for, well, regression testing, but also
because it'll give us a good baseline for future improvements. That said, the provided data
(and general reasoning) is convincing enough that this is a net improvement that if no-one
has time to write that jmh bench shortly, I won't hold back on committing since this is potentially
serious for some users. Still, let the record show that I dislike the idea of postponing the
write of such regression test because history so far shows that this is codename for "never
getting it done".

> Range movement causes CPU & performance impact
> ----------------------------------------------
>                 Key: CASSANDRA-9258
>                 URL:
>             Project: Cassandra
>          Issue Type: Bug
>         Environment: Cassandra 2.1.4
>            Reporter: Rick Branson
>            Assignee: Dikang Gu
>             Fix For: 2.1.x
>         Attachments: 0001-pending-ranges-map.patch, Screenshot 2015-12-16 16.11.36.png,
Screenshot 2015-12-16 16.11.51.png
> Observing big CPU & latency regressions when doing range movements on clusters with
many tens of thousands of vnodes. See CPU usage increase by ~80% when a single node is being
> Top methods are:
> 1) Ljava/math/BigInteger;.compareTo in Lorg/apache/cassandra/dht/ComparableObjectToken;.compareTo

> 2) Lcom/google/common/collect/AbstractMapBasedMultimap;.wrapCollection in Lcom/google/common/collect/AbstractMapBasedMultimap$AsMap$AsMapIterator;.next
> 3) Lorg/apache/cassandra/db/DecoratedKey;.compareTo in Lorg/apache/cassandra/dht/Range;.contains
> Here's a sample stack from a thread dump:
> {code}
> "Thrift:50673" daemon prio=10 tid=0x00007f2f20164800 nid=0x3a04af runnable [0x00007f2d878d0000]
>    java.lang.Thread.State: RUNNABLE
>       at org.apache.cassandra.dht.Range.isWrapAround(
>       at org.apache.cassandra.dht.Range.contains(
>       at org.apache.cassandra.dht.Range.contains(
>       at org.apache.cassandra.locator.TokenMetadata.pendingEndpointsFor(
>       at org.apache.cassandra.service.StorageProxy.performWrite(
>       at org.apache.cassandra.service.StorageProxy.mutate(
>       at org.apache.cassandra.service.StorageProxy.mutateWithTriggers(
>       at org.apache.cassandra.thrift.CassandraServer.doInsert(
>       at org.apache.cassandra.thrift.CassandraServer.doInsert(
>       at org.apache.cassandra.thrift.CassandraServer.batch_mutate(
>       at org.apache.cassandra.thrift.Cassandra$Processor$batch_mutate.getResult(
>       at org.apache.cassandra.thrift.Cassandra$Processor$batch_mutate.getResult(
>       at org.apache.thrift.ProcessFunction.process(
>       at org.apache.thrift.TBaseProcessor.process(
>       at org.apache.cassandra.thrift.CustomTThreadPoolServer$
>       at java.util.concurrent.ThreadPoolExecutor.runWorker(
>       at java.util.concurrent.ThreadPoolExecutor$
>       at{code}

This message was sent by Atlassian JIRA

View raw message