cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Michael Kjellman (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-9754) Make index info heap friendly for large CQL partitions
Date Fri, 14 Oct 2016 21:13:20 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-9754?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15576497#comment-15576497
] 

Michael Kjellman commented on CASSANDRA-9754:
---------------------------------------------

All of the threads that were responsible for generating load in the control cluster for the
two large partition read and write workloads had died because the cluster became so unstable.
As soon as I just restarted the stress load 60% of the instances in the cluster OOMed within
2 minutes of restarting the load. 

At this point I don't think I can drive any more data into the partitions with the old code
and I'm going to declare defeat and say that 17GB as the absolute max partition size possible
with the old/previous/current index implementation (given the JVM parameters as I detailed
in the test description above).

I'm going to leave the load at the current read and write rates in the two Birch clusters
until things explode to see the theoretical max partition size possible with the Birch implementation
today. After that I'll wipe the clusters and restart the same load at 2x the read and write
rates to see how things go with that configuration.

> Make index info heap friendly for large CQL partitions
> ------------------------------------------------------
>
>                 Key: CASSANDRA-9754
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-9754
>             Project: Cassandra
>          Issue Type: Improvement
>            Reporter: sankalp kohli
>            Assignee: Michael Kjellman
>            Priority: Minor
>             Fix For: 4.x
>
>         Attachments: gc_collection_times_with_birch.png, gc_collection_times_without_birch.png,
gc_counts_with_birch.png, gc_counts_without_birch.png, perf_cluster_1_with_birch_read_latency_and_counts.png,
perf_cluster_1_with_birch_write_latency_and_counts.png, perf_cluster_2_with_birch_read_latency_and_counts.png,
perf_cluster_2_with_birch_write_latency_and_counts.png, perf_cluster_3_without_birch_read_latency_and_counts.png,
perf_cluster_3_without_birch_write_latency_and_counts.png
>
>
>  Looking at a heap dump of 2.0 cluster, I found that majority of the objects are IndexInfo
and its ByteBuffers. This is specially bad in endpoints with large CQL partitions. If a CQL
partition is say 6,4GB, it will have 100K IndexInfo objects and 200K ByteBuffers. This will
create a lot of churn for GC. Can this be improved by not creating so many objects?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message