cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Pavel Yaskevich (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-11383) Avoid index segment stitching in RAM which lead to OOM on big SSTable files
Date Sat, 26 Mar 2016 20:06:25 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-11383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15213180#comment-15213180
] 

Pavel Yaskevich commented on CASSANDRA-11383:
---------------------------------------------

[~doanduyhai] 

bq. 2016-03-26 16:36:27,281 CassandraDaemon.java:195 - Exception in thread Thread[SASI-General:2,5,main]

This is an oversight on my part, I've fixed that and pushed it rebased with latest 3.5 to
my branch and kicked-off the build. I will let everything stay in the branch until Sunday
evening PST if you want to re-run your tests. 

bq. It is working as designed I guess, but my question is does the '5 keys' a hard-coded threshold
or is it configurable ?

No, it's a max optimal number of keys per term since we know how many terms in the worst case
fits into one index block and who big combined index is allowed to be. Ideally it should be
1-1 but we still allow 1-5 max since some of the timestamps even can overlap sometimes.


> Avoid index segment stitching in RAM which lead to OOM on big SSTable files 
> ----------------------------------------------------------------------------
>
>                 Key: CASSANDRA-11383
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-11383
>             Project: Cassandra
>          Issue Type: Bug
>          Components: CQL
>         Environment: C* 3.4
>            Reporter: DOAN DuyHai
>            Assignee: Jordan West
>              Labels: sasi
>             Fix For: 3.5
>
>         Attachments: CASSANDRA-11383.patch, SASI_Index_build_LCS_1G_Max_SSTable_Size_logs.tar.gz,
new_system_log_CMS_8GB_OOM.log, system.log_sasi_build_oom
>
>
> 13 bare metal machines
> - 6 cores CPU (12 HT)
> - 64Gb RAM
> - 4 SSD in RAID0
>  JVM settings:
> - G1 GC
> - Xms32G, Xmx32G
> Data set:
>  - ≈ 100Gb/per node
>  - 1.3 Tb cluster-wide
>  - ≈ 20Gb for all SASI indices
> C* settings:
> - concurrent_compactors: 1
> - compaction_throughput_mb_per_sec: 256
> - memtable_heap_space_in_mb: 2048
> - memtable_offheap_space_in_mb: 2048
> I created 9 SASI indices
>  - 8 indices with text field, NonTokenizingAnalyser,  PREFIX mode, case-insensitive
>  - 1 index with numeric field, SPARSE mode
>  After a while, the nodes just gone OOM.
>  I attach log files. You can see a lot of GC happening while index segments are flush
to disk. At some point the node OOM ...
> /cc [~xedin]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message