cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "DOAN DuyHai (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-11383) SASI index build leads to massive OOM
Date Sat, 19 Mar 2016 10:44:33 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-11383?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15202712#comment-15202712
] 

DOAN DuyHai commented on CASSANDRA-11383:
-----------------------------------------

bq. I've figured out what is going on and first of all period_end_month_int index is not sparse
- at least first term in that index has ~11M tokens assigned to it

 You're right, {{period_end_month_int}} is not *parse* in the sense we mean it in English
but SASI index mode {{SPARSE}} is the only one allowed for numeric fields, {{PREFIX}} and
{{CONTAINS}} are reserved to text fields. So we have a fundamental issue here, how to index
*dense* numeric values ?

bq. Temporary fix for this situation is switching to LCS with fixed maximum sstable size,
as I mentioned in my previous comment.

 Can you elaborate further ? What, in LCS, makes it work with current situation compared to
STCS ? Is it the total number of SSTables ? (currently with STCS there less than 100 SSTables
per node so it's not really a big issue) Is it the fact that a partition is guanrateed to
be in a single SSTable with LCS ? (again considering the schema we have mostly tiny rows but
a lot of them)

 For now I'm going to switch to LCS to see if we can finish building the index without OOM.
For long term,  LCS is not the solution because this table size will increase quickly over
time and having tombstones in level > L3 will make them rarely compacted


> SASI index build leads to massive OOM
> -------------------------------------
>
>                 Key: CASSANDRA-11383
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-11383
>             Project: Cassandra
>          Issue Type: Bug
>          Components: CQL
>         Environment: C* 3.4
>            Reporter: DOAN DuyHai
>         Attachments: CASSANDRA-11383.patch, new_system_log_CMS_8GB_OOM.log, system.log_sasi_build_oom
>
>
> 13 bare metal machines
> - 6 cores CPU (12 HT)
> - 64Gb RAM
> - 4 SSD in RAID0
>  JVM settings:
> - G1 GC
> - Xms32G, Xmx32G
> Data set:
>  - ≈ 100Gb/per node
>  - 1.3 Tb cluster-wide
>  - ≈ 20Gb for all SASI indices
> C* settings:
> - concurrent_compactors: 1
> - compaction_throughput_mb_per_sec: 256
> - memtable_heap_space_in_mb: 2048
> - memtable_offheap_space_in_mb: 2048
> I created 9 SASI indices
>  - 8 indices with text field, NonTokenizingAnalyser,  PREFIX mode, case-insensitive
>  - 1 index with numeric field, SPARSE mode
>  After a while, the nodes just gone OOM.
>  I attach log files. You can see a lot of GC happening while index segments are flush
to disk. At some point the node OOM ...
> /cc [~xedin]



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message