cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Anishek Agarwal <anis...@gmail.com>
Subject Re: High Bloom filter false ratio
Date Fri, 19 Feb 2016 03:52:53 GMT
Hey all,

@Jaydeep here is the cfstats output from one node.

Read Count: 1721134722

Read Latency: 0.04268825050756254 ms.

Write Count: 56743880

Write Latency: 0.014650376727851532 ms.

Pending Tasks: 0

Table: user_stay_points

SSTable count: 1289

Space used (live), bytes: 122141272262

Space used (total), bytes: 224227850870

Off heap memory used (total), bytes: 653827528

SSTable Compression Ratio: 0.4959736121441446

Number of keys (estimate): 345137664

Memtable cell count: 339034

Memtable data size, bytes: 106558314

Memtable switch count: 3266

Local read count: 1721134803

Local read latency: 0.048 ms

Local write count: 56743898

Local write latency: 0.018 ms

Pending tasks: 0

Bloom filter false positives: 40664437

Bloom filter false ratio: 0.69058

Bloom filter space used, bytes: 493777336

Bloom filter off heap memory used, bytes: 493767024

Index summary off heap memory used, bytes: 91677192

Compression metadata off heap memory used, bytes: 68383312

Compacted partition minimum bytes: 104

Compacted partition maximum bytes: 1629722

Compacted partition mean bytes: 1773

Average live cells per slice (last five minutes): 0.0

Average tombstones per slice (last five minutes): 0.0


@Tyler Hobbs

we are using cassandra 2.0.15 so
https://issues.apache.org/jira/browse/CASSANDRA-8525  shouldnt occur. Other
problems looks like will be fixed in 3.0 .. we will mostly try and slot in
an upgrade to 3.x version towards second quarter of this year.


@Daemon

Latencies seem to have higher ratios, attached is the graph.


I am mostly trying to look at Bloom filters, because the way we do reads,
we read data with non existent partition keys and it seems to be taking
long to respond, like for 720 queries it takes 2 seconds, with all 721
queries not returning anything. the 720 queries are done in sequence of 180
queries each with 180 of them running in parallel.


thanks

anishek



On Fri, Feb 19, 2016 at 3:09 AM, Jaydeep Chovatia <
chovatia.jaydeep@gmail.com> wrote:

> How many partition keys exists for the table which shows this problem (or
> provide nodetool cfstats for that table)?
>
> On Thu, Feb 18, 2016 at 11:38 AM, daemeon reiydelle <daemeonr@gmail.com>
> wrote:
>
>> The bloom filter buckets the values in a small number of buckets. I have
>> been surprised by how many cases I see with large cardinality where a few
>> values populate a given bloom leaf, resulting in high false positives, and
>> a surprising impact on latencies!
>>
>> Are you seeing 2:1 ranges between mean and worse case latencies (allowing
>> for gc times)?
>>
>> Daemeon Reiydelle
>> On Feb 18, 2016 8:57 AM, "Tyler Hobbs" <tyler@datastax.com> wrote:
>>
>>> You can try slightly lowering the bloom_filter_fp_chance on your table.
>>>
>>> Otherwise, it's possible that you're repeatedly querying one or two
>>> partitions that always trigger a bloom filter false positive.  You could
>>> try manually tracing a few queries on this table (for non-existent
>>> partitions) to see if the bloom filter rejects them.
>>>
>>> Depending on your Cassandra version, your false positive ratio could be
>>> inaccurate: https://issues.apache.org/jira/browse/CASSANDRA-8525
>>>
>>> There are also a couple of recent improvements to bloom filters:
>>> * https://issues.apache.org/jira/browse/CASSANDRA-8413
>>> * https://issues.apache.org/jira/browse/CASSANDRA-9167
>>>
>>>
>>> On Thu, Feb 18, 2016 at 1:35 AM, Anishek Agarwal <anishek@gmail.com>
>>> wrote:
>>>
>>>> Hello,
>>>>
>>>> We have a table with composite partition key with humungous
>>>> cardinality, its a combination of (long,long). On the table we have
>>>> bloom_filter_fp_chance=0.010000.
>>>>
>>>> On doing "nodetool cfstats" on the 5 nodes we have in the cluster we
>>>> are seeing  "Bloom filter false ratio:" in the range of 0.7 -0.9.
>>>>
>>>> I thought over time the bloom filter would adjust to the key space
>>>> cardinality, we have been running the cluster for a long time now but have
>>>> added significant traffic from Jan this year, which would not lead to
>>>> writes in the db but would lead to high reads to see if are any values.
>>>>
>>>> Are there any settings that can be changed to allow better ratio.
>>>>
>>>> Thanks
>>>> Anishek
>>>>
>>>
>>>
>>>
>>> --
>>> Tyler Hobbs
>>> DataStax <http://datastax.com/>
>>>
>>
>

Mime
View raw message