cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Stefania (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-10407) Benchmark and evaluate CASSANDRA-8894 improvements
Date Mon, 05 Oct 2015 08:50:27 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-10407?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14943083#comment-14943083
] 

Stefania commented on CASSANDRA-10407:
--------------------------------------

Thank you for the tests [~aboudreault]. It seems that when the patch was committed the performance
of std was really bad (44-45k ops/second) compared to mmap (66-68k ops/second) and neither
the patch nor readahead had any impact. However, this has changed quite radically with 3.0
rc1 (52k vs 57-59k). It's difficult to know why without bisecting but I think we should forget
what the code was like back then and focus on the current 3.0 head and aim to have std without
readahead as good as or better than mmap with readahead. We should also bring mmap back to
66-68k ops/second. 

I did some profiling by running mmap vs std locally and I've attached a couple of _.nps_ files
that can be opened with [visual vm|http://visualvm.java.net/download.html]. I've also attached
some screenshots.

Here are some things that I've noticed:

* In RAR.overrideLength() the call to channel.size() is expensive, we should cache the size
in the channel or avoid calling this (it is just to throw an IllegalArgumentException)

* Most of the time we spend in {{reBuffer}} follows a call from {{SSTableReader.getFileDataInput()}},
which is called when initializing iterators. Here a new reader is created so we cannot avoid
rebuffering, also we spend a lot of time in dealing with checksums, more than we spend reading.

* CRAR is allocating buffers that are too big for the buffer pool (> 64kb) and therefore
we spend time calling {{allocateDirect()}}: {{INFO  07:57:16 Requested buffer size 65813 is
bigger than 65536, allocating directly}}

We should really address these points first (the first one is trivial but the other two I
am not as familiar with). However, we could also in parallel try running these tests to see
if there are any differences with the STD case:

* 3.0 HEAD MMAP with readahead
* 3.0 HEAD STD without readahead and setting this in the yaml: {{disk_optimization_page_cross_chance:
0.1}} - this is actually the current default
* 3.0 HEAD STD without readahead and setting this in the yaml: {{disk_optimization_page_cross_chance:
0.5}}
* 3.0 HEAD STD without readahead and setting this in the yaml: {{disk_optimization_page_cross_chance:
0.9}}

[~benedict] WDYT regarding the points I've raised? Is it worth addressing them before running
more tests or would you expect a big difference in the tests anyway? Also, is there anything
that needs to be done to disable readahead, other than calling {{blockdev --setra}}?


> Benchmark and evaluate CASSANDRA-8894 improvements
> --------------------------------------------------
>
>                 Key: CASSANDRA-10407
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-10407
>             Project: Cassandra
>          Issue Type: Test
>            Reporter: Aleksey Yeschenko
>            Assignee: Alan Boudreault
>             Fix For: 3.0.0 rc2
>
>         Attachments: 3_0_head_mmap_wo_ra.nps, 3_0_head_std_wo_ra.nps, 8894_tiny.yaml,
allocateDirect.png, flight-recordings.tar.gz, reBufferStandardTime.png, size.png, test-with-8894-tiny.json,
test-without-8894-tiny.json
>
>
> The original ticket (CASSANDRA-8894) was committed to 3.0 alpha1 two months ago. We need
to get proper performance tests before GA.
> See [~benedict]'s [comment|https://issues.apache.org/jira/browse/CASSANDRA-8894?focusedCommentId=14631203&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-14631203]
for more details.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message