cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ariel Weisberg (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (CASSANDRA-8457) nio MessagingService
Date Wed, 07 Jan 2015 21:11:35 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-8457?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14268221#comment-14268221
] 

Ariel Weisberg edited comment on CASSANDRA-8457 at 1/7/15 9:11 PM:
-------------------------------------------------------------------

In GCE coalescing provided a 12% increase in throughput in this specific message heavy high
concurrency workload. The penalty is that at low concurrency there is an immediate loss of
performance with any coalescing and a large window has a greater impact at low concurrency
so there is tension there. The larger the window the better the performance bump.

Right scale does not offer instances with enhanced networking. To find out whether coalescing
provides real benefits in EC2/Xen or milder GCE like benefits I will have to get my hands
on some.

I wanted to account for the impact of coalescing at low concurrency. Low concurrency is not
a recipe for great performance, but it is part of the out of the box experience and people
do compare different databases at low concurrency.

Testing with 3 client threads each running on a dedicated client instance (3 threads total).
This is in GCE.

With TCP no delay on and coalescing
||Coalesce window microseconds|Throughput||
|0| 2191|
|6| 1910|
|12| 1873|
|25| 1867|
|50| 1779|
|100| 1667|
|150| 1566|
|200| 1491|

I also tried disabling coalescing when using HSHA and it didn't seem to make a difference.
Surprising considering the impact of 25 microseconds of coalescing intra-cluster.

I also experimented with some other things. Binding interrupts cores 0 and 8 and task setting
C* off of those cores. I didn't see a big impact.

I did see a small positive impact using 3 clients 8 servers which means the measurements with
2 clients might be a little suspect. With 3 clients and 200 microseconds of coalescing it
peaked at 165k in GCE.

I also found out that banned CPUs in irqbalance is broken and has no effect and this has been
the case for some time.



was (Author: aweisberg):
I wanted to account for the impact of coalescing at low concurrency. Low concurrency is not
a recipe for great performance, but it is part of the out of the box experience and people
do compare different databases at low concurrency.

In GCE coalescing provided a 12% increase in throughput in this specific message heavy high
concurrency workload. The penalty is that at low concurrency there is an immediate loss of
performance with any coalescing and a large window has a greater impact at low concurrency
so there is tension there. The larger the window the better the performance bump.

Testing with 3 client threads each running on a dedicated client instance (3 threads total).
This is in GCE.

With TCP no delay on and coalescing
||Coalesce window microseconds|Throughput||
|0| 2191|
|6| 1910|
|12| 1873|
|25| 1867|
|50| 1779|
|100| 1667|
|150| 1566|
|200| 1491|

I also tried disabling coalescing when using HSHA and it didn't seem to make a difference.
Surprising considering the impact of 25 microseconds of coalescing intra-cluster.

I also experimented with some other things. Binding interrupts cores 0 and 8 and task setting
C* off of those cores. I didn't see a big impact.

I did see a small positive impact using 3 clients 8 servers which means the measurements with
2 clients might be a little suspect. With 3 clients and 200 microseconds of coalescing it
peaked at 165k in GCE.

I also found out that banned CPUs in irqbalance is broken and has no effect and this has been
the case for some time.

Right scale does not offer instances with enhanced networking. To find out whether coalescing
provides real benefits in EC2/Xen or milder GCE like benefits I will have to get my hands
on some.


> nio MessagingService
> --------------------
>
>                 Key: CASSANDRA-8457
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-8457
>             Project: Cassandra
>          Issue Type: New Feature
>          Components: Core
>            Reporter: Jonathan Ellis
>            Assignee: Ariel Weisberg
>              Labels: performance
>             Fix For: 3.0
>
>
> Thread-per-peer (actually two each incoming and outbound) is a big contributor to context
switching, especially for larger clusters.  Let's look at switching to nio, possibly via Netty.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message