cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ariel Weisberg (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-7217) Native transport performance (with cassandra-stress) drops precipitously past around 1000 threads
Date Mon, 16 Nov 2015 20:29:11 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-7217?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15007258#comment-15007258
] 

Ariel Weisberg commented on CASSANDRA-7217:
-------------------------------------------

I was able to narrow this down to a configuration issue with the driver combined with less
than perfect behavior if you don't run with this configuration. If I increase the maximum
number of pending requests per connection from 128 to 256 then the performance at 1250 threads
goes back to normal.

For stress we can do something smarter when setting this tunable to reflect the number of
available threads. Generally if we have a thread submitting requests we would want it to default
to having a pending request against the server otherwise all you are really benchmarking is
the driver's ability to deal with pending requests.

Then there is separate driver issue of the degradation in performance when the number of pending
requests is not high enough. I wouldn't expect that kind of drop off. Whether the request
is pending at the client or languishing in a TCP buffer in the server shouldn't really matter.
I haven't looked, but my guess is that when the driver reaches the limit the thread submitting
a requests goes to sleep, and then it is woken up again. This means that every request has
to flow through some extra scheduling points per request to account for this.

A better way is to always flatten the serialized request to a shared buffer and when the connection
is ready to accept more work the network thread can wake up and write multiple requests to
the server at once.

> Native transport performance (with cassandra-stress) drops precipitously past around
1000 threads
> -------------------------------------------------------------------------------------------------
>
>                 Key: CASSANDRA-7217
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-7217
>             Project: Cassandra
>          Issue Type: Bug
>            Reporter: Benedict
>            Assignee: Ariel Weisberg
>              Labels: performance, stress, triaged
>             Fix For: 3.0.1, 3.1
>
>         Attachments: 2000-threads.svg, 500-threads.svg, FakeQuerySystem.java, stub_server.diff
>
>
> This is obviously bad. Let's figure out why it's happening and put a stop to it.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message