cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Michael Kjellman (JIRA)" <>
Subject [jira] [Commented] (CASSANDRA-8789) OutboundTcpConnectionPool should route messages to sockets by size not type
Date Sat, 25 Apr 2015 01:19:39 GMT


Michael Kjellman commented on CASSANDRA-8789:

I tried to cleanly revert the following commits to demonstrate  that stress functions "as
expected" without the changes from CASSANDRA-8789 but I got into conflict hell.


I tried to checkout 21bdf8700601f8150e8c13e0b4f71e061822c802, however the build is broken
in that commit and it was reverted by jbellis in b25adc765769869d16410f1ca156227745d9b17b.
I tried to next checkout 21bdf8700601f8150e8c13e0b4f71e061822c802-1 (1279009e0e29267d8fc3300071034e2ede6065ca)
which I could build and unlike a few other commits I tried there were no exceptions logged
while inserting data. In this commit though I do see issues with stress around 300k rows even
with the OutboundTcpConnection changes backed out.

I next checked out 8896a70b015102c212d0a27ed1f4e1f0fabe85c4 which is the previous commit to
828496492c51d7437b690999205ecc941f41a0a9 for OutboundTcpConnection. Testing against that commit
I'm able to insert all rows as expected and Gossiper does not down any nodes during the duration
of stress.

This commit however was logging intermittant NPEs (however otherwise load after stress looks

ERROR [CompactionExecutor:2] 2015-04-24 17:48:20,741 - Exception
in thread Thread[CompactionExecutor:2,1,main]
java.lang.NullPointerException: null

h4. Strees Output
Michaels-MacBook-Pro:cassandra-aml mkjellman$ tools/bin/cassandra-stress -l 3

Averages from the middle 80% of values:
interval_op_rate          : 11865
interval_key_rate         : 11865
latency median            : 2.0
latency 95th percentile   : 17.7
latency 99.9th percentile : 1495.2
Total operation time      : 00:01:35

h4. nodetool ring output
Datacenter: datacenter1
Address    Rack        Status State   Load            Owns                Token
                                                                          3074457345618258602  rack1       Up     Normal  294.67 MB       ?                   -9223372036854775808  rack1       Up     Normal  246.91 MB       ?                   -3074457345618258603  rack1       Up     Normal  247.19 MB       ?                   3074457345618258602

So -- I would agree that until the source of the regression (unrelated to this ticket) that
is causing stress to fail even without the changes to OutboundTcpConnection reverted, we can't
move forward evaluating the merits of changes to the actual TCP socket handling affecting
the timely delivery of GOSSIP messages/verbs.

> OutboundTcpConnectionPool should route messages to sockets by size not type
> ---------------------------------------------------------------------------
>                 Key: CASSANDRA-8789
>                 URL:
>             Project: Cassandra
>          Issue Type: Improvement
>          Components: Core
>            Reporter: Ariel Weisberg
>            Assignee: Ariel Weisberg
>             Fix For: 3.0
>         Attachments: 8789.diff
> I was looking at this trying to understand what messages flow over which connection.
> For reads the request goes out over the command connection and the response comes back
over the ack connection.
> For writes the request goes out over the command connection and the response comes back
over the command connection.
> Reads get a dedicated socket for responses. Mutation commands and responses both travel
over the same socket along with read requests.
> Sockets are used uni-directional so there are actually four sockets in play and four
threads at each node (2 inbounded, 2 outbound).
> CASSANDRA-488 doesn't leave a record of what the impact of this change was. If someone
remembers what situations were made better it would be good to know.
> I am not clear on when/how this is helpful. The consumer side shouldn't be blocking so
the only head of line blocking issue is the time it takes to transfer data over the wire.
> If message size is the cause of blocking issues then the current design mixes small messages
and large messages on the same connection retaining the head of line blocking.
> Read requests share the same connection as write requests (which are large), and write
acknowledgments (which are small) share the same connections as write requests. The only winner
is read acknowledgements.

This message was sent by Atlassian JIRA

View raw message