cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Michael Kjellman (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-8789) OutboundTcpConnectionPool should route messages to sockets by size not type
Date Sat, 25 Apr 2015 01:19:39 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-8789?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14512147#comment-14512147
] 

Michael Kjellman commented on CASSANDRA-8789:
---------------------------------------------

I tried to cleanly revert the following commits to demonstrate  that stress functions "as
expected" without the changes from CASSANDRA-8789 but I got into conflict hell.

{noformat}
ebd0ae820a3fc7c13d58b6ddb48ba4d26b3fcd65
144644bbf77a546c45db384e2dbc18e13f65c9ce
1caa4f942662cd49609e86e2cd747421a9d71700
16499ca9b0080ea4d3c4ed3bc55c753bacc3c24e
828496492c51d7437b690999205ecc941f41a0a9
{noformat}

I tried to checkout 21bdf8700601f8150e8c13e0b4f71e061822c802, however the build is broken
in that commit and it was reverted by jbellis in b25adc765769869d16410f1ca156227745d9b17b.
I tried to next checkout 21bdf8700601f8150e8c13e0b4f71e061822c802-1 (1279009e0e29267d8fc3300071034e2ede6065ca)
which I could build and unlike a few other commits I tried there were no exceptions logged
while inserting data. In this commit though I do see issues with stress around 300k rows even
with the OutboundTcpConnection changes backed out.

I next checked out 8896a70b015102c212d0a27ed1f4e1f0fabe85c4 which is the previous commit to
828496492c51d7437b690999205ecc941f41a0a9 for OutboundTcpConnection. Testing against that commit
I'm able to insert all rows as expected and Gossiper does not down any nodes during the duration
of stress.

This commit however was logging intermittant NPEs (however otherwise load after stress looks
sane...)

{noformat}
ERROR [CompactionExecutor:2] 2015-04-24 17:48:20,741 CassandraDaemon.java:182 - Exception
in thread Thread[CompactionExecutor:2,1,main]
java.lang.NullPointerException: null
        at org.apache.cassandra.io.sstable.format.SSTableReader$Tidier.tidy(SSTableReader.java:1798)
~[main/:na]
{noformat}

h4. Strees Output
{noformat}
Michaels-MacBook-Pro:cassandra-aml mkjellman$ tools/bin/cassandra-stress -l 3
null
total,interval_op_rate,interval_key_rate,latency,95th,99.9th,elapsed_time
31769,3176,3176,1.3,100.7,562.8,10
84820,5305,5305,1.1,76.7,452.8,20
152130,6731,6731,2.1,50.0,2137.6,30
267053,11492,11492,2.4,19.2,2137.6,40
346529,7947,7947,2.4,14.9,2159.1,51
455677,10914,10914,2.3,9.6,2131.6,61
619967,16429,16429,1.8,7.2,203.1,71
796739,17677,17677,1.3,5.3,202.4,81
967800,17106,17106,0.9,5.1,202.2,91
1000000,3220,3220,0.9,4.8,202.2,95


Averages from the middle 80% of values:
interval_op_rate          : 11865
interval_key_rate         : 11865
latency median            : 2.0
latency 95th percentile   : 17.7
latency 99.9th percentile : 1495.2
Total operation time      : 00:01:35
END
{noformat}

h4. nodetool ring output
{noformat}
Datacenter: datacenter1
==========
Address    Rack        Status State   Load            Owns                Token
                                                                          3074457345618258602
127.0.0.1  rack1       Up     Normal  294.67 MB       ?                   -9223372036854775808
127.0.0.2  rack1       Up     Normal  246.91 MB       ?                   -3074457345618258603
127.0.0.3  rack1       Up     Normal  247.19 MB       ?                   3074457345618258602
{noformat}

So -- I would agree that until the source of the regression (unrelated to this ticket) that
is causing stress to fail even without the changes to OutboundTcpConnection reverted, we can't
move forward evaluating the merits of changes to the actual TCP socket handling affecting
the timely delivery of GOSSIP messages/verbs.

> OutboundTcpConnectionPool should route messages to sockets by size not type
> ---------------------------------------------------------------------------
>
>                 Key: CASSANDRA-8789
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-8789
>             Project: Cassandra
>          Issue Type: Improvement
>          Components: Core
>            Reporter: Ariel Weisberg
>            Assignee: Ariel Weisberg
>             Fix For: 3.0
>
>         Attachments: 8789.diff
>
>
> I was looking at this trying to understand what messages flow over which connection.
> For reads the request goes out over the command connection and the response comes back
over the ack connection.
> For writes the request goes out over the command connection and the response comes back
over the command connection.
> Reads get a dedicated socket for responses. Mutation commands and responses both travel
over the same socket along with read requests.
> Sockets are used uni-directional so there are actually four sockets in play and four
threads at each node (2 inbounded, 2 outbound).
> CASSANDRA-488 doesn't leave a record of what the impact of this change was. If someone
remembers what situations were made better it would be good to know.
> I am not clear on when/how this is helpful. The consumer side shouldn't be blocking so
the only head of line blocking issue is the time it takes to transfer data over the wire.
> If message size is the cause of blocking issues then the current design mixes small messages
and large messages on the same connection retaining the head of line blocking.
> Read requests share the same connection as write requests (which are large), and write
acknowledgments (which are small) share the same connections as write requests. The only winner
is read acknowledgements.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message