cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "T Jake Luciani (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (CASSANDRA-8692) Coalesce intra-cluster network messages
Date Fri, 06 Mar 2015 15:34:40 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-8692?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14350474#comment-14350474
] 

T Jake Luciani commented on CASSANDRA-8692:
-------------------------------------------

bq. Are you testing with or without vnodes?

That was with vnodes. Actually without vnodes (on stock 2.1.3) I'm seeing almost a 2x improvement
in throughput and latency on this 99 node cluster.

This is pretty fundemental to vnodes as [~aweisberg] mentions.  Without vnodes we are always
sending replica writes to the same N nodes and with vnodes we split them up across all nodes
in the cluster. The OS is naturally coalescing these packets as you can see by the rAvs and
wAvs columns below.

It seems like there is a much bigger issue here related to vnodes we need to solve. perhaps
a more controlled/coordinated allocation of vnodes so we get behavior more like the no vnode
case (which we naturally get massive improvement in packet size).  


Here are the network stats with vnodes:

{code}
  Time      Int   rMbps   wMbps   rPk/s   wPk/s    rAvs    wAvs %rUtil %wUtil
15:08:23     eth0   73.66   55.39 45115.6 44654.8   214.0   162.6   7.72   5.81
15:08:24     eth0   78.23   71.56 53299.8 52009.3   192.4   180.3   8.20   7.50
15:08:25     eth0   78.27   65.26 54357.8 53115.0   188.7   161.0   8.21   6.84
15:08:26     eth0   92.81   78.24 63269.4 61833.7   192.3   165.8   9.73   8.20
15:08:27     eth0   87.64   73.58 59189.0 57346.1   194.1   168.2   9.19   7.72
15:08:28     eth0   98.08   82.57 65408.3 63701.3   196.5   169.9   10.3   8.66
15:08:29     eth0   99.20   82.75 61070.9 60120.2   212.9   180.4   10.4   8.68
15:08:30     eth0   87.33   72.21 55217.8 53205.2   207.3   177.9   9.16   7.57
15:08:31     eth0   100.2   82.88 62752.4 60735.0   209.3   178.9   10.5   8.69
15:08:32     eth0   93.25   76.35 60180.5 58639.9   203.1   170.7   9.78   8.01
15:08:33     eth0   89.56   77.24 57428.4 56417.9   204.4   179.4   9.39   8.10
15:08:34     eth0   94.51   79.95 63685.5 63274.3   194.5   165.6   9.91   8.38
15:08:35     eth0   93.94   78.01 60504.5 59120.5   203.5   173.0   9.85   8.18
15:08:36     eth0   95.42   77.78 62392.6 60861.0   200.5   167.5   10.0   8.16
15:08:37     eth0   81.13   70.63 57439.5 56605.9   185.1   163.5   8.51   7.41
15:08:38     eth0   83.14   70.22 58104.0 56434.0   187.6   163.1   8.72   7.36
15:08:39     eth0   92.02   76.24 58900.5 55923.6   204.8   178.7   9.65   7.99
15:08:40     eth0   102.5   86.29 66515.7 66032.8   202.0   171.3   10.8   9.05
15:08:41     eth0   92.04   70.23 58642.6 56185.1   205.7   163.8   9.65   7.36
15:08:42     eth0   97.59   87.51 64721.8 64347.4   197.6   178.3   10.2   9.18
{code}

And without vnodes:
{code}
    Time      Int   rMbps   wMbps   rPk/s   wPk/s    rAvs    wAvs %rUtil %wUtil
14:46:24     eth0   133.6   111.2 35240.4 33946.6   496.8   429.3   14.0   11.7
14:46:25     eth0   113.4   91.75 30029.4 29208.0   495.1   411.7   11.9   9.62
14:46:26     eth0   123.4   96.78 33802.7 32350.2   478.4   392.1   12.9   10.1
14:46:27     eth0   120.4   95.19 32919.5 31669.6   479.4   394.0   12.6   9.98
14:46:28     eth0   119.2   89.46 32969.8 30548.0   474.0   383.8   12.5   9.38
14:46:29     eth0   149.7   123.2 34880.8 33658.9   562.4   479.9   15.7   12.9
14:46:30     eth0   133.2   106.0 30253.4 28806.6   576.9   482.5   14.0   11.1
14:46:31     eth0   124.6   94.99 30638.8 27731.3   532.8   448.9   13.1   9.96
14:46:32     eth0   128.7   104.0 30356.9 29852.5   555.7   456.7   13.5   10.9
14:46:33     eth0   127.8   101.2 31735.1 30715.3   527.7   432.0   13.4   10.6
14:46:34     eth0   133.4   107.2 32772.6 31283.3   533.5   449.1   14.0   11.2
14:46:35     eth0   112.2   88.93 35378.2 34920.8   415.7   333.8   11.8   9.33
14:46:36     eth0   128.4   103.3 36949.9 37194.1   455.5   363.9   13.5   10.8
14:46:37     eth0   104.2   82.93 30794.0 29068.2   443.7   374.0   10.9   8.70
14:46:38     eth0   103.8   78.08 30582.8 29560.7   445.0   346.2   10.9   8.19
14:46:39     eth0   136.1   112.1 29640.9 28823.1   601.6   509.7   14.3   11.8
14:46:40     eth0   138.0   110.5 32089.0 31454.5   563.6   460.6   14.5   11.6
14:46:41     eth0   129.5   103.9 32848.7 32135.9   516.6   423.9   13.6   10.9
14:46:42     eth0   110.0   85.14 33232.1 31965.6   433.7   349.1   11.5   8.93
14:46:43     eth0   134.8   109.3 33557.0 33590.9   526.7   426.5   14.1   11.5
{code}

> Coalesce intra-cluster network messages
> ---------------------------------------
>
>                 Key: CASSANDRA-8692
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-8692
>             Project: Cassandra
>          Issue Type: Improvement
>          Components: Core
>            Reporter: Ariel Weisberg
>            Assignee: Ariel Weisberg
>             Fix For: 2.1.4
>
>         Attachments: batching-benchmark.png
>
>
> While researching CASSANDRA-8457 we found that it is effective and can be done without
introducing additional latency at low concurrency/throughput.
> The patch from that was used and found to be useful in a real life scenario so I propose
we implement this in 2.1 in addition to 3.0.
> The change set is a single file and is small enough to be reviewable.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message