cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Satoshi Konno (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (CASSANDRA-11303) New inbound throughput parameters for streaming
Date Tue, 19 Apr 2016 06:25:25 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-11303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15247249#comment-15247249
] 

Satoshi Konno edited comment on CASSANDRA-11303 at 4/19/16 6:25 AM:
--------------------------------------------------------------------

Hi [~pauloricardomg],

I have uploaded a new patch for the inbound limiter function. I added the inbound limiter
into read() of StreamReader and CompressedStreamReader instead of Row::dataSize() because
it seems that it is difficult to expand SSTableMultiWriter. I updated the read() to output
a debug parameter, 'rateLimitedTime', which is a total sleeping time for the specified inbound
limiter.

I have uploaded two debug logs for the test results using ccm too. To check the new function,
I add a new node into a cluster which has three nodes. The debug logs are outputed by the
new node which is added at last. The debug logs are set the inbound parameters in 'cassandra.yaml'
as the following:

--------------------------------------------------------------------------------
  11303_inbound_limit_debug_20160419.log :
      stream_throughput_inbound_megabits_per_sec: 1
      inter_dc_stream_throughput_inbound_megabits_per_sec: 1

  11303_inbound_nolimit_debug_20160419.log : (Not Specified)
      \# stream_throughput_inbound_megabits_per_sec: 200
      \# inter_dc_stream_throughput_inbound_megabits_per_sec: 200
--------------------------------------------------------------------------------

To check effects of the inbound limiter in the debug logs, please check only the lines which
has the 'rateLimitedTime' parameters. If the 'rateLimitedTime' outputs zero, the inbound limiters
are not effective.

Please let me know if you have any suggestions.


was (Author: skonno):
Hi [~pauloricardomg],

I have uploaded a new patch for the inbound limiter function. I added the inbound limiter
into read() of StreamReader and CompressedStreamReader instead of Row::dataSize() because
it seems that it is difficult to expand SSTableMultiWriter. I updated the read() to output
a debug parameter, 'rateLimitedTime', which is a total sleeping time for the specified inbound
limiter.

I have uploaded two debug logs for the test results using ccm too. To check the new function,
I add a new node into a cluster which has three nodes. The debug logs are outputed by the
new node which is added at last. The debug logs are set the inbound parameters in 'cassandra.yaml'
as the following:

  11303_inbound_limit_debug_20160419.log :
      stream_throughput_inbound_megabits_per_sec: 1
      inter_dc_stream_throughput_inbound_megabits_per_sec: 1

  11303_inbound_nolimit_debug_20160419.log : (Not Specified)
      \# stream_throughput_inbound_megabits_per_sec: 200
      \# inter_dc_stream_throughput_inbound_megabits_per_sec: 200

To check effects of the inbound limiter in the debug logs, please check only the lines which
has the 'rateLimitedTime' parameters. If the 'rateLimitedTime' outputs zero, the inbound limiters
are not effective.

Please let me know if you have any suggestions.

> New inbound throughput parameters for streaming
> -----------------------------------------------
>
>                 Key: CASSANDRA-11303
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-11303
>             Project: Cassandra
>          Issue Type: New Feature
>          Components: Configuration
>            Reporter: Satoshi Konno
>            Priority: Minor
>         Attachments: 11303_inbound_limit_debug_20160419.log, 11303_inbound_nolimit_debug_20160419.log,
11303_inbound_patch_for_trunk_20160419.diff, cassandra_inbound_stream.diff
>
>
> Hi,
> To specify stream throughputs of a node more clearly, I would like to add the following
new inbound parameters like existing outbound parameters in the cassandra.yaml.
> - stream_throughput_inbound_megabits_per_sec
> - inter_dc_stream_throughput_outbound_megabits_per_sec  
> We use only the existing outbound parameters now, but it is difficult to control the
total throughputs of a node. In our production network, some critical alerts occurs when
a node exceed the specified total throughput which is the sum of the input and output throughputs.
> In our operation of Cassandra, the alerts occurs during the bootstrap or repair processing
when a new node is added. In the worst case, we have to stop the operation of the exceed node.
> I have attached the patch under consideration. I would like to add a new limiter class,
StreamInboundRateLimiter, and use the limiter class in StreamDeserializer class. I use Row::dataSize(
)to get the input throughput in StreamDeserializer::newPartition(), but I am not sure whether
the dataSize() returns the correct data size.
> Can someone please tell me how to do it ?



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message