cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ariel Weisberg (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (CASSANDRA-13530) GroupCommitLogService
Date Mon, 15 May 2017 21:38:04 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-13530?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16011365#comment-16011365
] 

Ariel Weisberg edited comment on CASSANDRA-13530 at 5/15/17 9:37 PM:
---------------------------------------------------------------------

Thanks for investigating this.

I don't think the benchmark is realistic. If you simulate a workload with a fixed concurrency
(which is how most systems operate) and then vary the arrival rate of new tasks instead of
the concurrency you should find that batch commit log performs well without any tuning. I
think not having to tune to get good performance is very important in ensuring everyone using
C* has good performance.

What is the workload where concurrency is bounded between 2 and 16 and won't increase if the
database isn't returning a response fast enough? I know such workloads exist, but are they
C* workloads? Is it worth adding a system that has to be manually tuned to try and get better
synchronous commit log performance at concurrency between 2 and 16? It's not a lot of code
so I am not against it I just want to be sure we are doing it for the right reason and someone
is getting some value from it.

I think there is a much more serious issue which is the fact that batch sizes are limited
to mutation thread pool size if I understand things correctly. I think it should be possible
to demonstrate that issue by benchmarking batch vs periodic and observing the resulting difference
in throughput. Batch should be as almost high throughput as periodic with more latency.


was (Author: aweisberg):
Thanks for investigating this.

I don't think benchmark is realistic. If you simulate a workload with a fixed concurrency
(which is how most systems operate) and then vary the arrival rate of new tasks instead of
the concurrency you should find that batch commit log performs well without any tuning. I
think not having to tune to get good performance is very important in ensuring everyone using
C* has good performance.

What is the workload where concurrency is bounded between 2 and 16 and won't increase if the
database isn't returning a response fast enough? I know such workloads exist, but are they
C* workloads? Is it worth adding a system that has to be manually tuned to try and get better
synchronous commit log performance at concurrency between 2 and 16? It's not a lot of code
so I am not against it I just want to be sure we are doing it for the right reason and someone
is getting some value from it.

I think there is a much more serious issue which is the fact that batch sizes are limited
to mutation thread pool size if I understand things correctly. I think it should be possible
to demonstrate that issue by benchmarking batch vs periodic and observing the resulting difference
in throughput. Batch should be as almost high throughput as periodic with more latency.

> GroupCommitLogService
> ---------------------
>
>                 Key: CASSANDRA-13530
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-13530
>             Project: Cassandra
>          Issue Type: Improvement
>            Reporter: Yuji Ito
>            Assignee: Yuji Ito
>             Fix For: 2.2.x, 3.0.x, 3.11.x
>
>         Attachments: groupCommit22.patch, groupCommit30.patch, groupCommit3x.patch, MicroRequestThread.java
>
>
> I propose a new CommitLogService, GroupCommitLogService, to improve the throughput when
lots of requests are received.
> It improved the throughput by maximum 94%.
> I'd like to discuss about this CommitLogService.
> Currently, we can select either 2 CommitLog services; Periodic and Batch.
> In Periodic, we might lose some commit log which hasn't written to the disk.
> In Batch, we can write commit log to the disk every time. The size of commit log to write
is too small (< 4KB). When high concurrency, these writes are gathered and persisted to
the disk at once. But, when insufficient concurrency, many small writes are issued and the
performance decreases due to the latency of the disk. Even if you use SSD, processes of many
IO commands decrease the performance.
> GroupCommitLogService writes some commitlog to the disk at once.
> The patch adds GroupCommitLogService (It is enabled by setting `commitlog_sync` and `commitlog_sync_group_window_in_ms`
in cassandra.yaml).
> The difference from Batch is just only waiting for the semaphore.
> By waiting for the semaphore, some writes for commit logs are executed at the same time.
> In GroupCommitLogService, the latency becomes worse if the there is no concurrency.
> I measured the performance with my microbench (MicroRequestThread.java) by increasing
the number of threads.The cluster has 3 nodes (Replication factor: 3). Each nodes is AWS EC2
m4.large instance + 200IOPS io1 volume.
> The result is as below. The GroupCommitLogService with 10ms window improved update with
Paxos by 94% and improved select with Paxos by 76%.
> h6. SELECT / sec
> ||\# of threads||Batch 2ms||Group 10ms||
> |1|192|103|
> |2|163|212|
> |4|264|416|
> |8|454|800|
> |16|744|1311|
> |32|1151|1481|
> |64|1767|1844|
> |128|2949|3011|
> |256|4723|5000|
> h6. UPDATE / sec
> ||\# of threads||Batch 2ms||Group 10ms||
> |1|45|26|
> |2|39|51|
> |4|58|102|
> |8|102|198|
> |16|167|213|
> |32|289|295|
> |64|544|548|
> |128|1046|1058|
> |256|2020|2061|



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org


Mime
View raw message