cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ariel Weisberg (JIRA)" <>
Subject [jira] [Commented] (CASSANDRA-13530) GroupCommitLogService
Date Wed, 06 Sep 2017 15:00:02 GMT


Ariel Weisberg commented on CASSANDRA-13530:

{quote}I'm sorry from outside.
What do you mean `That documentation in the YAML looks wrong to me.` ?
In the apache doc, it also states 2ms is the default value.
I'm not really sure what you are trying to say here, 
but the current batch is not working as expected as I described below, and it is not very
I am agreeing with Benedict the documentation everywhere is wrong.

Even if `commitlog_sync_batch_window_in_ms` is set a big number, 
it is the maximum length of time that queries may be batched together for, not the minimum,
so, it is pretty nondeterministic and the behavior is not predictable.{quote}
Predictability isn't the goal though it's the lowest average latency and lowest P99 (or whatever).
More variability but lower average and P99 is still better.

{quote}You can't really balance between latency and throughput.{quote}
Is this true? Fsync latency has a fixed cost as well as a variable cost that is linear with
the amount of data being written. So calling fsync as often as possible with whatever data
is available seems like a reasonable strategy if you have a dedicated device that is doing
nothing but waiting for the commit log to flush.
This should balance latency and throughput at any load level. The more throughput you have
the more latency you have as the fsyncs take a little longer. The less throughput you have
the less latency you have as the fsyncs are a little faster. But either way the latency is
determined by the willingness of the device to sync data and not by a hard coded configuration
which may or may not be optimal. Devices also don't have predictable fsync latency over time.
As they fill up or run out of erase blocks or are contended by other IO the optimal batch
size may change.

We see this effect at low concurrency where I expect it to be pronounced. What's unexpected
is that we are also seeing worse throughput as op rate increases which is unexpected because
I would expect the batches to grow as fsync latency increases until it converges on the optimal
batch size for the device.

> GroupCommitLogService
> ---------------------
>                 Key: CASSANDRA-13530
>                 URL:
>             Project: Cassandra
>          Issue Type: Improvement
>            Reporter: Yuji Ito
>            Assignee: Yuji Ito
>             Fix For: 2.2.x, 3.0.x, 3.11.x
>         Attachments: groupCommit22.patch, groupCommit30.patch, groupCommit3x.patch, groupCommitLog_noSerial_result.xlsx,
> I propose a new CommitLogService, GroupCommitLogService, to improve the throughput when
lots of requests are received.
> It improved the throughput by maximum 94%.
> I'd like to discuss about this CommitLogService.
> Currently, we can select either 2 CommitLog services; Periodic and Batch.
> In Periodic, we might lose some commit log which hasn't written to the disk.
> In Batch, we can write commit log to the disk every time. The size of commit log to write
is too small (< 4KB). When high concurrency, these writes are gathered and persisted to
the disk at once. But, when insufficient concurrency, many small writes are issued and the
performance decreases due to the latency of the disk. Even if you use SSD, processes of many
IO commands decrease the performance.
> GroupCommitLogService writes some commitlog to the disk at once.
> The patch adds GroupCommitLogService (It is enabled by setting `commitlog_sync` and `commitlog_sync_group_window_in_ms`
in cassandra.yaml).
> The difference from Batch is just only waiting for the semaphore.
> By waiting for the semaphore, some writes for commit logs are executed at the same time.
> In GroupCommitLogService, the latency becomes worse if the there is no concurrency.
> I measured the performance with my microbench ( by increasing
the number of threads.The cluster has 3 nodes (Replication factor: 3). Each nodes is AWS EC2
m4.large instance + 200IOPS io1 volume.
> The result is as below. The GroupCommitLogService with 10ms window improved update with
Paxos by 94% and improved select with Paxos by 76%.
> h6. SELECT / sec
> ||\# of threads||Batch 2ms||Group 10ms||
> |1|192|103|
> |2|163|212|
> |4|264|416|
> |8|454|800|
> |16|744|1311|
> |32|1151|1481|
> |64|1767|1844|
> |128|2949|3011|
> |256|4723|5000|
> h6. UPDATE / sec
> ||\# of threads||Batch 2ms||Group 10ms||
> |1|45|26|
> |2|39|51|
> |4|58|102|
> |8|102|198|
> |16|167|213|
> |32|289|295|
> |64|544|548|
> |128|1046|1058|
> |256|2020|2061|

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message