cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Peter Schuller (JIRA)" <j...@apache.org>
Subject [jira] Commented: (CASSANDRA-1470) use direct io for compaction
Date Thu, 09 Dec 2010 17:46:03 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-1470?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12969827#action_12969827
] 

Peter Schuller commented on CASSANDRA-1470:
-------------------------------------------

Note that at minimum the amount written per direct write has to be high enough that the seek
overhead of constituent disks becomes irrelevant (and keeping in mind that there may be multiple
spindles under a RAID controller). In addition on a loaded system there will be additional
latency resulting from the I/O requests queueing, which increases the demands for the size
of individual writes. I would expect a sensible "chunk" size to be several megabyte at minimum
for writes.

The other trade-off for reads (and writes if not using battery-backed write caching in a RAID
controller) is that too large "chunk:s" will be detrimental to latency of live traffic. I
exclude writes on battery backed controllers under the assumption that the chunk size is well
below cache size in the controller. However, on the other hand there is a throttling issue
here so unless rate limiting is applied you would see detrimental affects with a RAID controller
too since it is going to eat your writes until it becomes full, at which point submission
of a huge write will probably again have the detrimental effects on latency.

> use direct io for compaction
> ----------------------------
>
>                 Key: CASSANDRA-1470
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-1470
>             Project: Cassandra
>          Issue Type: Improvement
>          Components: Core
>            Reporter: Jonathan Ellis
>            Assignee: Pavel Yaskevich
>             Fix For: 0.7.1
>
>         Attachments: 1470-v2.txt, 1470.txt, CASSANDRA-1470-for-0.6.patch, CASSANDRA-1470-v10-for-0.7.patch,
CASSANDRA-1470-v11-for-0.7.patch, CASSANDRA-1470-v12-0.7.patch, CASSANDRA-1470-v2.patch, CASSANDRA-1470-v3-0.7-with-LastErrorException-support.patch,
CASSANDRA-1470-v4-for-0.7.patch, CASSANDRA-1470-v5-for-0.7.patch, CASSANDRA-1470-v6-for-0.7.patch,
CASSANDRA-1470-v7-for-0.7.patch, CASSANDRA-1470-v8-for-0.7.patch, CASSANDRA-1470-v9-for-0.7.patch,
CASSANDRA-1470.patch, use.DirectIORandomAccessFile.for.commitlog.against.1022235.patch
>
>
> When compaction scans through a group of sstables, it forces the data in the os buffer
cache being used for hot reads, which can have a dramatic negative effect on performance.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message