cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Pavel Yaskevich (JIRA)" <>
Subject [jira] Commented: (CASSANDRA-1470) use direct io for compaction
Date Thu, 09 Dec 2010 21:31:02 GMT


Pavel Yaskevich commented on CASSANDRA-1470:

This is more complex that it seems to be from the first sight: All Direct I/O operations will
be synchronous; operations will not return until completed. On the other hand standard I/O
(using page cache) uses mechanism called Page Writeback - write syscalls place pages into
memory (if necessary), mark them as dirty and return immediately after that (pdflush kernel
threads are responsible for syncing page cache data to the device) actual memory<->device
synchronization happens when:

1) When free memory shrinks below a configurable threshold, dirty buffers are written back
to disk so that the now-clean buffers may be removed, freeing memory.
2) When a dirty buffer ages beyond a configurable threshold, the buffer is written back to
disk. This prevents data from remaining dirty indefinitely.

Deferred writes and buffer subsystem in Linux enable fast writes...

More on this could be found in the book "Linux System Programming" in section File I/O ->
Kernel Internals or in the book "Understanding The Linux Kernel 3rd Edition"...

> use direct io for compaction
> ----------------------------
>                 Key: CASSANDRA-1470
>                 URL:
>             Project: Cassandra
>          Issue Type: Improvement
>          Components: Core
>            Reporter: Jonathan Ellis
>            Assignee: Pavel Yaskevich
>             Fix For: 0.7.1
>         Attachments: 1470-v2.txt, 1470.txt, CASSANDRA-1470-for-0.6.patch, CASSANDRA-1470-v10-for-0.7.patch,
CASSANDRA-1470-v11-for-0.7.patch, CASSANDRA-1470-v12-0.7.patch, CASSANDRA-1470-v2.patch, CASSANDRA-1470-v3-0.7-with-LastErrorException-support.patch,
CASSANDRA-1470-v4-for-0.7.patch, CASSANDRA-1470-v5-for-0.7.patch, CASSANDRA-1470-v6-for-0.7.patch,
CASSANDRA-1470-v7-for-0.7.patch, CASSANDRA-1470-v8-for-0.7.patch, CASSANDRA-1470-v9-for-0.7.patch,
CASSANDRA-1470.patch, use.DirectIORandomAccessFile.for.commitlog.against.1022235.patch
> When compaction scans through a group of sstables, it forces the data in the os buffer
cache being used for hot reads, which can have a dramatic negative effect on performance.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message