activemq-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "james (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (AMQ-5578) preallocate journal files
Date Wed, 20 May 2015 18:14:00 GMT

    [ https://issues.apache.org/jira/browse/AMQ-5578?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14552808#comment-14552808
] 

james commented on AMQ-5578:
----------------------------

Two things.  First, i'm curious what the status of this issue is?  the code seems to be largely
complete, but the issue is still open?  i am interested in this issue because we have encountered
a performance bottleneck which seems to be improved by this patch.

Second, i have a proposed additional option for the preallocation strategy.  we use activemq
embedded within our application.  As such, i don't want to use the ZEROS strategy because
it requires allocating a 32MB chunk of memory.  However, our application can run in scenarios
where the underlying filesystem is networked.  This makes OS_KERNEL_COPY somewhat less ideal
since it puts the temp file in the same filesystem as the real data files.  while this strategy
works well in terms of the overall performance boost, it can cause up to a 5 second delay
in the application layer while the new journal file is being preallocated.  provided below
is a variant on the ZEROS policy which i have called CHUNKED_ZEROS.  this uses a small (1MB)
buffer to do the preallocation.  this seems to provide a similar overall performance improvement
while reducing the max application layer delay to 3 seconds (and not having the potentially
large overhead of the ZEROS strategy).

{code}
    private static final int PREALLOC_CHUNK_SIZE = 1 << 20;

    private void doPreallocationChunkedZeros(RecoverableRandomAccessFile file) {

        ByteBuffer buffer = ByteBuffer.allocate(PREALLOC_CHUNK_SIZE);
        buffer.position(0);
        buffer.limit(PREALLOC_CHUNK_SIZE);
            
        try {
            FileChannel channel = file.getChannel();

            int remLen = maxFileLength;
            while(remLen > 0) {
                if(remLen < buffer.remaining()) {
                    buffer.limit(remLen);
                }
                int writeLen = channel.write(buffer);
                remLen -= writeLen;
                buffer.rewind();
            }            
            
            channel.force(false);
            channel.position(0);
        } catch (IOException e) {
            LOG.error("Could not preallocate journal file with zeros! Will continue without
preallocation", e);
        }
    }
{code}

> preallocate journal files
> -------------------------
>
>                 Key: AMQ-5578
>                 URL: https://issues.apache.org/jira/browse/AMQ-5578
>             Project: ActiveMQ
>          Issue Type: Improvement
>          Components: Message Store
>    Affects Versions: 5.11.0
>            Reporter: Gary Tully
>            Assignee: Gary Tully
>              Labels: journal, kahaDB, perfomance
>             Fix For: 5.12.0
>
>
> Our journals are append only, however we use the size to track journal rollover on recovery
and replay. We can improve performance if we never update the size on disk and preallocate
on creation.
> Rework journal logic to ensure size is never updated. This will allow the configuration
option from https://issues.apache.org/jira/browse/AMQ-4947 to be the default.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message