logging-log4j-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Remko Popma (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (LOG4J2-431) Create MemoryMappedFileAppender
Date Mon, 23 Dec 2013 14:02:50 GMT

    [ https://issues.apache.org/jira/browse/LOG4J2-431?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13855647#comment-13855647

Remko Popma commented on LOG4J2-431:

Claude, thank you for your contribution!
I've taken a quick look and it looks pretty good.

There may be a few corner cases left when the mapped buffer is nearly full and a new region
needs to be mapped.
(E.g. if the remaining size is say 4 bytes and we want to write 10 bytes, we need to make
sure that the first 4 bytes are written to the old buffer, then the buffer is remapped and
the remaining 6 bytes are written to the new buffer. There may also be a (weird) case when
the mapped region is extremely small and the input byte array is larger than the total size
of the mapped buffer. In this case we need to write chunks of the input that fit in the buffer,
remap the buffer and repeat...)
I could be wrong but could you take another look at these corner cases?

Apart from that it looked pretty good. I hope to be able to spend more time for a more detailed
look next weekend or after New Year.

> Create MemoryMappedFileAppender
> -------------------------------
>                 Key: LOG4J2-431
>                 URL: https://issues.apache.org/jira/browse/LOG4J2-431
>             Project: Log4j 2
>          Issue Type: New Feature
>          Components: Appenders
>            Reporter: Remko Popma
>            Priority: Minor
>         Attachments: MemoryMappedFileAppender.java, MemoryMappedFileAppenderTest.java,
MemoryMappedFileAppenderTest.xml, MemoryMappedFileManager.java, MemoryMappedFileManagerTest.java
> A memory-mapped file appender may have better performance than the ByteBuffer + RandomAccessFile
combination used by the RandomAccessFileAppender. 
> *Drawbacks*
> * The drawback is that the file needs to be pre-allocated and only up to the file size
can be mapped into memory. When the end of the file is reached the appender would need to
extend the file and re-map.
> * Remapping is expensive (I think single-digit millisecond-range, need to check). For
low-latency apps this kind of spike may be unacceptable so careful tuning is required.
> * Memory usage: If re-mapping happens too often you lose the performance benefits, so
the memory-mapped buffer needs to be fairly large, which uses up memory.
> * At roll-over and shutdown the file should be truncated to immediately after the last
written data (otherwise the user is left with a log file that ends in garbage).
> *Advantages*
> Measuring on a Solaris box, the difference between flushing to disk (with {{RandomAccessFile.write(bytes[])}})
and putting data in a MappedByteBuffer is about 20x: around 600ns for a ByteBuffer put and
around 12-15 microseconds for a RandomAccessFile.write.
> (Of course different hardware and OS may give different results...)
> *Use cases*
> The difference may be most visible if {{immediateFlush}} is set to {{true}}, which is
only recommended if async loggers/appenders are not used. If {{immediateFlush=false}}, the
large buffer used by RandomAccessFileAppender means you won't need to touch disk very often.
> So a MemoryMappedFileAppender is most useful in _synchronous_ logging scenarios, where
you get the speed of writing to memory but the data is available on disk almost immediately.
(MMap writes directly to the OS disk buffer.)
> In case of a application crash, the OS ensures that all data in the buffer will be written
to disk. In case of an OS crash the data that was most recently added to the buffer may not
be written to disk.
> Because by nature this appender would occupy a fair amount of memory, it is most suitable
for applications running on server-class hardware with lots of memory available.

This message was sent by Atlassian JIRA

To unsubscribe, e-mail: log4j-dev-unsubscribe@logging.apache.org
For additional commands, e-mail: log4j-dev-help@logging.apache.org

View raw message