logging-log4j-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Afonso Murakami <murakam...@icloud.com>
Subject Re: Experimenting with NIO
Date Sun, 26 Feb 2017 19:03:12 GMT
I don’t know if that make any difference but I click on the link https://githhub.com/vz/nio-looger
<https://githhub.com/vz/nio-looger> to see what is about and when I finished I sign
of on my account may that screw up something or may be not, I’m not sure if Idid that or
not, sorry.
> On Feb 26, 2017, at 10:49 AM, Matt Sicker <boards@gmail.com> wrote:
> 
> I could be doing something wrong entirely here, but so far, the fastest way I've found
to write to a file (not including mmap yet) is via:
> 
> new BufferedOutputStream(Files.newOutputStream(Paths.get("test.log")))
> 
> And this is with added synchronization on the append() method, too. Also, one of my updates
to the async channel version is causing OOM errors in JMH now, so I broke something.
> 
> On 26 February 2017 at 12:22, Matt Sicker <boards@gmail.com <mailto:boards@gmail.com>>
wrote:
> I added some basic JMH tests to my repo along with a couple alternative appender implementations.
I got rid of the unnecessary file region locking in the async file channel one, but it's still
coming out quite a bit slower than the RandomAccessFile and Files.newOutputStream() based
appenders, though that could be due to the use of Phaser (which I only added to cleanly close
the appender synchronously).
> 
> On 26 February 2017 at 10:05, Matt Sicker <boards@gmail.com <mailto:boards@gmail.com>>
wrote:
> Perhaps something got optimized by the JVM? I'll add some JMH tests to this repo to try
out various approaches.
> 
> On Sat, Feb 25, 2017 at 21:12, Apache <ralph.goers@dslextreme.com <mailto:ralph.goers@dslextreme.com>>
wrote:
> I tried using a FileChannel for the FileAppender a week or so ago to see if passing the
ByteBuffer to the FileChannel would improve performance since it doesn’t have to be synchronized.
I didn’t see any improvement though and I ended up reverting it. But I might have done something
wrong.
> 
> Ralph
> 
>> On Feb 25, 2017, at 4:19 PM, Matt Sicker <boards@gmail.com <mailto:boards@gmail.com>>
wrote:
>> 
>> We already use a bit of NIO (ByteBuffer for layouts and appenders/managers, MappedByteBuffer
for mmap'd files, FileLock for locking files, etc.), and I've been playing around with the
NIO API lately. I have some sample code here <https://github.com/jvz/nio-logger <https://github.com/jvz/nio-logger>>
to show some trivial use case of AsynchronousFileChannel. In Java 7, there is also AsynchronousSocketChannel
which could theoretically be used instead of adding Netty for a faster socket appender. In
that regard, I'm curious as to how useful it would be to have similar appenders as the OutputStream
ones, but instead using WritableByteChannel, GatheringByteChannel (possible parallelization
of file writing?), and the async channels (there's an AsynchronousByteChannel class, but I
think they screwed this one up as only one of the three async channel classes implements it).
>> 
>> Another related issue I've seen is that in a message-oriented appender (e.g., the
Kafka one), being able to stream directly to a ByteBuffer is not the right way to go about
encoding log messages into the appender. Instead, I was thinking that a pool of reusable ByteBuffers
could be used here where a ByteBuffer is borrowed on write and returned on completion (via
a CompletionHandler callback). The Kafka client uses a similar strategy for producing messages
by dynamically allocating a pool of ByteBuffers based on available memory.
>> 
>> Also, I don't have much experience with this, but if we had a pool of reusable ByteBuffers,
could we use direct allocation to get off-heap buffers? That seems like an interesting use
case.
>> 
>> --
>> Matt Sicker <boards@gmail.com <mailto:boards@gmail.com>>
> 
> --
> Matt Sicker <boards@gmail.com <mailto:boards@gmail.com>>
> 
> 
> 
> --
> Matt Sicker <boards@gmail.com <mailto:boards@gmail.com>>
> 
> 
> 
> --
> Matt Sicker <boards@gmail.com <mailto:boards@gmail.com>>


Mime
View raw message