lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Uwe Schindler" <...@thetaphi.de>
Subject RE: why lucene not use DirectByteBuffer in NIOFSDirectory
Date Wed, 31 Jul 2013 10:18:58 GMT
Hi,

There is a misunderstanding: Just by allocating a direct buffer, there is still no difference
to a heap buffer in the workflow!

NIO will read the data from file, copy it to FS cache and then the positional read() system
call (used by NIO) copies the FS cache contents to the direct buffer, no real difference to
a heap buffer. So it is still the same: data needs to be copied. Please note: Direct buffers
have nothing to do with file system cache, they don't share the same memory areas! Hotspot
allocates direct buffers using malloc() outside of Java heap, so not really useful here.

The backside of using a non-heap buffer as target for the copy operation is the fact, that
direct buffers are approx. 2 times slower when accessed from Java code (because they are outside
java heap, the VM has to do extra work to prevent illegal accesses: so you have the same time
for copy but slower access from Java. The buffers allocated by NIO are small so it does not
matter for performance where they are. So heap is better. MappedByteBuffers are also direct
buffers (they have the same base class), so there is still the overhead when accessing them
from Java code, but to get rid of the additional copy, use MMapDirectory.

To conclude:
- MMapDirectory: No copying of the data from FS-cache to heap or direct buffers needed, which
wastes most of the time. Access times to MappedByteBuffer from Java code is slower, but the
spared  data copy makes it much better for large files as used by Lucene.
- NIOFSDirectory with direct buffers: Needs to copy data from FS cache to direct buffer memory
(outside heap). Access times slower to direct buffers than to heap buffers -> 2 times bad
- NIOFSDirectory with heap buffers: Needs to copy data from FS cache to heap. Access time
from java code is very good!

Uwe

-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: uwe@thetaphi.de


> -----Original Message-----
> From: wangzhijiang999@yahoo.com.cn
> [mailto:wangzhijiang999@yahoo.com.cn]
> Sent: Wednesday, July 31, 2013 11:59 AM
> To: java-user@lucene.apache.org
> Subject: why lucene not use DirectByteBuffer in NIOFSDirectory
> 
> I read this article "Use Lucene's MMapDirectory on 64bit platforms, please!"
> and it said the MMapDirectory is better than other Directory because it will
> void copy data between file system cache and java heap.
> 
> I checked the source code of NIOFSDirectory, and in new Buffer method it
> called "ByteBuffer.wrap(newBuffer)", the generated ByteBuffer is
> HeapByteBuffer. And it will indeed copy data between file system cache and
> java heap. Why not use ByteBuffer.allocateDirect to generate
> DirectyByteBuffer, and it will store data directly in file sysytem cache, not
> java heap. If in this case , what is the different in performance between NIO
> and MMap?  Or allocate directy memory in still slower than Mmap?
> 
> Maybe I made some misunderstanding of lucene code, thank you for any
> suggestion in advance.
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message