lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Uwe Schindler" <...@thetaphi.de>
Subject RE: MMapDirectory need twice more virtual memory than actually need?
Date Sat, 30 Jan 2010 12:04:19 GMT
I am sorry, we know nothing about how the JVM internally handles this. But we do not force
it to map into memory! MMapDirectory just maps but does not call MappedByteBuffer.load().
So it is completely handled by the kernel when something is swapped in or not.

I have a server with 32 GB RAM that mapps about 100 GB files using JAVA's MappedByteBuffer
into its address space and it still runs :-) (Solaris 64 bit, Java 5, no unmap hack).

-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: uwe@thetaphi.de


> -----Original Message-----
> From: luocanrao [mailto:luocan19826164@sohu.com]
> Sent: Saturday, January 30, 2010 12:58 PM
> To: java-user@lucene.apache.org
> Subject: 答复: MMapDirectory need twice more virtual memory than
> actually need?
> 
> I found this in http://osdir.com/ml/linux-kernel/2009-10/msg03503.html
> 
>    If application does mlockall(MCL_FUTURE) it is no longer possible to
> mmap
> file bigger than main memory or allocate big area of anonymous memory
> in a thread safe manner. Sometimes it is desirable to lock everything
> related to program execution into memory, but still be able to mmap
> big file or allocate huge amount of memory and allow OS to swap them on
> demand. MAP_UNLOCKED allows to do that
> 
>   if  mmap method use MAP_UNLOCKED flag, this can expain everything.
>   Is that right?
> 
> -----邮件原件-----
> 发件人: Uwe Schindler [mailto:uwe@thetaphi.de]
> 发送时间: 2010年1月30日 16:47
> 收件人: java-user@lucene.apache.org
> 主题: RE: MMapDirectory need twice more virtual memory than actually
> need?
> 
> MMapDirectory should only be used on 64bit machines with 64bit JVMs.
> This is described in the Javadocs.
> 
> The additional sizes are caused by reopens and have their reason in
> specialties with the Java VM, described in 2.9/3.0 Javadocs of the
> class, please take a look. You can also enable a workaround for the
> missing “unmap” call.
> 
> 
> 
> -----
> 
> Uwe Schindler
> 
> H.-H.-Meier-Allee 63, D-28213 Bremen
> 
>  <http://www.thetaphi.de/> http://www.thetaphi.de
> 
> eMail: uwe@thetaphi.de
> 
> 
> 
> From: luocanrao [mailto:luocan19826164@sohu.com]
> Sent: Saturday, January 30, 2010 8:31 AM
> To: java-user@lucene.apache.org
> Subject: MMapDirectory need twice more virtual memory than actually
> need?
> 
> 
> 
> Environment: 64 bit linux,memory 8G
> 
> When I used pmap instruction to see virtual memory, I found two big
> anon memory which grows with the index file size.
> 
> I had the two following pictures to show the problem, can you explain?
> 
>          This is why I got out of memory exception in 32 bit machine.
> Because my index file size is only 800m.
> 
>                                                                  Anon
> memory picture
> 
> 
> 
> 
> Indexfile memory picture
> 
> 
> 
> 
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message