lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Gaurav gupta <gupta.gaurav0...@gmail.com>
Subject Re: Optimum Lucene’s MMapDirectory size on 64bit OS
Date Sat, 27 Sep 2014 03:19:34 GMT
Thanks Uwe for the insight !

Also, is it advisable to set the lower chunk size for smaller indexes, like
below or let Lucene/OS manage by itself. I am just guessing that assigning
lower value to smaller index will make sure that bigger index are getting
higher mmap address space.

*Index Name  Total Records Size (in GB)   What should be the max. or
optimal chunk size ?*
Address index  106,192,963.00    65             1 GiB
Name index     97,924,594.00      44             1 GiB
GovtId index   81,178,958.00       11              512 MB
Phone index    169,691,376.00    14              512 MB
Email index    46,602,090.00        5               256 MB
Date index     77,243,714.00        6.5             256 MB

Thanks

On Sat, Sep 27, 2014 at 3:40 AM, Uwe Schindler <uwe@thetaphi.de> wrote:

> Hi,
>
> 1 GiB is the maximum possible. The chunk size is only applicable for 32
> bit JDKs because of limited address space.
>
> Uwe
>
> -----
> Uwe Schindler
> H.-H.-Meier-Allee 63, D-28213 Bremen
> http://www.thetaphi.de
> eMail: uwe@thetaphi.de
>
>
> > -----Original Message-----
> > From: Gaurav gupta [mailto:gupta.gaurav0125@gmail.com]
> > Sent: Friday, September 26, 2014 9:12 PM
> > To: java-user@lucene.apache.org
> > Subject: Optimum Lucene’s MMapDirectory size on 64bit OS
> >
> > Hi,
> >
> > As per the post "The Generics Policeman Blog
> > <http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-
> > 64bit.html>"
> > , I am using the MMapDirectory for faster access(search and update
> > operations ,mainly search) of Lucene 4.8.1 index files. I am
> contemplating
> > what is the optimal maximum MMap value for my indexes. Is default i.e. 1
> > GB
> > (1 << 30) or higher?
> >
> > I have 6 indexes of size varying from 65GB to 6 GB. Currently, I am
> using 1 GB
> > as maxChunkSize : - *MMapDirectory(file, null, 1<<30) *for all indexes.
> > But thinking of specifying the higher value for mmap (1 GB or higher) for
> > bigger index having 65GB size and lower value (0.5 GB or less) for
> smaller
> > index having size of 6 GB. Any suggestion/guidance on it ?
> >
> > Also, per blog mmap is not a size of physical memory allocation but just
> a
> > address space to map the index files. How to allocate more RAM to index
> > files for better performance? We have enough RAM free out of 64 GB. Per
> > blog, one should use the  mmap file, like - *MMapDirectory(file, null,
> > 1<<30) *and let OS manage the physical memory allocation for the index
> > files. Is my understanding correct ?
> >
> > The Generics Policeman Blog
> > <http://blog.thetaphi.de/2012/07/use-lucenes-mmapdirectory-on-
> > 64bit.html> :-
> >
> >    - *MMapDirectory does not consume additional memory and the size of
> >    mapped index files is not limited by the physical memory available on
> your
> >    server.* By mmap() files, we only reserve address space not memory!
> >    Remember, address space on 64bit platforms is for free!
> >
> > Thanks
> > Gaurav
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message