lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sadaf Aslam <Sadaf.As...@tpsgc-pwgsc.gc.ca>
Subject RE: OOM error
Date Fri, 06 Jan 2017 19:43:17 GMT
Thanks for the suggestion Uwe.

We will look into upgrading to a newer version. But upgrading to 6 would mean using Java 8
and we are not ready for that yet.

We also want to start using DrillSideways. What version was that introduced in? I am using
version 4.4 and DrillSideways is available but it is not the version that is being used in
most examples online. What is the first stable version that would allow me to use it with
the least amount of upgrade work involved. Also, I want to use FacetFields instead of CategoryPaths.

Is there a good webpage that gives the summary of the features introduced into a version?
I looked at https://lucene.apache.org/ but couldn’t find when DrillSideways (or FacetFields)
was introduced.

Thanks,
Sadaf

From: Uwe Schindler [mailto:uwe@thetaphi.de]
Sent: December-23-16 1:27 PM
To: java-user@lucene.apache.org; Sadaf Aslam <Sadaf.Aslam@tpsgc-pwgsc.gc.ca>
Subject: Re: OOM error

Hi,

For sorting you should add the fields as DocValues type in addition to indexed fields (if
you also query on them). That ist the problem. But for that you should update your Lucene
version.

For that reason Lucene 6 no longer has FieldCache, which emulates DocValues loading everything
to heap, which is horrible for StringFields.

Uwe
Am 23. Dezember 2016 18:14:50 MEZ schrieb Sadaf <Sadaf.Aslam@tpsgc-pwgsc.gc.ca<mailto:Sadaf.Aslam@tpsgc-pwgsc.gc.ca>>:

Hi,

This is the index we are using:
Number of fields: 355.
Number of documents: 225 thousand.
Number of terms: 5522 thousand.
The index size is around 800MB.
TermInfos index divisor: 1
Index format: Lucene 4.0.
We are getting Java OutOfMemory error with searches. We are using a heap
size of 1GB. We are not able to increase our heap size.
Looking at the HeapDump, we have two suspects: FieldCacheImpl and
FieldCacheImpl$BinaryDocValuesImpl.
The searches we are doing have a lot of sorts. The sorting is being done on
string fields. I don’t have much experience with OutOfMemory errors or with
Lucene. What should I try to do?
- Should I try to play with the swappiness on the server. Currently set to
60?
- We are creating a new IndexSearcher for each search? Should I just have
one? If I do that, will my results be updated as new documents are being
added to the index?
- If you think that sorting on StringFields is the problem can you give me
some pointers on what the usual suspects are?
-      Should I insist that we need more Heap?
Thanks, (I will not be checking my email until new year. Wishing everyone
here Happy Holidays)
Sadaf



--
View this message in context: http://lucene.472066.n3.nabble.com/OOM-error-tp4311073.html
Sent from the Lucene - Java Users mailing list archive at Nabble.com<http://Nabble.com>.

________________________________

To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org<mailto:java-user-unsubscribe@lucene.apache.org>
For additional commands, e-mail: java-user-help@lucene.apache.org<mailto:java-user-help@lucene.apache.org>

--
Uwe Schindler
Achterdiek 19, 28357 Bremen
https://www.thetaphi.de
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message