lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Clemens Wyss DEV <clemens...@mysign.ch>
Subject AW: [lucene 4.6] NPE when calling IndexReader#openIfChanged
Date Sun, 15 Jun 2014 18:16:23 GMT
> bytes (ie, ~56.5 MB) being used by the StandardDirectoryReader?
Right, namely in org.apache.lucene.util.fst.BytesStore's 

Class Name                                                                               
        | Shallow Heap | Retained Heap | Percentage
----------------------------------------------------------------------------------------------------------------------------------------------
org.apache.lucene.index.StandardDirectoryReader @ 0x783932460                            
        |           72 |    59'255'872 |      3.04%
|- org.apache.lucene.index.SegmentReader[24] @ 0x794089ee0                               
        |          112 |    59'190'960 |      3.03%
|  |- org.apache.lucene.index.SegmentReader @ 0x788820f40                                
        |           72 |    16'905'072 |      0.87%
|  |  |- org.apache.lucene.index.SegmentCoreReaders @ 0x7910cacc8                        
        |           56 |    16'895'576 |      0.87%
|  |  |  |- org.apache.lucene.codecs.perfield.PerFieldPostingsFormat$FieldsReader @ 0x780661c50
  |           24 |    16'864'864 |      0.86%
|  |  |  |  |- org.apache.lucene.codecs.BlockTreeTermsReader @ 0x7910cae50               
        |           56 |    16'864'240 |      0.86%
|  |  |  |  |  |- java.util.TreeMap @ 0x783902738                                        
        |           48 |    16'858'472 |      0.86%
|  |  |  |  |  |  '- java.util.TreeMap$Entry @ 0x77ec5f9f8                               
        |           40 |    16'858'424 |      0.86%
|  |  |  |  |  |     |- java.util.TreeMap$Entry @ 0x77ec5fa20                            
        |           40 |    10'895'656 |      0.56%
|  |  |  |  |  |     |  |- org.apache.lucene.codecs.BlockTreeTermsReader$FieldReader @ 0x78347fc08|
          72 |     5'497'328 |      0.28%
|  |  |  |  |  |     |  |  |- org.apache.lucene.util.fst.FST @ 0x788fe36d0               
        |          104 |     5'497'208 |      0.28%
|  |  |  |  |  |     |  |  |  |- org.apache.lucene.util.fst.BytesStore @ 0x77ec5fe80     
        |           40 |     5'485'824 |      0.28%
|  |  |  |  |  |     |  |  |  |- org.apache.lucene.util.fst.FST$Arc[128] @ 0x786efec80   
        |          528 |        11'216 |      0.00%
|  |  |  |  |  |     |  |  |  |- org.apache.lucene.util.BytesRef @ 0x780664410           
        |           24 |            48 |      0.00%
|  |  |  |  |  |     |  |  |  |- int[0] @ 0x7806643f8                                    
        |           16 |            16 |      0.00%
|  |  |  |  |  |     |  |  |  '- Total: 4 entries                                        
        |              |               |           
----------------------------------------------------------------------------------------------------------------------------------------------
-----Urspr√ľngliche Nachricht-----
Von: Michael McCandless [mailto:lucene@mikemccandless.com] 
Gesendet: Freitag, 13. Juni 2014 15:48
An: Lucene Users
Betreff: Re: [lucene 4.6] NPE when calling IndexReader#openIfChanged

On Fri, Jun 13, 2014 at 8:53 AM, Clemens Wyss DEV <clemensdev@mysign.ch> wrote:
> Thanks a lot!
>>"large text fields"
> What is a good limit (in characters) to switch from StringField to TextField? Do <Langugae>Analyzers
(e.g. GermanAnalyzer)  help a lot in reducing the size of an Index?

It's more based on your app's requirements.  StringField indexes everything as a single token.

>> Add XXXDocValuesField instead of e.g. StringField.
> Does this apply only for StringFields? Or for TextFields too?
>
>> Upgrade to the upcoming Lucene 4.9
> we have not yet transitionen to Java 7/8 ... hopefully soon ;)
>
>> and take a heap dump and see what's using RAM
> Find attached a snippet from MemoryAnalyzer

Does this say  59255872 bytes (ie, ~56.5 MB) being used by the StandardDirectoryReader?

I'm a little confused because I don't see which structures sum up to that total.  And I would
expect the FST (terms index) to take more RAM.

Mike McCandless

http://blog.mikemccandless.com

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org

Mime
View raw message