lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Michael McCandless <luc...@mikemccandless.com>
Subject Re: Lucene 3.0 Search Performance Stats
Date Mon, 22 Mar 2010 12:40:39 GMT
Looks like the bulk of your RAM usage is from the 370K index terms in
your terms dict...

The flex branch (once it lands) should substantially reduce that...

Mike

On Mon, Mar 22, 2010 at 8:35 AM, Jamie <jamie@stimulussoft.com> wrote:
> Hi Everyone
>
> The stats I sent through earlier were erroneous due to fact the date range
> query selected fewer records than stated.
>
> The correct stats are:
>
> Lucene 3.0 Stats:
>
> Search conducted using Lucene's Realtime search feature (writer.getReader()
> for each search)
> Analyzer: Russian Analyzer
> Total Docs: 26.04M (emails data - all attachments, body content indexed)
> Index Size: 37G
> Query: body: test AND  date: [200901010101 to 201003220225] with descending
> sort on date.
> Lucene Mem Usage: 32 MB (when sorted on date)
> Search Speed: 0.48 (unsorted)
> Search Speed: 0.49 (sorted on YYYYMMDDHHSS date)
>
> Hardware / Software:
>
> Index stored on 4 SAS HDD hitachi RAID 10
> 16G RAM
> 2x Xeon 4 core 2.4Gz
> OS FreeBSD 7.2
> Filesystem UFS2 gjournal
>
> From Yourkit, memory usage is as follows:
>
> Name    Number of Objects       Shallow Size
> org.apache.lucene.index.TermInfo        370610  14824400
> org.apache.lucene.index.Term    370505  11856160
> com.stimulus.archiva.search.LuceneResult        20000   960000
> org.apache.lucene.search.FieldDoc       10000   320000
> org.apache.lucene.search.ScoreDoc       10000   240000
> org.apache.lucene.index.FieldInfo       1027    41080
> org.apache.lucene.index.SegmentReader$Norm      840     67200
> org.apache.lucene.document.Field        578     36992
> org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput  415     49800
> org.apache.lucene.index.CompoundFileReader$CSIndexInput         380
> 36480
> org.apache.lucene.util.UnicodeUtil$UTF8Result   320     10240
> org.apache.lucene.index.TermBuffer      315     17640
> org.apache.lucene.util.UnicodeUtil$UTF16Result  315     12600
> org.apache.lucene.index.CompoundFileReader$FileEntry    280     8960
> org.apache.lucene.util.CloseableThreadLocal     260     8320
> org.apache.lucene.index.FreqProxTermsWriter$PostingList         256
> 12288
> org.apache.lucene.index.SegmentInfo     185     19240
> org.apache.lucene.index.SegmentReader$FieldsReaderLocal         140     5600
> org.apache.lucene.index.ReadOnlySegmentReader   105     12600
> org.apache.lucene.index.SegmentReader$Ref       105     2520
> org.apache.lucene.index.SegmentTermEnum         105     11760
> org.apache.lucene.search.FieldCacheImpl$Entry   105     3360
> org.apache.lucene.index.TermInfosReader$ThreadResources         70      2240
> org.apache.lucene.util.cache.SimpleLRUCache     70      1680
> org.apache.lucene.util.cache.SimpleLRUCache$1   70      6160
> org.apache.lucene.index.FieldsReader    50      4400
> org.apache.lucene.index.IndexFileDeleter$RefCount       42      1344
> org.apache.lucene.document.Document     41      1312
> org.apache.lucene.util.SimpleStringInterner$Entry       41      1640
> org.apache.lucene.index.FieldInfos      37      1480
> org.apache.lucene.index.CompoundFileReader      35      2520
> org.apache.lucene.index.SegmentReader   35      4200
> org.apache.lucene.index.SegmentReader$CoreReaders       35      4480
> org.apache.lucene.index.TermInfo[]      35      2963448
> org.apache.lucene.index.TermInfosReader         35      3360
> org.apache.lucene.index.Term[]  35      2963448
> org.apache.lucene.search.FieldCache$CreationPlaceholder         35      840
> org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexInput$Descriptor
>     35      2240
> org.apache.lucene.index.RawPostingList[]        30      8184
> org.apache.lucene.index.TermsHashPerField       24      3648
> org.apache.lucene.analysis.WhitespaceAnalyzer   16      512
> org.apache.lucene.document.Fieldable[]  13      416
> org.apache.lucene.index.DocFieldProcessorPerField       12      672
> org.apache.lucene.index.DocInverterPerField     12      768
> org.apache.lucene.index.FreqProxTermsWriterPerField     12      864
> org.apache.lucene.index.NormsWriterPerField     12      864
> org.apache.lucene.index.TermVectorsTermsWriterPerField  12      960
> org.apache.lucene.util.AttributeSource$State    12      384
> org.apache.lucene.util.Version  8       256
> org.apache.lucene.index.SegmentInfos    7       616
> org.apache.lucene.document.FieldSelectorResult  6       192
> org.apache.lucene.analysis.CharArraySet$UnmodifiableCharArraySet        5
>     160
> org.apache.lucene.analysis.tokenattributes.OffsetAttributeImpl  5       120
> org.apache.lucene.analysis.tokenattributes.TermAttributeImpl    5       160
> org.apache.lucene.analysis.tokenattributes.PositionIncrementAttributeImpl 4
>     96
> org.apache.lucene.index.BufferedDeletes         4       224
> org.apache.lucene.index.TermsHash       4       320
> org.apache.lucene.analysis.LowerCaseFilter      3       192
> org.apache.lucene.analysis.PerFieldAnalyzerWrapper      3       144
> org.apache.lucene.analysis.SimpleAnalyzer       3       96
> org.apache.lucene.analysis.StopFilter   3       264
> org.apache.lucene.analysis.ru.RussianAnalyzer   3       144
> org.apache.lucene.analysis.ru.RussianAnalyzer$SavedStreams      3       120
> org.apache.lucene.analysis.ru.RussianLetterTokenizer    3       288
> org.apache.lucene.analysis.ru.RussianStemFilter         3       216
> org.apache.lucene.analysis.ru.RussianStemmer    3       96
> org.apache.lucene.index.IndexReader[]   3       912
> org.apache.lucene.index.ReadOnlyDirectoryReader         3       384
> org.apache.lucene.index.SegmentReader[]         3       912
> org.apache.lucene.search.Sort   3       72
> org.apache.lucene.search.SortField      3       168
> org.apache.lucene.search.SortField[]    3       96
> org.apache.lucene.search.TermQuery      3       96
> org.apache.lucene.util.NamedThreadFactory       3       120
>
>
>
>
>
>
>
>
>
> org.apache.lucene.analysis.CharReader   2       80
> org.apache.lucene.index.ByteBlockPool   2       112
> org.apache.lucene.index.ConcurrentMergeScheduler        2       112
> org.apache.lucene.index.DocFieldProcessor       2       96
> org.apache.lucene.index.DocFieldProcessorPerField[]     2       432
> org.apache.lucene.index.DocInverter     2       80
> org.apache.lucene.index.DocumentsWriter         2       608
> org.apache.lucene.index.DocumentsWriter$ByteBlockAllocator      2       64
> org.apache.lucene.index.DocumentsWriter$DocWriter[]     2       208
> org.apache.lucene.index.DocumentsWriter$SkipDocWriter   2       64
> org.apache.lucene.index.DocumentsWriter$WaitQueue       2       112
> org.apache.lucene.index.DocumentsWriterThreadState[]    2       56
> org.apache.lucene.index.FreqProxTermsWriter     2       80
> org.apache.lucene.index.IndexFileDeleter        2       192
> org.apache.lucene.index.IndexFileDeleter$CommitPoint    2       176
> org.apache.lucene.index.IndexWriter     2       592
> org.apache.lucene.index.IndexWriter$MaxFieldLength      2       64
> org.apache.lucene.index.IndexWriter$ReaderPool  2       64
> org.apache.lucene.index.IntBlockPool    2       112
> org.apache.lucene.index.KeepOnlyLastCommitDeletionPolicy        2       32
> org.apache.lucene.index.LogByteSizeMergePolicy  2       112
> org.apache.lucene.index.NormsWriter     2       48
> org.apache.lucene.index.StoredFieldsWriter      2       128
> org.apache.lucene.index.StoredFieldsWriter$PerDoc[]     2       64
> org.apache.lucene.index.TermVectorsTermsWriter  2       176
> org.apache.lucene.index.TermVectorsTermsWriter$PerDoc[]         2       64
> org.apache.lucene.index.TermsHashPerThread      2       176
> org.apache.lucene.queryParser.QueryParser$Operator      2       64
> org.apache.lucene.search.BooleanClause  2       64
> org.apache.lucene.search.NumericRangeQuery      2       160
> org.apache.lucene.search.ParallelMultiSearcher  2       144
> org.apache.lucene.search.QueryWrapperFilter     2       48
> org.apache.lucene.search.ScoreDoc[]     2       48
> org.apache.lucene.search.Searchable[]   2       64
> org.apache.lucene.store.NIOFSDirectory  2       96
> org.apache.lucene.store.NativeFSLock    2       128
> org.apache.lucene.store.NativeFSLockFactory     2       80
> org.apache.lucene.analysis.CharArraySet         1       32
> org.apache.lucene.analysis.LowerCaseTokenizer   1       96
> org.apache.lucene.document.Field$Index$1        1       32
> org.apache.lucene.document.Field$Index$2        1       32
> org.apache.lucene.document.Field$Index$3        1       32
> org.apache.lucene.document.Field$Index$4        1       32
> org.apache.lucene.document.Field$Index$5        1       32
> org.apache.lucene.document.Field$Index[]        1       64
> org.apache.lucene.document.Field$Store$1        1       32
> org.apache.lucene.document.Field$Store$2        1       32
> org.apache.lucene.document.Field$Store[]        1       40
> org.apache.lucene.document.Field$TermVector$1   1       32
> org.apache.lucene.document.Field$TermVector$2   1       32
> org.apache.lucene.document.Field$TermVector$3   1       32
> org.apache.lucene.document.Field$TermVector$4   1       32
> org.apache.lucene.document.Field$TermVector$5   1       32
> org.apache.lucene.document.Field$TermVector[]   1       64
> org.apache.lucene.document.FieldSelectorResult[]        1       72
> org.apache.lucene.document.Field[]      1       24
> org.apache.lucene.index.ByteSliceReader         1       80
> org.apache.lucene.index.CharBlockPool   1       56
> org.apache.lucene.index.DocFieldProcessorPerThread      1       112
> org.apache.lucene.index.DocFieldProcessorPerThread$PerDoc[]     1       32
> org.apache.lucene.index.DocInverterPerThread    1       72
> org.apache.lucene.index.DocInverterPerThread$SingleTokenAttributeSource
>     1       64
> org.apache.lucene.index.DocumentsWriter$1       1       16
> org.apache.lucene.index.DocumentsWriter$DocState        1       72
> org.apache.lucene.index.DocumentsWriterThreadState      1       48
> org.apache.lucene.index.FieldInvertState        1       48
> org.apache.lucene.index.FieldsWriter    1       48
> org.apache.lucene.index.FreqProxTermsWriterPerThread    1       32
> org.apache.lucene.index.IndexFileNameFilter     1       32
> org.apache.lucene.index.NormsWriterPerThread    1       32
> org.apache.lucene.index.ReusableStringReader    1       48
> org.apache.lucene.index.SegmentWriteState       1       72
> org.apache.lucene.index.StoredFieldsWriter$PerDoc       1       56
> org.apache.lucene.index.StoredFieldsWriterPerThread     1       48
> org.apache.lucene.index.TermVectorsTermsWriterPerThread         1       72
> org.apache.lucene.queryParser.QueryParser$Operator[]    1       40
> org.apache.lucene.search.BooleanClause$Occur$1  1       32
> org.apache.lucene.search.BooleanClause$Occur$2  1       32
> org.apache.lucene.search.BooleanClause$Occur$3  1       32
> org.apache.lucene.search.BooleanClause$Occur[]  1       48
> org.apache.lucene.search.BooleanQuery   1       40
> org.apache.lucene.search.DefaultSimilarity      1       24
> org.apache.lucene.search.DocIdSet$1     1       24
> org.apache.lucene.search.DocIdSet$1$1   1       32
> org.apache.lucene.search.FieldCache$1   1       16
> org.apache.lucene.search.FieldCache$10  1       16
> org.apache.lucene.search.FieldCache$2   1       16
> org.apache.lucene.search.FieldCache$3   1       16
> org.apache.lucene.search.FieldCache$4   1       16
> org.apache.lucene.search.FieldCache$5   1       16
> org.apache.lucene.search.FieldCache$6   1       16
> org.apache.lucene.search.FieldCache$7   1       16
> org.apache.lucene.search.FieldCache$8   1       16
> org.apache.lucene.search.FieldCache$9   1       16
> org.apache.lucene.search.FieldCacheImpl         1       32
> org.apache.lucene.search.FieldCacheImpl$ByteCache       1       32
> org.apache.lucene.search.FieldCacheImpl$DoubleCache     1       32
> org.apache.lucene.search.FieldCacheImpl$FloatCache      1       32
> org.apache.lucene.search.FieldCacheImpl$IntCache        1       32
> org.apache.lucene.search.FieldCacheImpl$LongCache       1       32
> org.apache.lucene.search.FieldCacheImpl$ShortCache      1       32
> org.apache.lucene.search.FieldCacheImpl$StringCache     1       32
> org.apache.lucene.search.FieldCacheImpl$StringIndexCache        1       32
> org.apache.lucene.search.MultiTermQuery$1       1       32
> org.apache.lucene.search.MultiTermQuery$ConstantScoreBooleanQueryRewrite
>    1       16
> org.apache.lucene.search.MultiTermQuery$ConstantScoreFilterRewrite      1
>     16
> org.apache.lucene.search.MultiTermQuery$ScoringBooleanQueryRewrite      1
>     16
> org.apache.lucene.search.TopDocs        1       32
> org.apache.lucene.store.RAMFile         1       56
> org.apache.lucene.store.RAMOutputStream         1       72
> org.apache.lucene.util.AttributeSource$AttributeFactory$DefaultAttributeFactory
>         1       16
> org.apache.lucene.util.SimpleStringInterner     1       32
> org.apache.lucene.util.SimpleStringInterner$Entry[]     1       8216
> org.apache.lucene.util.UnicodeUtil$UTF8Result[]         1       40
> org.apache.lucene.util.Version[]        1       88
>
>
>        34562928 <mailto:=@SUM%28C1:C193%29>
>
>
>        32.96
>
>
>
>
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message