lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jamie <ja...@stimulussoft.com>
Subject Re: Lucene 3.0 Search Performance Stats
Date Mon, 22 Mar 2010 12:35:08 GMT
Hi Everyone

The stats I sent through earlier were erroneous due to fact the date 
range query selected fewer records than stated.

The correct stats are:

Lucene 3.0 Stats:

Search conducted using Lucene's Realtime search feature 
(writer.getReader() for each search)
Analyzer: Russian Analyzer
Total Docs: 26.04M (emails data - all attachments, body content indexed)
Index Size: 37G
Query: body: test AND  date: [200901010101 to 201003220225] with 
descending sort on date.
Lucene Mem Usage: 32 MB (when sorted on date)
Search Speed: 0.48 (unsorted)
Search Speed: 0.49 (sorted on YYYYMMDDHHSS date)

Hardware / Software:

Index stored on 4 SAS HDD hitachi RAID 10
16G RAM
2x Xeon 4 core 2.4Gz
OS FreeBSD 7.2
Filesystem UFS2 gjournal

 From Yourkit, memory usage is as follows:

Name 	Number of Objects 	Shallow Size
org.apache.lucene.index.TermInfo 	370610 	14824400
org.apache.lucene.index.Term 	370505 	11856160
com.stimulus.archiva.search.LuceneResult 	20000 	960000
org.apache.lucene.search.FieldDoc 	10000 	320000
org.apache.lucene.search.ScoreDoc 	10000 	240000
org.apache.lucene.index.FieldInfo 	1027 	41080
org.apache.lucene.index.SegmentReader$Norm 	840 	67200
org.apache.lucene.document.Field 	578 	36992
org.apache.lucene.store.NIOFSDirectory$NIOFSIndexInput 	415 	49800
org.apache.lucene.index.CompoundFileReader$CSIndexInput 	380 	36480
org.apache.lucene.util.UnicodeUtil$UTF8Result 	320 	10240
org.apache.lucene.index.TermBuffer 	315 	17640
org.apache.lucene.util.UnicodeUtil$UTF16Result 	315 	12600
org.apache.lucene.index.CompoundFileReader$FileEntry 	280 	8960
org.apache.lucene.util.CloseableThreadLocal 	260 	8320
org.apache.lucene.index.FreqProxTermsWriter$PostingList 	256 	12288
org.apache.lucene.index.SegmentInfo 	185 	19240
org.apache.lucene.index.SegmentReader$FieldsReaderLocal 	140 	5600
org.apache.lucene.index.ReadOnlySegmentReader 	105 	12600
org.apache.lucene.index.SegmentReader$Ref 	105 	2520
org.apache.lucene.index.SegmentTermEnum 	105 	11760
org.apache.lucene.search.FieldCacheImpl$Entry 	105 	3360
org.apache.lucene.index.TermInfosReader$ThreadResources 	70 	2240
org.apache.lucene.util.cache.SimpleLRUCache 	70 	1680
org.apache.lucene.util.cache.SimpleLRUCache$1 	70 	6160
org.apache.lucene.index.FieldsReader 	50 	4400
org.apache.lucene.index.IndexFileDeleter$RefCount 	42 	1344
org.apache.lucene.document.Document 	41 	1312
org.apache.lucene.util.SimpleStringInterner$Entry 	41 	1640
org.apache.lucene.index.FieldInfos 	37 	1480
org.apache.lucene.index.CompoundFileReader 	35 	2520
org.apache.lucene.index.SegmentReader 	35 	4200
org.apache.lucene.index.SegmentReader$CoreReaders 	35 	4480
org.apache.lucene.index.TermInfo[] 	35 	2963448
org.apache.lucene.index.TermInfosReader 	35 	3360
org.apache.lucene.index.Term[] 	35 	2963448
org.apache.lucene.search.FieldCache$CreationPlaceholder 	35 	840
org.apache.lucene.store.SimpleFSDirectory$SimpleFSIndexInput$Descriptor 	35 
	2240
org.apache.lucene.index.RawPostingList[] 	30 	8184
org.apache.lucene.index.TermsHashPerField 	24 	3648
org.apache.lucene.analysis.WhitespaceAnalyzer 	16 	512
org.apache.lucene.document.Fieldable[] 	13 	416
org.apache.lucene.index.DocFieldProcessorPerField 	12 	672
org.apache.lucene.index.DocInverterPerField 	12 	768
org.apache.lucene.index.FreqProxTermsWriterPerField 	12 	864
org.apache.lucene.index.NormsWriterPerField 	12 	864
org.apache.lucene.index.TermVectorsTermsWriterPerField 	12 	960
org.apache.lucene.util.AttributeSource$State 	12 	384
org.apache.lucene.util.Version 	8 	256
org.apache.lucene.index.SegmentInfos 	7 	616
org.apache.lucene.document.FieldSelectorResult 	6 	192
org.apache.lucene.analysis.CharArraySet$UnmodifiableCharArraySet 	5 	160
org.apache.lucene.analysis.tokenattributes.OffsetAttributeImpl 	5 	120
org.apache.lucene.analysis.tokenattributes.TermAttributeImpl 	5 	160
org.apache.lucene.analysis.tokenattributes.PositionIncrementAttributeImpl 
4 	96
org.apache.lucene.index.BufferedDeletes 	4 	224
org.apache.lucene.index.TermsHash 	4 	320
org.apache.lucene.analysis.LowerCaseFilter 	3 	192
org.apache.lucene.analysis.PerFieldAnalyzerWrapper 	3 	144
org.apache.lucene.analysis.SimpleAnalyzer 	3 	96
org.apache.lucene.analysis.StopFilter 	3 	264
org.apache.lucene.analysis.ru.RussianAnalyzer 	3 	144
org.apache.lucene.analysis.ru.RussianAnalyzer$SavedStreams 	3 	120
org.apache.lucene.analysis.ru.RussianLetterTokenizer 	3 	288
org.apache.lucene.analysis.ru.RussianStemFilter 	3 	216
org.apache.lucene.analysis.ru.RussianStemmer 	3 	96
org.apache.lucene.index.IndexReader[] 	3 	912
org.apache.lucene.index.ReadOnlyDirectoryReader 	3 	384
org.apache.lucene.index.SegmentReader[] 	3 	912
org.apache.lucene.search.Sort 	3 	72
org.apache.lucene.search.SortField 	3 	168
org.apache.lucene.search.SortField[] 	3 	96
org.apache.lucene.search.TermQuery 	3 	96
org.apache.lucene.util.NamedThreadFactory 	3 	120

	
	

	
	

	
	
org.apache.lucene.analysis.CharReader 	2 	80
org.apache.lucene.index.ByteBlockPool 	2 	112
org.apache.lucene.index.ConcurrentMergeScheduler 	2 	112
org.apache.lucene.index.DocFieldProcessor 	2 	96
org.apache.lucene.index.DocFieldProcessorPerField[] 	2 	432
org.apache.lucene.index.DocInverter 	2 	80
org.apache.lucene.index.DocumentsWriter 	2 	608
org.apache.lucene.index.DocumentsWriter$ByteBlockAllocator 	2 	64
org.apache.lucene.index.DocumentsWriter$DocWriter[] 	2 	208
org.apache.lucene.index.DocumentsWriter$SkipDocWriter 	2 	64
org.apache.lucene.index.DocumentsWriter$WaitQueue 	2 	112
org.apache.lucene.index.DocumentsWriterThreadState[] 	2 	56
org.apache.lucene.index.FreqProxTermsWriter 	2 	80
org.apache.lucene.index.IndexFileDeleter 	2 	192
org.apache.lucene.index.IndexFileDeleter$CommitPoint 	2 	176
org.apache.lucene.index.IndexWriter 	2 	592
org.apache.lucene.index.IndexWriter$MaxFieldLength 	2 	64
org.apache.lucene.index.IndexWriter$ReaderPool 	2 	64
org.apache.lucene.index.IntBlockPool 	2 	112
org.apache.lucene.index.KeepOnlyLastCommitDeletionPolicy 	2 	32
org.apache.lucene.index.LogByteSizeMergePolicy 	2 	112
org.apache.lucene.index.NormsWriter 	2 	48
org.apache.lucene.index.StoredFieldsWriter 	2 	128
org.apache.lucene.index.StoredFieldsWriter$PerDoc[] 	2 	64
org.apache.lucene.index.TermVectorsTermsWriter 	2 	176
org.apache.lucene.index.TermVectorsTermsWriter$PerDoc[] 	2 	64
org.apache.lucene.index.TermsHashPerThread 	2 	176
org.apache.lucene.queryParser.QueryParser$Operator 	2 	64
org.apache.lucene.search.BooleanClause 	2 	64
org.apache.lucene.search.NumericRangeQuery 	2 	160
org.apache.lucene.search.ParallelMultiSearcher 	2 	144
org.apache.lucene.search.QueryWrapperFilter 	2 	48
org.apache.lucene.search.ScoreDoc[] 	2 	48
org.apache.lucene.search.Searchable[] 	2 	64
org.apache.lucene.store.NIOFSDirectory 	2 	96
org.apache.lucene.store.NativeFSLock 	2 	128
org.apache.lucene.store.NativeFSLockFactory 	2 	80
org.apache.lucene.analysis.CharArraySet 	1 	32
org.apache.lucene.analysis.LowerCaseTokenizer 	1 	96
org.apache.lucene.document.Field$Index$1 	1 	32
org.apache.lucene.document.Field$Index$2 	1 	32
org.apache.lucene.document.Field$Index$3 	1 	32
org.apache.lucene.document.Field$Index$4 	1 	32
org.apache.lucene.document.Field$Index$5 	1 	32
org.apache.lucene.document.Field$Index[] 	1 	64
org.apache.lucene.document.Field$Store$1 	1 	32
org.apache.lucene.document.Field$Store$2 	1 	32
org.apache.lucene.document.Field$Store[] 	1 	40
org.apache.lucene.document.Field$TermVector$1 	1 	32
org.apache.lucene.document.Field$TermVector$2 	1 	32
org.apache.lucene.document.Field$TermVector$3 	1 	32
org.apache.lucene.document.Field$TermVector$4 	1 	32
org.apache.lucene.document.Field$TermVector$5 	1 	32
org.apache.lucene.document.Field$TermVector[] 	1 	64
org.apache.lucene.document.FieldSelectorResult[] 	1 	72
org.apache.lucene.document.Field[] 	1 	24
org.apache.lucene.index.ByteSliceReader 	1 	80
org.apache.lucene.index.CharBlockPool 	1 	56
org.apache.lucene.index.DocFieldProcessorPerThread 	1 	112
org.apache.lucene.index.DocFieldProcessorPerThread$PerDoc[] 	1 	32
org.apache.lucene.index.DocInverterPerThread 	1 	72
org.apache.lucene.index.DocInverterPerThread$SingleTokenAttributeSource 	1 
	64
org.apache.lucene.index.DocumentsWriter$1 	1 	16
org.apache.lucene.index.DocumentsWriter$DocState 	1 	72
org.apache.lucene.index.DocumentsWriterThreadState 	1 	48
org.apache.lucene.index.FieldInvertState 	1 	48
org.apache.lucene.index.FieldsWriter 	1 	48
org.apache.lucene.index.FreqProxTermsWriterPerThread 	1 	32
org.apache.lucene.index.IndexFileNameFilter 	1 	32
org.apache.lucene.index.NormsWriterPerThread 	1 	32
org.apache.lucene.index.ReusableStringReader 	1 	48
org.apache.lucene.index.SegmentWriteState 	1 	72
org.apache.lucene.index.StoredFieldsWriter$PerDoc 	1 	56
org.apache.lucene.index.StoredFieldsWriterPerThread 	1 	48
org.apache.lucene.index.TermVectorsTermsWriterPerThread 	1 	72
org.apache.lucene.queryParser.QueryParser$Operator[] 	1 	40
org.apache.lucene.search.BooleanClause$Occur$1 	1 	32
org.apache.lucene.search.BooleanClause$Occur$2 	1 	32
org.apache.lucene.search.BooleanClause$Occur$3 	1 	32
org.apache.lucene.search.BooleanClause$Occur[] 	1 	48
org.apache.lucene.search.BooleanQuery 	1 	40
org.apache.lucene.search.DefaultSimilarity 	1 	24
org.apache.lucene.search.DocIdSet$1 	1 	24
org.apache.lucene.search.DocIdSet$1$1 	1 	32
org.apache.lucene.search.FieldCache$1 	1 	16
org.apache.lucene.search.FieldCache$10 	1 	16
org.apache.lucene.search.FieldCache$2 	1 	16
org.apache.lucene.search.FieldCache$3 	1 	16
org.apache.lucene.search.FieldCache$4 	1 	16
org.apache.lucene.search.FieldCache$5 	1 	16
org.apache.lucene.search.FieldCache$6 	1 	16
org.apache.lucene.search.FieldCache$7 	1 	16
org.apache.lucene.search.FieldCache$8 	1 	16
org.apache.lucene.search.FieldCache$9 	1 	16
org.apache.lucene.search.FieldCacheImpl 	1 	32
org.apache.lucene.search.FieldCacheImpl$ByteCache 	1 	32
org.apache.lucene.search.FieldCacheImpl$DoubleCache 	1 	32
org.apache.lucene.search.FieldCacheImpl$FloatCache 	1 	32
org.apache.lucene.search.FieldCacheImpl$IntCache 	1 	32
org.apache.lucene.search.FieldCacheImpl$LongCache 	1 	32
org.apache.lucene.search.FieldCacheImpl$ShortCache 	1 	32
org.apache.lucene.search.FieldCacheImpl$StringCache 	1 	32
org.apache.lucene.search.FieldCacheImpl$StringIndexCache 	1 	32
org.apache.lucene.search.MultiTermQuery$1 	1 	32
org.apache.lucene.search.MultiTermQuery$ConstantScoreBooleanQueryRewrite 	1 
	16
org.apache.lucene.search.MultiTermQuery$ConstantScoreFilterRewrite 	1 	16
org.apache.lucene.search.MultiTermQuery$ScoringBooleanQueryRewrite 	1 	16
org.apache.lucene.search.TopDocs 	1 	32
org.apache.lucene.store.RAMFile 	1 	56
org.apache.lucene.store.RAMOutputStream 	1 	72
org.apache.lucene.util.AttributeSource$AttributeFactory$DefaultAttributeFactory 
	1 	16
org.apache.lucene.util.SimpleStringInterner 	1 	32
org.apache.lucene.util.SimpleStringInterner$Entry[] 	1 	8216
org.apache.lucene.util.UnicodeUtil$UTF8Result[] 	1 	40
org.apache.lucene.util.Version[] 	1 	88

	
	34562928 <mailto:=@SUM%28C1:C193%29>

	
	32.96






Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message