lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Anshum <anshum.gu...@naukri.com>
Subject Re: Lucene performance: benchmarktemplate.xml
Date Fri, 18 Apr 2008 04:01:13 GMT
Hi Glenn,

I am not too clear about it, but isn't there a limit to the memory
consumption specified for the JVM? The limit being 1.3Gigs of resident
and 2 Gigs of memory in all? You just mentioned the Memory consumption:
-Xms4000m -Xmx6000m.
Could someone please help me with the same.

--
Anshum

On Wed, 2008-04-16 at 23:21 +0530, Glen Newton wrote:
> On 16/04/2008, Michael McCandless <lucene@mikemccandless.com> wrote:
> > These are great results!  Thanks for posting.
> 
> Thanks!
> 
> >
> >  I'd be curious if you'd get better indexing throughput by using a single
> > IndexWriter, fed by all 8 indexing threads, with an 8X bigger RAM buffer,
> > instead of 8 IndexWriters that merge in the end.
> 
> While I am new to this list, I have been trying different
> combinations/configurations for this over the last 18 months. To
> answer your questions: no, not on our multi-core machine. But I
> haven't  tried your suggested scenario with >= v2.2, so it is possible
> that it might be better.
> 
> >  How long does that final merge take now?
> 
> I don't have that timed. I will alter the app to record this.
> 
> >  Also, 64 threads doing document construction seems too high?  You may be
> > losing some performance to the cost of thread context switching.
> 
> I did performance tests on 8,16,24,32,48,64,72,96,128 threads. 64 was
> the sweet spot, for this particular configuration.
> 
> >  Did you use autoCommit=false?  I think it should help since you have so
> > many stored fields and some term vectors.
> 
> Damn, I am using the defaults (autoCommit=true). I will re-run and
> post the results!
> 
> Thanks,  :-)
> 
> -glen
> 
> >  Mike
> >
> >
> >  Glen Newton wrote:
> >
> > > Cass,
> > > Thanks for converting it. I've posted it to my blog:
> > >
> > http://zzzoot.blogspot.com/2008/04/lucene-indexing-performance-benchmarks.html
> > >
> > > Sorry for the XML tags: I guess I followed the instructions on the
> > > Lucene performance benchmarks page to literally ("Post these figures
> > > to the lucene-user mailing list using this template.").
> > >
> > > Sorry if it hurt your eyes!  :-)
> > >
> > > -Glen
> > >
> > > On 15/04/2008, Cass Costello <casspc@gmail.com> wrote:
> > >
> > > > I just did that so I could read it. :)  I'll leave it up until Glen
> > resends
> > > >  or posts it somewhere...
> > > >  http://www.casscostello.com/?page_id=28
> > > >
> > > >
> > > >
> > > >
> > > >  On Tue, Apr 15, 2008 at 5:18 PM, Ian Holsman <lists@holsman.net>
wrote:
> > > >
> > > >
> > > > > Hi Glen.
> > > > > can you resend this in plain text?
> > > > > or put the HTML up on a server somewhere and point to it with a brief
> > > > > summary in the post?
> > > > > I'd love to look and read it, all those tags are making me go blind.
> > > > >
> > > > >
> > > > > Glen Newton wrote:
> > > > >
> > > > >
> > > > > > <benchmark>
> > > > > >  <ul>
> > > > > >  <p>
> > > > > >  <b>Hardware Environment</b><br/>
> > > > > >  <li><i>Dedicated machine for indexing</i>:
yes</li>
> > > > > >  <li><i>CPU</i>: Dual processor dual core
Xeon CPU 3.00GHz;
> > > > > > hyperthreading ON for 8 virtual cores</li>
> > > > > >  <li><i>RAM</i>: 8GB</li>
> > > > > >  <li><i>Drive configuration</i>: Dell EMC
AX150 storage array fibre
> > > > > > channel</li>
> > > > > >  </p>
> > > > > >  <p>
> > > > > >  <b>Software environment</b><br/>
> > > > > >  <li><i>Lucene Version</i>: 2.3.1</li>
> > > > > >  <li><i>Java Version</i>:  Java(TM) SE Runtime
Environment (build
> > > > > > 1.6.0_02-b05)</li>
> > > > > >  <li><i>Java VM</i>: Java HotSpot(TM) 64-Bit
Server VM (build
> > > > > > 1.6.0_02-b05, mixed mode)</li>
> > > > > >  <li><i>OS Version</i>: Linux OpenSUSE 10.2
(64-bit X86-64)</li>
> > > > > >  <li><i>Location of index</i>: Filesystem,
on attached storage</li>
> > > > > >  </p>
> > > > > >  <p>
> > > > > >  <b>Lucene indexing variables</b><br/>
> > > > > >  <li><i>Number of source documents</i>: 6,404,464</li>
> > > > > >  <li><i>Total filesize of source documents</i>:
141GB; Note that
> > this
> > > > > > is only the full-text: the metadata (title, author(s), abstract,
> > > > > > keywords, journal name) are in addition to this</li>
> > > > > >  <li><i>Average filesize of source documents</i>:
> > > > > > 22KB + metadata (see above)</li>
> > > > > >  <li><i>Source documents storage location</i>:
Where are the
> > documents
> > > > > > being indexed located?
> > > > > >  Filesystem</li>
> > > > > >  <li><i>File type of source documents</i>:
text (PDFs converted to
> > > > > > text then gzipped)</li>
> > > > > >  <li><i>Parser(s) used, if any</i>: None,
but files GZIPed & had to
> > > > > > be un-gziped by Java application which also did indexing</li>
> > > > > >  <li><i>Analyzer(s) used</i>: StandardAnalyzer</li>
> > > > > >  <li><i>Number of fields per document</i>:
24</li>
> > > > > >  <li><i>Type of fields</i>: all text; 20 stored;
3 of indexed
> > > > > > tokenized with term vector (full-text [not stored], title,
> > abstract);
> > > > > > 10 stored with no parsing; </li>
> > > > > >  <li><i>Index persistence</i>: FSDirectory</li>
> > > > > >  <li><i>Index size</i>: 83GB</li>
> > > > > >  <li><i>Number of terms</i>: 143,298,010</li>
> > > > > >  </p>
> > > > > >  <p>
> > > > > >  <b>Figures</b><br/>
> > > > > >  <li><i>Time taken (in ms/s as an average of at
least 3 indexing
> > > > > > runs)</i>: 20.5 hours</li>
> > > > > >  <li><i>Time taken / 1000 docs indexed</i>:
11.5 seconds </li>
> > > > > >  <li><i>Memory consumption</i>:  -Xms4000m
 -Xmx6000m</li>
> > > > > >  <li><i>Query speed</i>: average time a query
takes, type
> > > > > >  of queries (e.g. simple one-term query, phrase query),
> > > > > >  not measuring any overhead outside Lucene</li>
> > > > > >  </p>
> > > > > >  <p>
> > > > > >  <b>Notes</b><br/>
> > > > > >  <li><i>Notes</i>:
> > > > > >      <ul>
> > > > > >        <li>
> > > > > >          These are journal articles, so the additional fields
> > besides
> > > > > > the
> > > > > > full-text are bibliographic metadata, such as title, authors,
> > > > > > abstract, keywords, journal name, volume, issue, start page,
year.
> > > > > >        </li>
> > > > > >        <li>Java command line directives: -XX:+AggressiveOpts
> > > > > > -XX:+ScavengeBeforeFullGC -XX:-UseParallelGC   -server  -Xms4000m
> > > > > > -Xmx6000m
> > > > > >        </li>
> > > > > >        <li>Highly multithreaded & pipelined architecture
using
> > > > > > java.util.concurrent.ThreadPoolExecutor
> > > > > >        </li>
> > > > > >        <li>File system file reading and Un-gzip performed
> > multithreaded
> > > > > >        </li>
> > > > > >        <li>Eight separate parallel IndexWriters are fed
by the
> > pipeline
> > > > > > (creation of Document objects occurs in parallel with 64 threads),
> > > > > > merged at end into single index. Each parallel index had slightly
> > > > > > different RAM_BUFFER_SIZE_MB (64, 67, 70, 73, 76, 79, 83, 85
MB
> > > > > > respectively), so that flushing wouldn't all happen at the same
> > time.
> > > > > >        </li>
> > > > > >        <li>
> > > > > >          Contact: glen DOT newton AT nrc-cnrc DOT gc DOT ca
> > > > > >        </li>
> > > > > >
> > > > > >      </ul>
> > > > > > </li>
> > > > > >  </p>
> > > > > >  </ul>
> > > > > > </benchmark>
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > ---------------------------------------------------------------------
> > > > > To unsubscribe, e-mail:
> > java-user-unsubscribe@lucene.apache.org
> > > > > For additional commands, e-mail:
> > java-user-help@lucene.apache.org
> > > > >
> > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > >  Lego timeline:
> > > >
> > http://cache.gizmodo.com/assets/resources/2008/01/lego-brick4-timeline.jpg
> > > >
> > > >
> > >
> > >
> > > --
> > >
> > > -
> > >
> > >
> > ---------------------------------------------------------------------
> > > To unsubscribe, e-mail:
> > java-user-unsubscribe@lucene.apache.org
> > > For additional commands, e-mail:
> > java-user-help@lucene.apache.org
> > >
> > >
> >
> >
> > ---------------------------------------------------------------------
> >  To unsubscribe, e-mail:
> > java-user-unsubscribe@lucene.apache.org
> >  For additional commands, e-mail:
> > java-user-help@lucene.apache.org
> >
> >
> 
> 
> --
> 
> -
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
> 


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message