lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Marvin Humphrey <mar...@rectangular.com>
Subject Revised benchmarker results
Date Mon, 10 Apr 2006 05:16:22 GMT
Hello,

A few things have changed since the first benchmarker results were  
reported, and a fresh data set is mandated.  A serious bug in the  
LuceneIndexer was fixed.  Multiple runs are now triggered from within  
a single process.  The report now uses a truncated mean that discards  
both high and low outliers.  The OS on the machine running the  
benchmarks also changed from OS X 10.4.5 to 10.4.6, though that  
doesn't seem to have made a difference.

The following results were prepared using subversion repository  
revision 779.

RESULTS A: 'body' neither stored nor vectorized
======================================================================== 
===
configuration             truncated mean secs (6 reps)   max memory  
(1 rep)
------------------------------------------------------------------------ 
---
Lucene / JVM 1.4                  43.68                         79 MB
Lucene / JVM 1.5                  44.95                         93 MB
KinoSearch / Perl 5.8.8           75.67                         28 MB
KinoSearch / Perl 5.8.6           86.97                         30 MB
Plucene / Perl 5.8.6            2054.79                       skipped

RESULTS B: 'body' stored and vectorized
======================================================================== 
===
configuration             truncated mean secs (6 reps)   max memory  
(1 rep)
------------------------------------------------------------------------ 
---
Lucene / JVM 1.4                  71.96                        118 MB
Lucene / JVM 1.5                  73.81                        214 MB
KinoSearch / Perl 5.8.8           81.66                         30 MB
KinoSearch / Perl 5.8.6           93.25                         31 MB
Plucene / Perl 5.8.6            2067.92                       skipped


Marvin Humphrey
Rectangular Research
http://www.rectangular.com/


RAW DATA -- fixed LuceneIndexer
============================================================

slothbear:~/Desktop/ks/t/benchmarks marvin$ java -server -Xmx500M - 
XX:CompileThreshold=100 LuceneIndexer -reps 6
---------------------------------------------------
1   Secs: 56.46  Docs: 19043
2   Secs: 43.38  Docs: 19043
3   Secs: 43.40  Docs: 19043
4   Secs: 43.88  Docs: 19043
5   Secs: 44.08  Docs: 19043
6   Secs: 43.37  Docs: 19043
---------------------------------------------------
Lucene 1.9.1
JVM 1.4.2_09 (Apple Computer, Inc.)
Mac OS X 10.4.6 ppc
Mean: 45.76 secs
Truncated mean (4 kept, 2 discarded): 43.68 secs
---------------------------------------------------
slothbear:~/Desktop/ks/t/benchmarks marvin$ vim indexers/ 
LuceneIndexer.java
slothbear:~/Desktop/ks/t/benchmarks marvin$ javac -d . indexers/ 
LuceneIndexer.java
slothbear:~/Desktop/ks/t/benchmarks marvin$ java -server -Xmx500M - 
XX:CompileThreshold=100 LuceneIndexer -reps 6
---------------------------------------------------
1   Secs: 74.72  Docs: 19043
2   Secs: 70.72  Docs: 19043
3   Secs: 71.63  Docs: 19043
4   Secs: 71.51  Docs: 19043
5   Secs: 70.85  Docs: 19043
6   Secs: 73.84  Docs: 19043
---------------------------------------------------
Lucene 1.9.1
JVM 1.4.2_09 (Apple Computer, Inc.)
Mac OS X 10.4.6 ppc
Mean: 72.21 secs
Truncated mean (4 kept, 2 discarded): 71.96 secs
---------------------------------------------------
slothbear:~/Desktop/ks/t/benchmarks marvin$ vim indexers/ 
LuceneIndexer.java
slothbear:~/Desktop/ks/t/benchmarks marvin$ javac15 -d . indexers/ 
LuceneIndexer.java
Note: indexers/LuceneIndexer.java uses unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.
slothbear:~/Desktop/ks/t/benchmarks marvin$ java15 -server -Xmx500M - 
XX:CompileThreshold=100 LuceneIndexer -reps 6
---------------------------------------------------
1   Secs: 48.02  Docs: 19043
2   Secs: 44.81  Docs: 19043
3   Secs: 44.73  Docs: 19043
4   Secs: 44.66  Docs: 19043
5   Secs: 45.36  Docs: 19043
6   Secs: 44.89  Docs: 19043
---------------------------------------------------
Lucene 1.9.1
JVM 1.5.0_02 (Apple Computer, Inc.)
Mac OS X 10.4.6 ppc
Mean: 45.41 secs
Truncated mean (4 kept, 2 discarded): 44.95 secs
---------------------------------------------------
slothbear:~/Desktop/ks/t/benchmarks marvin$ vim indexers/ 
LuceneIndexer.java
slothbear:~/Desktop/ks/t/benchmarks marvin$ javac15 -d . indexers/ 
LuceneIndexer.java
Note: indexers/LuceneIndexer.java uses unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.
slothbear:~/Desktop/ks/t/benchmarks marvin$ java15 -server -Xmx500M - 
XX:CompileThreshold=100 LuceneIndexer -reps 6
---------------------------------------------------
1   Secs: 76.78  Docs: 19043
2   Secs: 72.94  Docs: 19043
3   Secs: 73.51  Docs: 19043
4   Secs: 72.56  Docs: 19043
5   Secs: 73.19  Docs: 19043
6   Secs: 75.59  Docs: 19043
---------------------------------------------------
Lucene 1.9.1
JVM 1.5.0_02 (Apple Computer, Inc.)
Mac OS X 10.4.6 ppc
Mean: 74.09 secs
Truncated mean (4 kept, 2 discarded): 73.81 secs
---------------------------------------------------
slothbear:~/Desktop/ks/t/benchmarks marvin$


RAW DATA -- buggy LuceneIndexer
=========================================================
slothbear:~/Desktop/ks/t/benchmarks marvin$ javac -d . indexers/ 
LuceneIndexer.java
slothbear:~/Desktop/ks/t/benchmarks marvin$ java -server -Xmx500M - 
XX:CompileThreshold=100 LuceneIndexer -reps 6
---------------------------------------------------
1   Secs: 50.63  Docs: 19043
2   Secs: 49.69  Docs: 19043
3   Secs: 49.19  Docs: 19043
4   Secs: 49.42  Docs: 19043
5   Secs: 49.47  Docs: 19043
6   Secs: 49.94  Docs: 19043
---------------------------------------------------
Lucene 1.9.1
JVM 1.4.2_09 (Apple Computer, Inc.)
Mac OS X 10.4.6 ppc
Mean: 49.72 secs
Truncated mean (4 kept, 2 discarded): 49.63 secs
---------------------------------------------------
slothbear:~/Desktop/ks/t/benchmarks marvin$ vim indexers/ 
LuceneIndexer.java
slothbear:~/Desktop/ks/t/benchmarks marvin$ javac -d . indexers/ 
LuceneIndexer.java
slothbear:~/Desktop/ks/t/benchmarks marvin$ java -server -Xmx500M - 
XX:CompileThreshold=100 LuceneIndexer -reps 6
---------------------------------------------------
1   Secs: 87.15  Docs: 19043
2   Secs: 85.20  Docs: 19043
3   Secs: 84.97  Docs: 19043
4   Secs: 83.87  Docs: 19043
5   Secs: 83.39  Docs: 19043
6   Secs: 84.87  Docs: 19043
---------------------------------------------------
Lucene 1.9.1
JVM 1.4.2_09 (Apple Computer, Inc.)
Mac OS X 10.4.6 ppc
Mean: 84.91 secs
Truncated mean (4 kept, 2 discarded): 84.73 secs
---------------------------------------------------
slothbear:~/Desktop/ks/t/benchmarks marvin$ vim indexers/ 
LuceneIndexer.java

slothbear:~/Desktop/ks/t/benchmarks marvin$ javac15 -d . indexers/ 
LuceneIndexer.java  6
Note: indexers/LuceneIndexer.java uses unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.
slothbear:~/Desktop/ks/t/benchmarks marvin$ java15 -server -Xmx500M - 
XX:CompileThreshold=100 LuceneIndexer -reps 6
---------------------------------------------------
1   Secs: 51.33  Docs: 19043
2   Secs: 50.74  Docs: 19043
3   Secs: 50.89  Docs: 19043
4   Secs: 50.65  Docs: 19043
5   Secs: 51.02  Docs: 19043
6   Secs: 51.08  Docs: 19043
---------------------------------------------------
Lucene 1.9.1
JVM 1.5.0_02 (Apple Computer, Inc.)
Mac OS X 10.4.6 ppc
Mean: 50.95 secs
Truncated mean (4 kept, 2 discarded): 50.93 secs
---------------------------------------------------
slothbear:~/Desktop/ks/t/benchmarks marvin$ vim indexers/ 
LuceneIndexer.java
slothbear:~/Desktop/ks/t/benchmarks marvin$ javac15 -d . indexers/ 
LuceneIndexer.java
Note: indexers/LuceneIndexer.java uses unchecked or unsafe operations.
Note: Recompile with -Xlint:unchecked for details.
slothbear:~/Desktop/ks/t/benchmarks marvin$ java15 -server -Xmx500M - 
XX:CompileThreshold=100 LuceneIndexer -reps 6
---------------------------------------------------
1   Secs: 90.84  Docs: 19043
2   Secs: 89.21  Docs: 19043
3   Secs: 89.32  Docs: 19043
4   Secs: 88.34  Docs: 19043
5   Secs: 88.79  Docs: 19043
6   Secs: 88.50  Docs: 19043
---------------------------------------------------
Lucene 1.9.1
JVM 1.5.0_02 (Apple Computer, Inc.)
Mac OS X 10.4.6 ppc
Mean: 89.17 secs
Truncated mean (4 kept, 2 discarded): 88.96 secs
---------------------------------------------------
slothbear:~/Desktop/ks/t/benchmarks marvin$

RAW DATA -- KinoSearch
=========================================================
slothbear:~/Desktop/ks588/t/benchmarks marvin$ perl588 -Mblib  
indexers/kinosearch_indexer.plx --reps=6
------------------------------------------------------------
1    Secs: 75.69  Docs: 19043
2    Secs: 81.10  Docs: 19043
3    Secs: 81.43  Docs: 19043
4    Secs: 84.72  Docs: 19043
5    Secs: 81.50  Docs: 19043
6    Secs: 82.59  Docs: 19043
------------------------------------------------------------
KinoSearch 0.09_03
Perl 5.8.8
Thread support: no
Darwin 8.6.0 Power Macintosh
Mean: 81.17 secs
Truncated mean (4 kept, 2 discarded): 81.66 secs
------------------------------------------------------------
slothbear:~/Desktop/ks588/t/benchmarks marvin$ vim indexers/ 
kinosearch_indexer.plx
slothbear:~/Desktop/ks588/t/benchmarks marvin$ perl588 -Mblib  
indexers/kinosearch_indexer.plx --reps=6
------------------------------------------------------------
1    Secs: 73.03  Docs: 19043
2    Secs: 74.42  Docs: 19043
3    Secs: 76.44  Docs: 19043
4    Secs: 75.66  Docs: 19043
5    Secs: 78.89  Docs: 19043
6    Secs: 76.16  Docs: 19043
------------------------------------------------------------
KinoSearch 0.09_03
Perl 5.8.8
Thread support: no
Darwin 8.6.0 Power Macintosh
Mean: 75.77 secs
Truncated mean (4 kept, 2 discarded): 75.67 secs
------------------------------------------------------------
slothbear:~/Desktop/ks588/t/benchmarks marvin$ cd ~/Desktop/ks/t/ 
benchmarks/
slothbear:~/Desktop/ks/t/benchmarks marvin$ perl -Mblib indexers/ 
kinosearch_indexer.plx --reps=6
------------------------------------------------------------
1    Secs: 88.10  Docs: 19043
2    Secs: 93.33  Docs: 19043
3    Secs: 97.28  Docs: 19043
4    Secs: 94.39  Docs: 19043
5    Secs: 94.04  Docs: 19043
6    Secs: 91.22  Docs: 19043
------------------------------------------------------------
KinoSearch 0.09_03
Perl 5.8.6
Thread support: yes
Darwin 8.6.0 Power Macintosh
Mean: 93.06 secs
Truncated mean (4 kept, 2 discarded): 93.25 secs
------------------------------------------------------------
slothbear:~/Desktop/ks/t/benchmarks marvin$ vim indexers/ 
kinosearch_indexer.plx
slothbear:~/Desktop/ks/t/benchmarks marvin$ perl -Mblib indexers/ 
kinosearch_indexer.plx --reps=6
------------------------------------------------------------
1    Secs: 82.37  Docs: 19043
2    Secs: 86.87  Docs: 19043
3    Secs: 87.22  Docs: 19043
4    Secs: 93.85  Docs: 19043
5    Secs: 87.48  Docs: 19043
6    Secs: 86.33  Docs: 19043
------------------------------------------------------------
KinoSearch 0.09_03
Perl 5.8.6
Thread support: yes
Darwin 8.6.0 Power Macintosh
Mean: 87.35 secs
Truncated mean (4 kept, 2 discarded): 86.97 secs
------------------------------------------------------------
slothbear:~/Desktop/ks/t/benchmarks marvin$


RAW DATA - Plucene
==================

slothbear:~/Desktop/ks/t/benchmarks marvin$ perl indexers/ 
plucene_indexer.plx --reps=6
------------------------------------------------------------
1    Secs: 1974.26  Docs: 19043
2    Secs: 2024.60  Docs: 19043
3    Secs: 2060.43  Docs: 19043
4    Secs: 2064.22  Docs: 19043
5    Secs: 2072.20  Docs: 19043
6    Secs: 2069.92  Docs: 19043
------------------------------------------------------------
Plucene 1.24
Perl 5.8.6
Thread support: yes
Darwin 8.6.0 Power Macintosh
Mean: 2044.27 secs
Truncated mean (4 kept, 2 discarded): 2054.79 secs
------------------------------------------------------------
slothbear:~/Desktop/ks/t/benchmarks marvin$ vim indexers/ 
plucene_indexer.plx
slothbear:~/Desktop/ks/t/benchmarks marvin$ perl indexers/ 
plucene_indexer.plx --reps=6
------------------------------------------------------------
1    Secs: 1981.76  Docs: 19043
2    Secs: 2037.07  Docs: 19043
3    Secs: 2072.37  Docs: 19043
4    Secs: 2082.80  Docs: 19043
5    Secs: 2079.45  Docs: 19043
6    Secs: 2086.64  Docs: 19043
------------------------------------------------------------
Plucene 1.24
Perl 5.8.6
Thread support: yes
Darwin 8.6.0 Power Macintosh
Mean: 2056.68 secs
Truncated mean (4 kept, 2 discarded): 2067.92 secs
------------------------------------------------------------
slothbear:~/Desktop/ks/t/benchmarks marvin$









---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


Mime
View raw message