lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Eger, Patrick" <>
Subject FW: Binary indexing / query efficiency
Date Wed, 15 Apr 2009 17:37:59 GMT
Resending, I think this got dropped by the list for some reason


Hi, was recently looking to incorporate Lucene for a simple
"parametric"/"faceted" type search.  The documents are very small,
roughly 15 fields of short length (5-15 characters, generally strings
and padded integers). When profiling query performance of our
application, which inserts 1 million documents then
 1) filters on 1-3 fields with simple boolean/term matches
 2) stores these docids in a BitSet
 3) calls IndexSearcher.doc() to retrieve all matching documents (all
fields, 100 - 1,000,000 results per call)

It turns out that 98% of the query time was spent not actually doing the
query, but within the IndexSearcher.doc() call.

My first question is, is there any way to more efficiently get
(all/most) of the fields for a set of documents, other than iterating
and calling doc()?

Additionally, is there any way (or planned feature) to index *binary*
data? Using a profiler, I have determined that String decoding is a
significant performance limiter for my use-case:

90% of the application time is spent in this method:
org.apache.lucene.index.FieldsReader.addField(Document, FieldInfo,
boolean, boolean, boolean)

46% of the application time is spent decoding strings (half of the above
addField() time):
	java.lang.String.<init>(byte[], int, int, String)
		java.lang.StringCoding.decode(String, byte[], int, int)
java.lang.StringCoding$StringDecoder.decode(byte[], int, int)

(YJP profiler output available if needed)

String.intern() was my top hot spot, but my patch was accepted and fixed
this: I'm not
familiar enough with the lucene codebase to figure out the above though,
so thought I would ask.

//ideally i'd be able to do add a binary field as such:
doc.add(new Field("f1",new

//then query like:
Query q = new TermQuery(new Term("f1",byte[]{1,2,3,4})),...);

Which would allow me to avoid the Integer -> String -> Padded String ->
String -> Integer coding/decoding to index an integer, and avoid Object
-> String -> Object conversion (which per above is quite expensive). 

Thanks for any help!



To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message