lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From <>
Subject RE: Design Consideration for lucene index
Date Fri, 06 Oct 2006 20:30:15 GMT
Thanks Erick for your suggestions. I am sure that I might be thinking
with the DB cap. Let me look into your suggestions for the question #1.
I will get back to you if I need more inputs from you.

-----Original Message-----
From: Erick Erickson [] 
Sent: Friday, October 06, 2006 12:34 PM
Subject: Re: Design Consideration for lucene index

If you're *sure* that your database solution isn't adequate <G>.... see

On 10/6/06, <> wrote:
> I am a newbie to the lucene search area. I would like to best way to
> the following using lucene in terms of efficiency and the size of the
> index.
> Question : #1
> I have a table that contains some tags. These tags are tagged against
> multiple images that are in a different table (potentially 20 to
> images). If I am searching for a tag phrase and get the corresponding
> images, the approach that I was thinking is to join these two tables
> index the result set.
> For example:
> Tag(abc)- ImageId1, Tag(abc)-ImageId2, Tag(abc)-ImageId3 etc. Hence
> is a fairly fat joint. Assuming that we are doing like this how is the
> performance on lucene? If it is a bad design, what should be a better
> way of doing this? Looking forward to your valuable suggestions.

So, really, you're de-normalizing your database into an index. It seems
what you're really doing here is, for each tag, storing a list of
Then, given a tag, you want all the images. What do you think about
something like this....
doc = new Document();
doc.add("ID", "Tag(abc)", STORED, UNTOKENIZED); (note, IDs are often
untokenized, since you really don't want to split them up).
doc.add("images", "ImageId1", STORED, NO); (not indexed, but stored).
doc.add("images", "ImageId2", STORED, NO);

Now, to get the images associated with a tag, you just search for the
whose ID is your tag, get the doc and read the stored images field.
have to parse the image IDs out, but that should be trivial. The search
should be extremely fast since one and only one "document" matches.

There's no problem storing multiple data into the same document field.
you could assemble the whole list of IDs into a string and add the
field only once. or.....

You can vary this as you see fit. For instance, you could store each
in its own field in the doc. There are ways to enumerate the fields in a
given document, so once your search was satisfied by tag id, you'd be
and running.

doc.add("image1", "ImageId1", STORED, NO); (not indexed, but stored).
doc.add("image2", "ImageId2", STORED, NO);

NOTE: there is no requirement that each document in a lucene index have
same number or name of fields. In fact, you could create an index that
which no two documents had any field in common. Not, perhaps, a *useful*
index, but you could do it. If your head is in the DB table world, this
not immediately occur to you <G>....

Don't know if this helps, but I thought I'd mention it.

Question : #2
> I need to search the multiple fields from a table. The search phrase
> needs to look for the fields DESCRIPTION1 and DESCRIPTION2 in the
> I have done something like this:
> while ( {
> Document doc = new Document();
> doc.add(new Field("ID", String.valueOf(rs.getInt("ID")),
> Field.Store.YES, Field.Index.UN_TOKENIZED));
> doc.add(new Field("Description1", rs.getString("Description1"),
> Field.Store.YES, Field.Index.TOKENIZED));
> doc.add(new Field("Description2", rs.getString("Description2"),
> Field.Store.YES, Field.Index.TOKENIZED));
> String content = rs.getString("Description1") + " " +
> rs.getString("Description2")
> doc.add(new Field("cContent", content, Field.Store.YES,
> Field.Index.TOKENIZED));
> list[0].add(doc);
> }
> Do I need to do the cContent part for searching? Is this increasing
> size of the index? Is it better to create a dynamic query that looks
> the description1 description2 field or use the cContent?

No, you do not need the cContent part for searching. Yes, it'll increase
size of your index to include both (how could it not?).

Whether you should store description1 and description2, or just the
combination of the two depends upon whether you ever expect to need to
distinguish between them during searching. All other things being equal,
tend to favor leaving them in two distinct fields, as I don't believe
there's a noticable penalty for searching both, and you preserve

OTOH, it depends also on how you want to search your data. Let's say you
want to ask "Are terms A and B in the description fields?" If you store
as distinct fields, you need to form something like if (A is in
or description2) and (B is indescription1 or description2). Whereas if
are combined, all you have to ask is if (A and B are in combined).

So, let's assume that you have two description fields "because we had to
split them up to fit them in fixed length columns in the DB". Putting
back together actually makes the index representation of the problem
to the real problem space, so that's yet another consideration.....

Hope this helps

Please help me in figuring out these things.
> Thanks
> Mathews
> ---------------------------------------------------------------------
> To unsubscribe, e-mail:
> For additional commands, e-mail:

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message