lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Owen Densmore <>
Subject Re: Multiple Keywords/Keyphrases fields
Date Wed, 16 Feb 2005 05:49:46 GMT
> From: Erik Hatcher <>
> Date: February 12, 2005 3:09:15 PM MST
> To: "Lucene Users List" <>
> Subject: Re: Multiple Keywords/Keyphrases fields
> The real question to answer is what types of queries you're planning 
> on making.  Rather than look at it from indexing forward, consider it 
> from searching backwards.
> How will users query using those keyword phrases?

Hi Erik.  Good point.

There are two uses we are making of the keyphrases:

	- Graphical Navigation: A Flash graphical browser will allow users to 
fly around in a space of documents, choosing what to be viewing: 
Authors, Keyphrases and Textual terms.  In any of these cases, the 
"closeness" of any of the fields will govern how close they will appear 
graphically.  In the case of authors, we will weight collaboration .. 
how often the authors work together.  In the case of Keyphrases, we 
will want to use something like distance vectors like you show in the 
book using the cosine measure.  Thus the keyphrases need to be separate 
entities within the document .. it would be a bug for us if the terms 
leaked across the separate kephrases within the document.

	- Textual Search: In this case, we will have two ways to search the 
keyphrases.  The first would be like the graphical navigation above 
where searching for "complex system" should require the terms to be in 
a single keyphrase.  The second way will be looser, where we may simply 
pool the keyphrases with titles and abstract, and allow them all to be 
searched together within the document.

Does this make sense?  So the question from the search standpoint is: 
do multiple instances of a field act like there are barriers across the 
instances, or are they somehow treated as a single instance somehow.  
In terms of the closeness calculation, for example, can we get separate 
term vectors for each instance of the keyphrase field, or will we get a 
single vector combining all the keyphrase terms within a single 

I hope this is clear!  Kinda hard to articulate.


> 	Erik
> On Feb 12, 2005, at 3:08 PM, Owen Densmore wrote:
>> I'm getting a bit more serious about the final form of our lucene 
>> index.  Each document has DocNumber, Authors, Title, Abstract, and 
>> Keywords.  By Keywords, I mean a comma separated list, each entry 
>> having possibly many terms in a phrase like:
>> 	temporal infomax, finite state automata, Markov chains,
>> 	conditional entropy, neural information processing
>> I presume I should be using a field "Keywords" which have many 
>> "entries" or "instances" per document (one per comma separated 
>> phrase).  But I'm not sure the right way to handle all this.  My 
>> assumption is that I should analyze them individually, just as we do 
>> for free text (the Abstract, for example), thus in the example above 
>> having 5 entries of the nature
>> 	doc.add(Field.Text("Keywords", "finite state automata"));
>> etc, analyzing them because these are author-supplied strings with no 
>> canonical form.
>> For guidance, I looked in the archive and found the attached email, 
>> but I didn't see the answer.  (I'm not concerned about the dups, I 
>> presume that is equivalent to a boos of some sort) Does this seem 
>> right?
>> Thanks once again.
>> Owen
>>> From: <>
>>> Subject: Multiple equal Fields?
>>> Date: Tue, 17 Feb 2004 12:47:58 +0100
>>> Hi!
>>> What happens if I do this:
>>> doc.add(Field.Text("foo", "bar"));
>>> doc.add(Field.Text("foo", "blah"));
>>> Is there a field "foo" with value "blah" or are there two "foo"s 
>>> (actually not
>>> possible) or is there one "foo" with the values "bar" and "blah"?
>>> And what does happen in this case:
>>> doc.add(Field.Text("foo", "bar"));
>>> doc.add(Field.Text("foo", "bar"));
>>> doc.add(Field.Text("foo", "bar"));
>>> Does lucene store this only once?
>>> Timo

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message