lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From markharw00d <markharw...@yahoo.co.uk>
Subject Re: Reverse search
Date Sun, 25 Mar 2007 15:35:51 GMT

On app startup:
1) parse all Queries and place in an array.
2) Create a RAMIndex containing a doc for each query with content 
consisting of the query's terms (see Query.extractTerms). For optimal 
performance only index the most rare term for queries with multiple 
mandatory criteria e.g. PhraseQuerys. "Most rare" can be determined by 
looking at IndexReader.docFreq(t) using an existing index which is 
representative of  your type of content.
3) For any queries that can't be handled by 2) e.g. FuzzyQueries - add 
to list of "run always queries".

Whenever you receive a new document:
1) Put it in a MemoryIndex
2) Get a list of the document's terms by calling 
memoryIndex.getReader().terms();
3) For each term hit your query RAMIndex and get 
queryIndexReader.termDocs(term) - this will give you the ids of queries 
that need to be run - you can use the doc id to index straight into your 
parsed queries array.
4) Run all queries found in 3) and all those held in your "run always" 
list against the MemoryIndex containing your new document

Hope this helps,
Mark


Melanie Langlois wrote:
> Hi Mark,
> If I follow you, I should list the key terms in my incoming document, then select the
queries which contains these key terms, and then run those queries on my index ? If this is
correct there is two things I don't understand:
> -how do I know which term is a key term in my document ?
> -how can I select the queries? Should I index them in a separate index?
>
> Thanks,
>
>
> Mélanie Langlois 
>   
> -----Original Message-----
> From: mark harwood [mailto:markharw00d@yahoo.co.uk] 
> Sent: Friday, March 23, 2007 11:19 PM
> To: java-user@lucene.apache.org
> Subject: Re: Reverse search
>
> Bear in mind that the million queries you run on the MemoryIndex can be shortlisted if
you place those queries in a RAMIndex and use the source document's terms to "query the queries".
The list of unique terms for your document is readily available in the MemoryIndex's TermEnum.
> You can take this list and find "likely related queries" to execute from your Query index.
> Note that for phrase queries or other forms of query with multiple mandatory terms you
should only index one of the terms (preferably the rarest) to ensure that your query is not
needlessly executed. For example - using this approach I need only run the phrase query for
"XYZ limited" whenever I encounter a document with the rare term "XYZ" in it, rather than
the much more commonplace "limited". 
>
> Cheers
> Mark
>
> ----- Original Message ----
> From: karl wettin <karl.wettin@gmail.com>
> To: java-user@lucene.apache.org
> Sent: Friday, 23 March, 2007 12:54:36 PM
> Subject: Re: Reverse search
>
>
> 23 mar 2007 kl. 09.57 skrev Melanie Langlois:
>
>   
>> Well, I though to use the PerFieldAnalyzerWrapper which contains as  
>> basic the snowballAnalyzer with English stopwords and use  
>> snowballAnalyzer with language specific keywords for the fields  
>> which will be in different languages. But I'm seeing that in your  
>> MemoryIndexTest you commented the use of SnowballAnalyzer, is it  
>> because it's too slow. In this case, I think I could use the  
>> StandardAnalyzer... what do you think?
>>     
>
> I think that creating an index with a couple of documents takes a  
> fraction of the time it will take to place a million queries on that  
> index. There is no real need to optimize something that takes  
> milliseconds when you in the same process do something that takes  
> half a minute.
>
>   



	
	
		
___________________________________________________________ 
All new Yahoo! Mail "The new Interface is stunning in its simplicity and ease of use." - PC
Magazine 
http://uk.docs.yahoo.com/nowyoucan.html

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message