lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Askar Zaidi" <askar.za...@gmail.com>
Subject Re: Fine Tuning Lucene implementation
Date Wed, 25 Jul 2007 17:26:21 GMT
Hey guys,

One last question and I think I'll have an optimized algorithm.

How can I build a query in my program ?

This is what I am doing:

QueryParser queryParser = new QueryParser("contents", new
StandardAnalyzer());

 queryParser.setDefaultOperator(QueryParser.Operator.AND);

 Query q = queryParser.parse(queryString);

So doing : System.out.println(q) shows:

+contents:harvard +contents:business +contents:review

I'd like to modify Query q to read:

+contents:harvard +contents:business +contents:review +itemID: (id passed in
the search method)

So this would pick the one document I need from the Index and give me the
score. I don't have to iterate over Hits.

Any clues ? I can't find any examples on query building .

thanks !

Askar


On 7/25/07, Grant Ingersoll <grant.ingersoll@gmail.com> wrote:
>
> Yes, you can do that.
>
>
> On Jul 25, 2007, at 12:31 PM, Askar Zaidi wrote:
>
> > Heres what I mean:
> >
> > http://lucene.apache.org/java/docs/queryparsersyntax.html#Fields
> >
> > title:"The Right Way" AND text:go
> >
> >
> > Although, I am not searching for the title "the right way" , I am
> > looking
> > for the score by specifying a unique field (itemID).
> >
> > when I do System.out.println(query);
> >
> > I get:
> >
> > +contents:Harvard +contents:Business + contents: Review
> >
> > Can I just add:
> >
> > +contents:Harvard +contents:Business + contents: Review
> > +itemID=id       ??
> >
> > That query would just return one document.
> >
> > On 7/25/07, Askar Zaidi <askar.zaidi@gmail.com> wrote:
> >>
> >> Instead of refactoring the code, would there be a way to just
> >> modify the
> >> query in each search routine ?
> >>
> >> Such as, "search contents:<text> and item:<itemID>"; This means
it
> >> would
> >> just collect the score of that one document whose itemID field =
> >> itemID
> >> passed from while( rs.next()).
> >>
> >> I just need to collect the score of the <itemID> already in the
> >> index.
> >>
> >> Would there be a way to modify the query ? Add a clause ?
> >>
> >> thanks,
> >> Askar
> >>
> >>
> >> On 7/25/07, Grant Ingersoll <grant.ingersoll@gmail.com> wrote:
> >>>
> >>> So, you really want a single Lucene score (based on the scores of
> >>> your 4 fields) for every itemID, correct?  And this score
> >>> consists of
> >>> scoring the title, tag, summary and body against some keywords
> >>> correct?
> >>>
> >>> Here's what I would do:
> >>>
> >>> while (rs.next())
> >>> {
> >>>      doc = getDocument(itemId);  // Get your document, including
> >>> contents from your database, no need even to put them in Lucene,
> >>> although you could
> >>>      add the doc to a MemoryIndex (see contrib/memory)
> >>>      Run your 4 searches against that memory index to get your
> >>> score.  Even better, combine your query into a single query that
> >>> searches all 4 fields at once, then Lucene will combine the score
> >>> for
> >>> you
> >>> }
> >>>
> >>> MemoryIndex info can be found at http://lucene.zones.apache.org:
> >>> 8080/
> >>> hudson/job/Lucene-Nightly/javadoc/org/apache/lucene/index/memory/
> >>> package-summary.html
> >>>
> >>> -Grant
> >>>
> >>> On Jul 25, 2007, at 11:45 AM, Askar Zaidi wrote:
> >>>
> >>>> Hi Grant,
> >>>>
> >>>> Thanks for the response. Heres what I am trying to accomplish:
> >>>>
> >>>> 1. Iterate over itemID (unique) in the database using one SQL
> >>>> query.
> >>>> 2. For every itemID found, run 4 searches on Lucene Index.
> >>>> 3. doTagSearch(itemID....) ; collect score
> >>>> 4. doTitleSearch(itemID...) ; collect score
> >>>> 5. doSummarySearch(itemID...) ; collect score
> >>>> 6. doBodySearch(itemID....) ; collect score
> >>>>
> >>>> These scores are then added and I get a total score for each unique
> >>>> item in
> >>>> the database.
> >>>>
> >>>> Lucene Index has: <itemID><tags><title><summary><contents>
> >>>>
> >>>> So if I am running a body search, I have 92 hits from over 300
> >>>> documents for
> >>>> a query. I already know my hit with the <itemID> .
> >>>>
> >>>> For instance, from step (1) if itemID 16 is passed to all the 4
> >>>> searches, I
> >>>> just need to get the score of the document which has itemID field =
> >>>> 16. I
> >>>> don't have to iterate over all the hits.
> >>>>
> >>>> I suppose I have to change my query to look for <contents> where
> >>>> itemID=16.
> >>>> Can you guide me as to how to do it ?
> >>>>
> >>>> thanks a ton,
> >>>>
> >>>> Askar
> >>>>
> >>>> On 7/25/07, Grant Ingersoll <gsingers@apache.org > wrote:
> >>>>>
> >>>>> Hi Askar,
> >>>>>
> >>>>> I suggest we take a step back, and ask the question, what are you
> >>>>> trying to accomplish?  That is, what is your application trying
to
> >>>>> do?  Forget the code, etc. just explain what you want the end
> >>>>> result
> >>>>> to be and we can work from there.   Based on what you have
> >>>>> described,
> >>>>> I am not sure you need access to the hits.  It seems like you just
> >>>>> need to make better queries.
> >>>>>
> >>>>> Is your itemID a unique identifier?  If yes, then you shouldn't
> >>>>> need
> >>>>> to loop over hits at all, as you should only ever have one
> >>>>> result IF
> >>>>> your query contains a required term.  Also, if this is the
> >>>>> case, why
> >>>>> do you need to do a search at all?  Haven't you already identified
> >>>>> the items of interest when you did your select query in the
> >>>>> database?  Or is it that you want to score the item based on some
> >>>>> terms as well.  If that is the case, there are other ways of doing
> >>>>> this and we can discuss them.
> >>>>>
> >>>>> -Grant
> >>>>>
> >>>>> On Jul 25, 2007, at 10:10 AM, Askar Zaidi wrote:
> >>>>>
> >>>>>> Hey Guys,
> >>>>>>
> >>>>>> I need to know how I can use the HitCollector class ? I am using
> >>>>>> Hits and
> >>>>>> looping over all the possible document hits (turns out its 92
> >>>>>> times
> >>>>>> I am
> >>>>>> looping; for 300 searches, its 300*92 !!). Can I avoid this
using
> >>>>>> HitCollector ? I can't seem to understand how its used.
> >>>>>>
> >>>>>> thanks a lot,
> >>>>>>
> >>>>>> Askar
> >>>>>>
> >>>>>> On 7/25/07, Dmitry <dmitrytkach1@hotmail.com> wrote:
> >>>>>>>
> >>>>>>> Askar,
> >>>>>>> why do you need to add +id:<idWeCareAbout>?
> >>>>>>> thanks,
> >>>>>>> dt,
> >>>>>>> www.ejinz.com
> >>>>>>> search engine news forms
> >>>>>>> ----- Original Message -----
> >>>>>>> From: "Askar Zaidi" <askar.zaidi@gmail.com >
> >>>>>>> To: <java-user@lucene.apache.org>; <nhira@cognocys.com>
> >>>>>>> Sent: Wednesday, July 25, 2007 12:39 AM
> >>>>>>> Subject: Re: Fine Tuning Lucene implementation
> >>>>>>>
> >>>>>>>
> >>>>>>>> Hey Hira ,
> >>>>>>>>
> >>>>>>>> Thanks so much for the reply. Much appreciate it.
> >>>>>>>>
> >>>>>>>> Quote:
> >>>>>>>>
> >>>>>>>> Would it be possible to just include a query clause?
> >>>>>>>>   - i.e., instead of just contents:<userQuery>,
also add
> >>>>>>>> +id:<idWeCareAbout>
> >>>>>>>>
> >>>>>>>> How can I do that ?
> >>>>>>>>
> >>>>>>>> I see my query as :
> >>>>>>>>
> >>>>>>>> +contents:harvard +contents:business +contents:review
> >>>>>>>>
> >>>>>>>> where the search phrase was: harvard business review
> >>>>>>>>
> >>>>>>>> Now how can I add +id:<idWeCareAbout>  ??
> >>>>>>>>
> >>>>>>>> This would give me that one exact document I am looking
> >>>>>>>> for , for
> >>>>>>>> that
> >>>>>>> id.
> >>>>>>>> I
> >>>>>>>> don't have to iterate through hits.
> >>>>>>>>
> >>>>>>>> thanks,
> >>>>>>>>
> >>>>>>>> Askar
> >>>>>>>>
> >>>>>>>>
> >>>>>>>>
> >>>>>>>> On 7/24/07, N. Hira < nhira@cognocys.com> wrote:
> >>>>>>>>>
> >>>>>>>>> I'm no expert on this (so please accept the comments
in that
> >>>>>>>>> context)
> >>>>>>>>> but 2 things seem weird to me:
> >>>>>>>>>
> >>>>>>>>> 1.  Iterating over each hit is an expensive proposition.
 I've
> >>>>>>>>> often
> >>>>>>>>> seen people recommending a HitCollector.
> >>>>>>>>>
> >>>>>>>>> 2.  It seems that doBodySearch() is essentially
saying, do
> >>>>>>>>> this
> >>>>>>>>> search
> >>>>>>>>> and return the score pertinent to this ID (using
an exhaustive
> >>>>>>>>> loop).
> >>>>>>>>> Would it be possible to just include a query clause?
> >>>>>>>>>     - i.e., instead of just contents:<userQuery>,
also add
> >>>>>>>>> +id:<idWeCareAbout>
> >>>>>>>>>
> >>>>>>>>> In general though, I think your algorithm seems
inefficient
> >>>>>>>>> (if I
> >>>>>>>>> understand it correctly):-- if I want to search
for one term
> >>>>>>>>> among 3 in
> >>>>>>>>> a "collection" of 300 documents (as defined by some
external
> >>>>>>> attribute),
> >>>>>>>>> I will wind up executing 300 x 3 searches, and for
each search
> >>>>>>>>> that is
> >>>>>>>>> executed, I will iterate over every Hit, even if
I've already
> >>>>>>>>> found the
> >>>>>>>>> one that I "care about".
> >>>>>>>>>
> >>>>>>>>> What would break if you:
> >>>>>>>>> 1.  Included "creator" in the Lucene index (or,
filtered
> >>>>>>>>> out the
> >>>>>>>>> Hits
> >>>>>>>>> using a BitSet or something like it)
> >>>>>>>>> 2.  Executed 1 search
> >>>>>>>>> 3.  Collected the results of the first N Hits (where
N is some
> >>>>>>>>> reasonable limit, like 100 or 500)
> >>>>>>>>>
> >>>>>>>>> -h
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>> On Tue, 2007-07-24 at 20:14 -0400, Askar Zaidi wrote:
> >>>>>>>>>
> >>>>>>>>>> Sure.
> >>>>>>>>>>
> >>>>>>>>>>  public float doBodySearch(Searcher searcher,String
query,
> >>>>>>>>>> int
> >>>>>>>>>> id){
> >>>>>>>>>>
> >>>>>>>>>>                  try{
> >>>>>>>>>>                                 score = search(searcher,
> >>>>>>>>>> query,id);
> >>>>>>>>>>                      }
> >>>>>>>>>>                       catch(IOException io){}
> >>>>>>>>>>                       catch(ParseException pe){}
> >>>>>>>>>>
> >>>>>>>>>>                       return score;
> >>>>>>>>>>
> >>>>>>>>>>                 }
> >>>>>>>>>>
> >>>>>>>>>>  private float search(Searcher searcher, String
queryString,
> >>>>>>>>>> int id)
> >>>>>>>>>> throws ParseException, IOException {
> >>>>>>>>>>
> >>>>>>>>>>         // Build a Query object
> >>>>>>>>>>
> >>>>>>>>>>         QueryParser queryParser = new QueryParser("contents",
> >>>>>>>>>> new
> >>>>>>>>>> KeywordAnalyzer());
> >>>>>>>>>>
> >>>>>>>>>>         queryParser.setDefaultOperator
> >>>>>>>>>> ( QueryParser.Operator.AND);
> >>>>>>>>>>
> >>>>>>>>>>         Query query = queryParser.parse(queryString);
> >>>>>>>>>>
> >>>>>>>>>>         // Search for the query
> >>>>>>>>>>
> >>>>>>>>>>         Hits hits = searcher.search(query);
> >>>>>>>>>>         Document doc = null;
> >>>>>>>>>>
> >>>>>>>>>>         // Examine the Hits object to see if
there were any
> >>>>>>>>>> matches
> >>>>>>>>>>         int hitCount = hits.length();
> >>>>>>>>>>
> >>>>>>>>>>                 for(int i=0;i<hitCount;i++){
> >>>>>>>>>>                 doc = hits.doc(i);
> >>>>>>>>>>                 String str = doc.get("item");
> >>>>>>>>>>                 int tmp = Integer.parseInt (str);
> >>>>>>>>>>                 if(tmp==id)
> >>>>>>>>>>                 score = hits.score(i);
> >>>>>>>>>>                 }
> >>>>>>>>>>
> >>>>>>>>>>         return score;
> >>>>>>>>>>     }
> >>>>>>>>>>
> >>>>>>>>>> I really need to optimize doBodySearch(...)
as this takes the
> >>>>>>>>>> most
> >>>>>>>>>> time.
> >>>>>>>>>>
> >>>>>>>>>> thanks guys,
> >>>>>>>>>> Askar
> >>>>>>>>>>
> >>>>>>>>>>
> >>>>>>>>>> On 7/24/07, N. Hira <nhira@cognocys.com>
wrote:
> >>>>>>>>>>
> >>>>>>>>>>         Could you show us the relevant source
from
> >>>>>>>>>> doBodySearch()?
> >>>>>>>>>>
> >>>>>>>>>>         -h
> >>>>>>>>>>
> >>>>>>>>>>         On Tue, 2007-07-24 at 19:58 -0400, Askar
Zaidi wrote:
> >>>>>>>>>>> I ran some tests and it seems that the slowness
is from
> >>>>>>>>>>         Lucene calls when I
> >>>>>>>>>>> do "doBodySearch", if I remove that call,
Lucene gives me
> >>>>>>>>>>         results in 5
> >>>>>>>>>>> seconds. otherwise it takes about 50 seconds.
> >>>>>>>>>>>
> >>>>>>>>>>> But I need to do Body search and that field
contains lots
> >>>>>>> of
> >>>>>>>>>>         text. The field
> >>>>>>>>>>> is <contents>. How can I optimize
that ?
> >>>>>>>>>>>
> >>>>>>>>>>> thanks,
> >>>>>>>>>>> Askar
> >>>>>>>>>>>
> >>>>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>>
> >>>>>>>>
> >>>>>>>
> >>>>>>>
> >>>>>>> ----------------------------------------------------------------
> >>>>>>> ---
> >>>>>>> --
> >>>>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> >>>>>>> For additional commands, e-mail: java-user-
> >>>>>>> help@lucene.apache.org
> >>>>>>>
> >>>>>>>
> >>>>>
> >>>>> --------------------------
> >>>>> Grant Ingersoll
> >>>>> Center for Natural Language Processing
> >>>>> http://www.cnlp.org/tech/lucene.asp
> >>>>>
> >>>>> Read the Lucene Java FAQ at http://wiki.apache.org/lucene-java/
> >>>>> LuceneFAQ
> >>>>>
> >>>>>
> >>>>>
> >>>>> ------------------------------------------------------------------
> >>>>> ---
> >>>
> >>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> >>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
> >>>>>
> >>>>>
> >>>
> >>> ------------------------------------------------------
> >>> Grant Ingersoll
> >>> http://www.grantingersoll.com/
> >>> http://lucene.grantingersoll.com
> >>> http://www.paperoftheweek.com/
> >>>
> >>>
> >>>
> >>> --------------------------------------------------------------------
> >>> -
> >>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> >>> For additional commands, e-mail: java-user-help@lucene.apache.org
> >>>
> >>>
> >>
>
> ------------------------------------------------------
> Grant Ingersoll
> http://www.grantingersoll.com/
> http://lucene.grantingersoll.com
> http://www.paperoftheweek.com/
>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message