Mailing-List: contact java-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-user@lucene.apache.org
Received-SPF: pass (nike.apache.org: domain of pulkitsinghal@gmail.com
 designates 74.125.83.48 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:date:message-id:subject:from:to
         :content-type:content-transfer-encoding;
        b=lAmivNuBQkCettWrq5OMjuqqXc9m/+yNYoDRLISEfjPJ5tp2ONcBwGOxKNz/D3nbXV
         cS94UNMM6wh2NzbOEkwDWNbzMXWqjbPsR5mK34nOKPovhUOv2CM7WcWsQpYWCiQvC05g
         vGcOrgzx1lUFMjqZQ7vQNLQtUTLjKqFHoN5yA=
MIME-Version: 1.0
In-Reply-To: <AANLkTikm=1jUZ8H4B5mObrCCXzJgr0_K9nW4ZPnQUMTF@mail.gmail.com>
References: <1290012469.1798.23.camel@maschine>
	<OFF76851AF.C9EE71BA-ON852577DE.0062190C-852577DE.006277F3@us.ibm.com>
	<1290017988.1798.34.camel@maschine>
	<AANLkTikm=1jUZ8H4B5mObrCCXzJgr0_K9nW4ZPnQUMTF@mail.gmail.com>
Date: Thu, 18 Nov 2010 13:05:07 -0500
Message-ID: <AANLkTi=Fh25DFu3jZehWbfOSHn79j7PskO9axR+Uyi2G@mail.gmail.com>
Subject: Re: uncorrect results
From: Pulkit Singhal <pulkitsinghal@gmail.com>
To: java-user@lucene.apache.org
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

Briefly looked at your code and there is no way that I'm right about
this but I'll say it anyway:
Every single field you index doesn't have any NORMS so how will the
scoring happen?
It probably happens based on the matches at query time but its not
like you are specifying any boosts in you query.
Lucene has a complex scoring formula that I don't claim to fully
understand ... but what if somehow (stay with me, don't shoot the
messenger) due to the fact that you have no NORMS at all, the results
being collected somehow give a score to the document that doesn't have
a match at all and therefore present it in the results?

Just a theory (a bad one perhaps) ... but one which can be easily
blown away by using ANALYZED in your indexer and then trying again.

- Pulkit

On Thu, Nov 18, 2010 at 12:55 PM, Pulkit Singhal
<pulkitsinghal@gmail.com> wrote:
> Wow, you live in a really great country and attend an awesome
> university where they have classes like "Text Analytics" I'm gonna
> send my kid there to study :)
>
> In all seriousness I think the problem may be with how you are
> collecting your results.
>
> I find this very amusing:
>> 80. 896889 phrase occurs 0 times
>
> How can it claim there are zero hits and still be returning you a result?=
 Weird.
>
> Have you tried removing all other docs and then only leaving the one
> problem child in there indexing just that and seeing what comes back?
>
> On Wed, Nov 17, 2010 at 1:19 PM, Jan <fajerski@informatik.hu-berlin.de> w=
rote:
>> thats what i figured...i can't find out what i'm doing wrong though ;)
>>
>> so the query is "experiment" (i know not really a phrase...but the
>> assignment requested precisely so). The program constructs the following
>> query
>>
>> +(AbstractText:"experiment" ArticleTitle:"experiment")
>>
>> which looks good to me. the results look like this:
>>
>> Found 95 hits.
>> 1. 19810 phrase occurs 3 times
>> 2. 587340 phrase occurs once
>> ...
>> 80. 896889 phrase occurs 0 times
>> ...
>> 95. 900325 phrase occurs once
>>
>> so here is the document 896889
>> PMID
>> =C2=A0 =C2=A0 =C2=A0 =C2=A0896889
>> ArticleTitle
>> =C2=A0 =C2=A0 =C2=A0 =C2=A0Estrogen-induced sexual receptivity and local=
ization of 3H-estradiol in
>> brains of female mice: effects of 5 alpha-reduced androgens, progestins
>> and cyproterone acetate.
>> AbstractText
>> =C2=A0 =C2=A0 =C2=A0 =C2=A0Sexual receptivity induced in ovariectomized =
CD-1 mice with chronic
>> daily administration of estradiol benzoate (E2 B) was blocked by
>> concurrent administration of the 5 alpha-reduced androgen,
>> dihydrotestosterone (DHT). Receptivity was restored in these females
>> with progesterone-, but not with dihydroprogesterone-priming 6 hr prior
>> to testing. Delaying the DHT injections until 12 hr after the E2 B
>> injections greatly reduced its inhibitory properties. Receptivity in E2
>> B-primed females was also blocked by concurrent treatment with
>> cyproterone acetate and 3 alpha-, but not 3 beta-adrostanediol.
>> Pretreatment with DHT, or 3 alpha- or 3 beta-androstanediol failed to
>> consistently affects 3H-estradiol accumulation in crude nuclear and
>> supernatant fractions from brain and pituitary
>>
>> so apart from doing something wrong while indexing/analyzing (the text
>> above is from the xml, but i double checked...it is put in teh index
>> with these textfragments) or so, the token "experiment" does not even
>> occur. thats what baffles me.
>>
>> thanks for the very quick reaction
>> jan
>>
>> Am Mittwoch, den 17.11.2010, 12:57 -0500 schrieb Donna L Gresh:
>>> As it is probably more likely that you're doing something incorrect tha=
n
>>> that Lucene is reporting incorrect results :), it might help if you
>>> reported the exact query that is being submitted to the IndexSearcher, =
and
>>> then showing us the document that was incorrectly returned. My guess is
>>> that either looking at the query itself will immediately reveal the
>>> problem to you, or that the query in combination with the document and
>>> knowledge of which analyzers you are using will reveal the problem-
>>>
>>> Donna
>>>
>>>
>>> Jan <fajerski@informatik.hu-berlin.de> wrote on 11/17/2010 11:47:49 AM:
>>>
>>> > [image removed]
>>> >
>>> > uncorrect results
>>> >
>>> > Jan
>>> >
>>> > to:
>>> >
>>> > java-user
>>> >
>>> > 11/17/2010 11:51 AM
>>> >
>>> > Please respond to java-user
>>> >
>>> > Hi,
>>> > i have an assignment in my Text Analytics class. I am supposed to cre=
ate
>>> > an index and search it. The corpus is a PubMed-like XML file. it is
>>> > possible to query terms (programcall a few terms) and phrases
>>> > (programcall "a phrase").
>>> > When a phrase is queried the program should answer how often the phra=
se
>>> > occured.
>>> > The problem is, on certain queries the IndexSearcher returns some
>>> > documents that do not have that particular query in its fields.
>>> > I'd be delighted if someone could tell me what i am doing wrong.
>>> > See the source code at my github repo
>>> >
>>> https://github.com/jangingnicht/TextAnalytics2/tree/master/src/textanal=
ytics2/
>>>
>>> >
>>> > Thanks in advance
>>> > jan
>>> >
>>> > PS: I use Lucene 3.0.2 and the OpenJDK Runtime Environment (IcedTea6
>>> > 1.8.2) on an 64 bit Linux machine.
>>> > [attachment "signature.asc" deleted by Donna L Gresh/Watson/IBM]
>>
>>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org