Mailing-List: contact java-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-user@lucene.apache.org
Subject: Re: Multi-field IDF
To: java-user@lucene.apache.org
References: <7318cff8-9275-a525-c0ea-e04b1e7dcb42@wolfram.com>
 <897856915.2053641.1479417908846@mail.yahoo.com>
 <89f7dac8-4c08-7e57-fd5d-d546d57cfc3b@wolfram.com>
From: Will Martin <wmartinusa@gmail.com>
Message-ID: <51571211-1b4e-9de9-fb0c-8b39034a31b2@gmail.com>
Date: Thu, 17 Nov 2016 21:21:30 -0500
User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:45.0) Gecko/20100101
 Thunderbird/45.4.0
MIME-Version: 1.0
In-Reply-To: <89f7dac8-4c08-7e57-fd5d-d546d57cfc3b@wolfram.com>
Content-Type: multipart/alternative;
 boundary="------------53EB7D1BB181DA9D03A937B1"
archived-at: Fri, 18 Nov 2016 02:21:43 -0000

--------------53EB7D1BB181DA9D03A937B1
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 8bit

are you familiar with pivoted normalized document length practice or 
theory? or croft's recent work on relevance algorithms accounting for 
structured field presence?


On 11/17/2016 5:20 PM, Nicolás Lichtmaier wrote:
> That depends on what you want. In this case I want to use a 
> discrimination power based in all the body text, not just the titles. 
> Because otherwise terms that are really not that relevant end up being 
> very high!
>
>
> El 17/11/16 a las 18:25, Ahmet Arslan escribió:
>> Hi Nicholas,
>>
>> IDF, among others, is a measure of term specificity. If 'or' is not 
>> so usual in titles, then it has some discrimination power in that 
>> domain.
>>
>> I think it's OK 'or' to get a high IDF value in this case.
>>
>> Ahmet
>>
>>
>>
>> On Thursday, November 17, 2016 9:09 PM, Nicolás Lichtmaier 
>> <nicolasl@wolfram.com> wrote:
>> IDF measures the selectivity of a term. But the calculation is
>> per-field. That can be bad for very short fields (like titles). One
>> example of this problem: If I don't delete stop words, then "or", "and",
>> etc. should be dealt with low IDF values, however "or" is, perhaps, not
>> so usual in titles. Then, "or" will have a high IDF value and be treated
>> as an important term. That's bad.
>>
>> One solution I see is to modify the Similarity to have a global, or
>> multi-field IDF value. This value would include in its calculation
>> longer fields that has more "normal text"-like stats. However this is
>> not trivial because I can't just add document-frequencies (I would be
>> counting some documents several times if "or" is present in more than
>> one field). I would need need to OR the bit-vectors that signal the
>> presence of the term, right? Not trivial.
>>
>> Has anyone encountered this issue? Has it been solved? Is my thinking 
>> wrong?
>>
>> Should I also try the developers' list?
>>
>> Thanks!
>>
>> Nicolás.-
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>


--------------53EB7D1BB181DA9D03A937B1--