lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jonathan Rochkind <rochk...@jhu.edu>
Subject Re: ampersand, dismax, combining two fields, one of which is keywordTokenizer
Date Wed, 15 Jun 2011 18:25:35 GMT
Okay, I figured this one out -- I'm participating in a thread with 
myself here, but for benefit of posterity, or if anyone's interested, 
it's kind of interesting.

It's actually a variation of the known issue with dismax, mm, and fields 
with varying stopwords. Actually a pretty tricky problem with dismax, 
which it's now clear goes way beyond just stopwords.

http://lucene.472066.n3.nabble.com/Dismax-Minimum-Match-Stopwords-Bug-td493483.html
http://bibwild.wordpress.com/2010/04/14/solr-stop-wordsdismax-gotcha/

So to understand, first familiarize yourself with that.

However, none of the fields involved here had any stopwords at all, so 
at first it wasn't obvious this was the problem. But having different 
tokenization and other analysis between fields can result in exactly the 
same problem, for certain queries.

One field in the dismax qf used an analyzer that stripped punctuation. 
(I'm actually not positive at this point _which_ analyzer in my chain 
was stripping punctuation, I'm using a bunch including some custom ones, 
but I was aware that punctuation was being stripped, this was intentional.)

So "monkey's" turns into "monkey".  "monkey:" turns into "monkey".  So 
far so good. But what happens if you have punctuation all by itself 
seperated by whitespace?  "Roosevlet & Churchill" turns into 
['roosevelt', 'churchill'].  That ampersand in the middle was stripped 
out, essentially _just as if_ it were a stopword. Only two tokens result 
from that input.

You can see where this is going -- another field involved in the dismax 
qf did NOT strip out punctuation. So three tokens result from that 
input, ['Roosevelt', '&', 'Churchill'].

Now we have exactly the situation that gives ride the dismax stopwords 
mm-behaving-funny situation, it's exactly the same thing.

Now I've fixed this for punctuation just by making those fields strip 
out punctuation, by adding these analyzers to the bottom of those 
previously-not-stripping-punctuation field definitions:

<!-- strip punctuation, to avoid dismax stopwords-like mm bug -->
<filter class="solr.PatternReplaceFilterFactory"
                 pattern="([\p{Punct}])" replacement="" replace="all"
         />
<!-- if after stripping punc we have any 0-length tokens, make
               sure to eliminate them. We can use LengthFilter min=1 for 
that,
               we dont' care about the max here, just a very large 
number. -->
<filter class="solr.LengthFilterFactory" min="1" max="100"/>


And things are working are how I expect again, at least for this 
punctuation issue. But there may be other edge cases where differences 
in analysis result in different number of tokens from different fields, 
which if they are both included in a dismax qf, will have bad effects on 
'mm'.

The lesson I think, is that the only absolute safe way to use dismax 
'mm', is when all fields in the 'qf' have exactly the same analysis.  
But obviously that's not very practical, it destroys much of the power 
of dismax. And some differences in analysis are certainly acceptable -- 
but it's rather tricky to figure out if your differences in analysis are 
going to be significant for this problem, under what input, and if so 
fix them. It is not an easy thing to do.  So dismax definitely has this 
gotcha potentially waiting for you, whenever mixing fields with 
different analysis in a 'qf'.


On 6/14/2011 5:25 PM, Jonathan Rochkind wrote:
> Okay, let's try the debug trace again without a pf to be less confusing.
>
> One field in qf, that's ordinary text tokenized, and does get hits:
>
> q=churchill%20%3A%20roosevelt&qt=search&qf=title1_t&mm=100%&debugQuery=true&pf=

>
>
> <str name="rawquerystring">churchill : roosevelt</str>
> <str name="querystring">churchill : roosevelt</str>
> <str name="parsedquery">
> +((DisjunctionMaxQuery((title1_t:churchil)~0.01) 
> DisjunctionMaxQuery((title1_t:roosevelt)~0.01))~2) ()
> </str>
> <str name="parsedquery_toString">
> +(((title1_t:churchil)~0.01 (title1_t:roosevelt)~0.01)~2) ()
> </str>
>
> And that gets 25 hits. Now we add in a second field to the qf, this 
> second field is also ordinarily tokenized. We expect no _fewer_ than 
> 25 hits, adding another field into qf, right? And indeed it still 
> results in exactly 25 hits (no additional hits from the additional qf 
> field).
>
> ?q=churchill%20%3A%20roosevelt&qt=search&qf=title1_t%20title2_t&mm=100%&debugQuery=true&pf=

>
>
> <str name="parsedquery">
> +((DisjunctionMaxQuery((title2_t:churchil | title1_t:churchil)~0.01) 
> DisjunctionMaxQuery((title2_t:roosevelt | 
> title1_t:roosevelt)~0.01))~2) ()
> </str>
> <str name="parsedquery_toString">
> +(((title2_t:churchil | title1_t:churchil)~0.01 (title2_t:roosevelt | 
> title1_t:roosevelt)~0.01)~2) ()
> </str>
>
>
>
> Okay, now we go back to just that first (ordinarily tokenized) field, 
> but add a second field in that uses KeywordTokenizerFactory.  We 
> expect this not neccesarily to ever match for a multi-word query, but 
> we don't expect it to be fewer than 25 hits, the 25 hits from the 
> first field in the qf should still be there, right? But it's not. What 
> happened, why not?
>
> q=churchill%20%3A%20roosevelt&qt=search&qf=title1_t%20isbn_t&mm=100%&debugQuery=true&pf=

>
>
>
> str name="rawquerystring">churchill : roosevelt</str>
> <str name="querystring">churchill : roosevelt</str>
> <str name="parsedquery">+((DisjunctionMaxQuery((isbn_t:churchill | 
> title1_t:churchil)~0.01) DisjunctionMaxQuery((isbn_t::)~0.01) 
> DisjunctionMaxQuery((isbn_t:roosevelt | title1_t:roosevelt)~0.01))~3) 
> ()</str>
> <str name="parsedquery_toString">+(((isbn_t:churchill | 
> title1_t:churchil)~0.01 (isbn_t::)~0.01 (isbn_t:roosevelt | 
> title1_t:roosevelt)~0.01)~3) ()</str>
>
>
>
> On 6/14/2011 5:19 PM, Jonathan Rochkind wrote:
>> I'm aware that using a field tokenized with KeywordTokenizerFactory 
>> is in a dismax 'qf' is often going to result in 0 hits on that field 
>> -- (when a whitespace-containing query is entered).  But I do it 
>> anyway, for cases where a non-whitespace-containing query is entered, 
>> then it hits.  And in those cases where it doesn't hit, I figure 
>> okay, well, the other fields in qf will hit or not, that's good enough.
>>
>> And usually that works. But it works _differently_ when my query 
>> contains an ampersand (or any other punctuation), result in 0 hits 
>> when it shoudln't, and I can't figure out why.
>>
>> basically,
>>
>> &defType=dismax&mm=100%&q=one : two&qf=text_field
>>
>> gets hits.  The ":" is thrown out the text_field, but the mm still 
>> passes somehow, right?
>>
>> But, in the same index:
>>
>> &defType=dismax&mm=100%&q=one : two&qf=text_field 
>> keyword_tokenized_text_field
>>
>> gets 0 hits.  Somehow maybe the inclusion of the 
>> keyword_tokenized_text_field in the qf causes dismax to calculate the 
>> mm differently, decide there are three tokens in there and they all 
>> must match, and the token ":" can never match because it's not in my 
>> index it's stripped out... but somehow this isn't a problem unless I 
>> include a keyword-tokenized  field in the qf?
>>
>> This is really confusing, if anyone has any idea what I'm talking 
>> about it and can shed any light on it, much appreciated.
>>
>> The conclusion I am reaching is just NEVER include anything but a 
>> more or less ordinarily tokenized field in a dismax qf. Sadly, it was 
>> useful for certain use cases for me.
>>
>> Oh, hey, the debugging trace woudl probably be useful:
>>
>>
>> <lstname="debug">
>> <strname="rawquerystring">
>> churchill : roosevelt
>> </str>
>> <strname="querystring">
>> churchill : roosevelt
>> </str>
>> <strname="parsedquery">
>> +((DisjunctionMaxQuery((isbn_t:churchill | title1_t:churchil)~0.01) 
>> DisjunctionMaxQuery((isbn_t::)~0.01) 
>> DisjunctionMaxQuery((isbn_t:roosevelt | title1_t:roosevelt)~0.01))~3) 
>> DisjunctionMaxQuery((title2_unstem:"churchill roosevelt"~3^240.0 | 
>> text:"churchil roosevelt"~3^10.0 | title2_t:"churchil 
>> roosevelt"~3^50.0 | author_unstem:"churchill roosevelt"~3^400.0 | 
>> title_exactmatch:churchill roosevelt^500.0 | title1_t:"churchil 
>> roosevelt"~3^60.0 | title1_unstem:"churchill roosevelt"~3^320.0 | 
>> author2_unstem:"churchill roosevelt"~3^240.0 | 
>> title3_unstem:"churchill roosevelt"~3^80.0 | subject_t:"churchil 
>> roosevelt"~3^10.0 | other_number_unstem:"churchill roosevelt"~3^40.0 
>> | subject_unstem:"churchill roosevelt"~3^80.0 | 
>> title_series_t:"churchil roosevelt"~3^40.0 | 
>> title_series_unstem:"churchill roosevelt"~3^60.0 | 
>> text_unstem:"churchill roosevelt"~3^80.0)~0.01)
>> </str>
>> <strname="parsedquery_toString">
>> +(((isbn_t:churchill | title1_t:churchil)~0.01 (isbn_t::)~0.01 
>> (isbn_t:roosevelt | title1_t:roosevelt)~0.01)~3) 
>> (title2_unstem:"churchill roosevelt"~3^240.0 | text:"churchil 
>> roosevelt"~3^10.0 | title2_t:"churchil roosevelt"~3^50.0 | 
>> author_unstem:"churchill roosevelt"~3^400.0 | 
>> title_exactmatch:churchill roosevelt^500.0 | title1_t:"churchil 
>> roosevelt"~3^60.0 | title1_unstem:"churchill roosevelt"~3^320.0 | 
>> author2_unstem:"churchill roosevelt"~3^240.0 | 
>> title3_unstem:"churchill roosevelt"~3^80.0 | subject_t:"churchil 
>> roosevelt"~3^10.0 | other_number_unstem:"churchill roosevelt"~3^40.0 
>> | subject_unstem:"churchill roosevelt"~3^80.0 | 
>> title_series_t:"churchil roosevelt"~3^40.0 | 
>> title_series_unstem:"churchill roosevelt"~3^60.0 | 
>> text_unstem:"churchill roosevelt"~3^80.0)~0.01
>> </str>
>> <lstname="explain"/>
>> <strname="QParser">
>> DisMaxQParser
>> </str>
>> <nullname="altquerystring"/>
>> <nullname="boostfuncs"/>
>> <lstname="timing">
>> <doublename="time">
>> 6.0
>> </double>
>> <lstname="prepare">
>> <doublename="time">
>> 3.0
>> </double>
>> <lstname="org.apache.solr.handler.component.QueryComponent">
>> <doublename="time">
>> 2.0
>> </double>
>> </lst>
>> <lstname="org.apache.solr.handler.component.FacetComponent">
>> <doublename="time">
>> 0.0
>> </double>
>> </lst>
>> <lstname="org.apache.solr.handler.component.MoreLikeThisComponent">
>> <doublename="time">
>> 0.0
>> </double>
>> </lst>
>> <lstname="org.apache.solr.handler.component.HighlightComponent">
>> <doublename="time">
>> 0.0
>> </double>
>> </lst>
>> <lstname="org.apache.solr.handler.component.StatsComponent">
>> <doublename="time">
>> 0.0
>> </double>
>> </lst>
>> <lstname="org.apache.solr.handler.component.SpellCheckComponent">
>> <doublename="time">
>> 0.0
>> </double>
>> </lst>
>> <lstname="org.apache.solr.handler.component.DebugComponent">
>> <doublename="time">
>> 0.0
>> </double>
>> </lst>
>> </lst>
>>
>>
>>
>

Mime
View raw message