lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stefan Moises <moi...@shoptimax.de>
Subject Re: NullpointerException when combining spellcheck component and synonyms
Date Wed, 22 Sep 2010 17:58:22 GMT
  Hi all,

wow, this is weird...
now before I file the JIRA issue - one thing I forgot to mention is that 
I am using the edismax query parser.
I've just done the following:

1) searched with edismax parser:
/select?indent=on&version=2.2&q=beffy&fq=&start=0&rows=10&fl=*%2Cscore&qt=dismax&wt=standard&debugQuery=on&explainOther=&hl.fl=&spellcheck=true
and got the NullPointer as usual

2) changed "dismax" to "standard", just to make sure - using the same query:
/select?indent=on&version=2.2&q=beffy&fq=&start=0&rows=10&fl=*%2Cscore&qt=standard&wt=standard&debugQuery=on&explainOther=&hl.fl=&spellcheck=true
and got 6 results

3) changed back to "dismax" and got the 6 results, again - no more 
NullPointer... !?

Is that something cache-related maybe? Because I've had other indexes 
which at first gave me the NullPointer exception using this setup and 
after a while the synonyms were working, even with spellchecking turned 
on and using edismax...

Just for reference, here is my handler config:
<requestHandler name="dismax" class="solr.SearchHandler" >
<lst name="defaults">
<str name="defType">edismax</str>
<str name="echoParams">explicit</str>
<float name="tie">0.05</float>
<str name="qf">
         oxtitle^136.9 oxartnum^20.9 oxartnum_exact^30 oxartnum_rev^30 
oxtags^20 seokeywords^25 seodesc^15 oxlongdesc^10 oxcattitle^80.4 
allcattitles_exact^55 allcattitles^50 allcattitles_rev^45 
allcattitles_preserve_rev^50 oxmanu_title^60.0 oxvendor_title^60.0
</str>
<str name="pf">
         oxtitle^136.9 oxartnum^20.9 oxartnum_exact^30 oxartnum_rev^30 
oxtags^20 seokeywords^25 seodesc^15 oxlongdesc^10 oxcattitle^80.4 
allcattitles_exact^55 allcattitles^50 allcattitles_rev^45 
allcattitles_preserve_rev^50 oxmanu_title^60.0 oxvendor_title^60.0
</str>
<str name="mm">
         2&lt;-1 5&lt;-2 6&lt;90%
</str>
<int name="ps">100</int>
<str name="q.alt">*:*</str>
<!-- example highlighter config, enable per-query with hl=true -->
<str name="hl.fl">oxcattitle oxtitle oxshortdesc oxlongdesc</str>
<!-- for this field, we want no fragmenting, just highlighting -->
<str name="f.name.hl.fragsize">0</str>
<!-- instructs Solr to return the field itself if no query terms are
           found -->
<str name="f.name.hl.alternateField">oxtitle</str>
<str name="f.text.hl.fragmenter">regex</str> <!-- defined below -->
</lst>
<arr name="last-components">
<str>spellcheck</str>
<str>elevator</str>
</arr>
</requestHandler>


Thanks for any hint :)
Stefan

Am 21.09.2010 21:14, schrieb Stefan Moises:
>  Sure, no problem, I'll submit a JIRA entry :)
>
> Am 21.09.2010 16:13, schrieb Robert Muir:
>> I don't think you should get an error like this from SynonymFilter... 
>> would
>> you mind opening a JIRA issue?
>>
>> On Tue, Sep 21, 2010 at 9:49 AM, Stefan Moises<moises@shoptimax.de>  
>> wrote:
>>
>>>   Hi again,
>>>
>>> well it turns out that it still doesn't work ...
>>> Sometimes it works (i.e. for some cores), sometimes I still get the
>>> nullpointer - e.g. if I create a new core and use the same settings 
>>> as a
>>> working one, but index different data, then I add a synonym (e.g. 
>>> "foo =>
>>> bar") and activate spellchecking, then search for "foo" -  boom :(
>>> And I can't really tell where this error comes from... any idea 
>>> where to
>>> start looking?
>>>
>>> Thanks,
>>> Stefan
>>>
>>> Am 01.09.2010 17:20, schrieb Stefan Moises:
>>>
>>>    doh, looks like I only forgot to add the spellcheck component to my
>>>> edismax request handler... now it works with:
>>>>
>>>> ...
>>>> <arr name="last-components">
>>>> <str>spellcheck</str>
>>>> <str>elevator</str>
>>>> </arr>
>>>>
>>>> What's strange is that spellchecking seemed to work *without* that 
>>>> entry,
>>>> too....
>>>>
>>>> Cheers,
>>>> Stefan
>>>>
>>>> Am 01.09.2010 13:33, schrieb Stefan Moises:
>>>>
>>>>>   Hi there,
>>>>>
>>>>> I am using Solr from SVN,
>>>>> https://svn.apache.org/repos/asf/lucene/dev/trunk (my last 
>>>>> update/build
>>>>> on my dev server was in July I think)...
>>>>>
>>>>> I've encountered a strange problem when using the Spellcheck 
>>>>> component in
>>>>> combination with the SynonymFilter...
>>>>> My "text" field is pretty standard, using the default synonyms.txt 
>>>>> file:
>>>>> <fieldType name="text" class="solr.TextField" 
>>>>> positionIncrementGap="100">
>>>>> <analyzer type="index">
>>>>> <tokenizer class="solr.WhitespaceTokenizerFactory"/>
>>>>> <!-- in this example, we will only use synonyms at query time
>>>>> <filter class="solr.SynonymFilterFactory" 
>>>>> synonyms="index_synonyms.txt"
>>>>> ignoreCase="true" expand="false"/>
>>>>>         -->
>>>>> <filter class="solr.StopFilterFactory" ignoreCase="true"
>>>>> words="stopwords.txt"/>
>>>>> <filter class="solr.WordDelimiterFilterFactory" generateWordParts="1"
>>>>> generateNumberParts="1" catenateWords="1" catenateNumbers="1"
>>>>> catenateAll="0" splitOnCaseChange="1"/>
>>>>> <filter class="solr.LowerCaseFilterFactory"/>
>>>>> <!--<filter class="solr.EnglishPorterFilterFactory"
>>>>> protected="protwords.txt"/>  -->
>>>>> <filter class="solr.SnowballPorterFilterFactory"
>>>>> protected="protwords.txt"/>
>>>>> <filter class="solr.RemoveDuplicatesTokenFilterFactory"/>
>>>>> </analyzer>
>>>>> <analyzer type="query">
>>>>> <tokenizer class="solr.WhitespaceTokenizerFactory"/>
>>>>> <filter class="solr.SynonymFilterFactory" synonyms="synonyms.txt"
>>>>> ignoreCase="true" expand="true"/>
>>>>> <filter class="solr.StopFilterFactory" ignoreCase="true"
>>>>> words="stopwords.txt"/>
>>>>> <filter class="solr.WordDelimiterFilterFactory" generateWordParts="1"
>>>>> generateNumberParts="1" catenateWords="0" catenateNumbers="0"
>>>>> catenateAll="0" splitOnCaseChange="1"/>
>>>>> <filter class="solr.LowerCaseFilterFactory"/>
>>>>> <!--<filter class="solr.EnglishPorterFilterFactory"
>>>>> protected="protwords.txt"/>  -->
>>>>> <filter class="solr.SnowballPorterFilterFactory"
>>>>> protected="protwords.txt"/>
>>>>> <filter class="solr.RemoveDuplicatesTokenFilterFactory"/>
>>>>> </analyzer>
>>>>> </fieldType>
>>>>>
>>>>> I have only added some terms at the end of synonyms.txt:
>>>>> ...
>>>>> # Synonym mappings can be used for spelling correction too
>>>>> pixima =>  pixma
>>>>>
>>>>> tekanne =>  teekanne
>>>>> teekane =>  teekanne
>>>>> flashen =>  flaschen
>>>>> flasen =>  flaschen
>>>>>
>>>>> Here is my query and the exception... if I turn off spellcheck,
>>>>> everything works as expected and the synonyms are found...
>>>>>
>>>>> INFO: [db] webapp=/solr path=/select
>>>>> params={mlt.minwl=3&spellcheck=true&facet=true&mlt.fl=oxmanu_oxid,oxvendor_oxid,oxtags,oxsearchkeys&spellcheck.q=flasen&mlt.mintf=1&facet.limit=-1&mlt=true&

>>>>>
>>>>> json.nl=map&hl.fl=oxtitle&hl.fl=oxshortdesc&hl.fl=oxlongdesc&hl.fl=oxtags&hl.fl=seodesc&hl.fl=seokeywords&wt=json&hl=true&rows=10&version=1.2&mlt.mindf=1&debugQuery=true&facet.sort=lex&start=0&q=flasen&facet.field=oxcat_oxid&facet.field=oxcat_oxidtitle&facet.field=oxprice&facet.field=oxmanu_oxid&facet.field=oxmanu_oxidtitle&facet.field=oxvendor_oxid&facet.field=oxvendor_oxidtitle&facet.field=attrgroup_oxid&facet.field=attrgroup_oxidtitle&facet.field=attrgroup_oxidvalue&facet.field=attrvalue_oxid&facet.field=attrvalue_oxidtitle&facet.field=attr2attrgroup_oxidtitle&qt=dismax&spellcheck.build=false}

>>>>>
>>>>> hits=2 status=500 QTime=14
>>>>> 01.09.2010 12:54:47 org.apache.solr.common.SolrException log
>>>>> SCHWERWIEGEND: java.lang.NullPointerException
>>>>>         at
>>>>> org.apache.lucene.util.AttributeSource.cloneAttributes(AttributeSource.java:470)

>>>>>
>>>>>         at
>>>>> org.apache.lucene.analysis.synonym.SynonymFilter.incrementToken(SynonymFilter.java:128)

>>>>>
>>>>>         at
>>>>> org.apache.lucene.analysis.core.StopFilter.incrementToken(StopFilter.java:260)

>>>>>
>>>>>         at
>>>>> org.apache.lucene.analysis.miscellaneous.WordDelimiterFilter.incrementToken(WordDelimiterFilter.java:336)

>>>>>
>>>>>         at
>>>>> org.apache.lucene.analysis.core.LowerCaseFilter.incrementToken(LowerCaseFilter.java:62)

>>>>>
>>>>>         at
>>>>> org.apache.solr.handler.component.SpellCheckComponent.getTokens(SpellCheckComponent.java:380)

>>>>>
>>>>>         at
>>>>> org.apache.solr.handler.component.SpellCheckComponent.process(SpellCheckComponent.java:127)

>>>>>
>>>>>         at
>>>>> org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:203)

>>>>>
>>>>>         at
>>>>> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:131)

>>>>>
>>>>>         at org.apache.solr.core.SolrCore.execute(SolrCore.java:1323)
>>>>>         at
>>>>> org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:337)

>>>>>
>>>>>         at
>>>>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:240)

>>>>>
>>>>>         at
>>>>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)

>>>>>
>>>>>         at
>>>>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)

>>>>>
>>>>>         at
>>>>> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)

>>>>>
>>>>>         at
>>>>> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)

>>>>>
>>>>>         at
>>>>> org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:465)

>>>>>
>>>>>         at
>>>>> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)

>>>>>
>>>>>         at
>>>>> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)

>>>>>
>>>>>         at
>>>>> org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:555)

>>>>>
>>>>>         at
>>>>> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)

>>>>>
>>>>>         at
>>>>> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298)

>>>>>
>>>>>         at
>>>>> org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:852)

>>>>>
>>>>>         at
>>>>> org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:588)

>>>>>
>>>>>         at
>>>>> org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)

>>>>>
>>>>>         at java.lang.Thread.run(Thread.java:619)
>>>>>
>>>>> Thanks for any hint what I may be doing wrong! :)
>>>>> Stefan
>>>>>
>>>>>
>>>>>
>>> -- 
>>> *******************************************
>>> Stefan Moises
>>> Senior Softwareentwickler
>>>
>>> shoptimax GmbH
>>> Guntherstraße 45 a
>>> 90461 Nürnberg
>>> Amtsgericht Nürnberg HRB 21703
>>> GF Friedrich Schreieck
>>>
>>> Tel.: 0911/25566-25
>>> Fax:  0911/25566-29
>>> moises@shoptimax.de
>>> http://www.shoptimax.de
>>> *******************************************
>>>
>>>
>>
>

-- 
*******************************************
Stefan Moises
Senior Softwareentwickler

shoptimax GmbH
Guntherstraße 45 a
90461 Nürnberg
Amtsgericht Nürnberg HRB 21703
GF Friedrich Schreieck

Tel.: 0911/25566-25
Fax:  0911/25566-29
moises@shoptimax.de
http://www.shoptimax.de
*******************************************


Mime
View raw message