lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stefan Moises <moi...@shoptimax.de>
Subject Re: NullpointerException when combining spellcheck component and synonyms
Date Wed, 22 Sep 2010 20:10:55 GMT
  well, to sum it up... it doesn't really matter if I use standard or 
dismax, at the moment both give me NullPointers for the same query, 
although I didn't change anything since it was working ... it seems 
totally random, sometimes it works a couple of times, sometimes it 
doesn't :(

Weird...

Stefan

Am 22.09.2010 19:58, schrieb Stefan Moises:
>  Hi all,
>
> wow, this is weird...
> now before I file the JIRA issue - one thing I forgot to mention is 
> that I am using the edismax query parser.
> I've just done the following:
>
> 1) searched with edismax parser:
> /select?indent=on&version=2.2&q=beffy&fq=&start=0&rows=10&fl=*%2Cscore&qt=dismax&wt=standard&debugQuery=on&explainOther=&hl.fl=&spellcheck=true

>
> and got the NullPointer as usual
>
> 2) changed "dismax" to "standard", just to make sure - using the same 
> query:
> /select?indent=on&version=2.2&q=beffy&fq=&start=0&rows=10&fl=*%2Cscore&qt=standard&wt=standard&debugQuery=on&explainOther=&hl.fl=&spellcheck=true

>
> and got 6 results
>
> 3) changed back to "dismax" and got the 6 results, again - no more 
> NullPointer... !?
>
> Is that something cache-related maybe? Because I've had other indexes 
> which at first gave me the NullPointer exception using this setup and 
> after a while the synonyms were working, even with spellchecking 
> turned on and using edismax...
>
> Just for reference, here is my handler config:
> <requestHandler name="dismax" class="solr.SearchHandler" >
> <lst name="defaults">
> <str name="defType">edismax</str>
> <str name="echoParams">explicit</str>
> <float name="tie">0.05</float>
> <str name="qf">
>         oxtitle^136.9 oxartnum^20.9 oxartnum_exact^30 oxartnum_rev^30 
> oxtags^20 seokeywords^25 seodesc^15 oxlongdesc^10 oxcattitle^80.4 
> allcattitles_exact^55 allcattitles^50 allcattitles_rev^45 
> allcattitles_preserve_rev^50 oxmanu_title^60.0 oxvendor_title^60.0
> </str>
> <str name="pf">
>         oxtitle^136.9 oxartnum^20.9 oxartnum_exact^30 oxartnum_rev^30 
> oxtags^20 seokeywords^25 seodesc^15 oxlongdesc^10 oxcattitle^80.4 
> allcattitles_exact^55 allcattitles^50 allcattitles_rev^45 
> allcattitles_preserve_rev^50 oxmanu_title^60.0 oxvendor_title^60.0
> </str>
> <str name="mm">
>         2&lt;-1 5&lt;-2 6&lt;90%
> </str>
> <int name="ps">100</int>
> <str name="q.alt">*:*</str>
> <!-- example highlighter config, enable per-query with hl=true -->
> <str name="hl.fl">oxcattitle oxtitle oxshortdesc oxlongdesc</str>
> <!-- for this field, we want no fragmenting, just highlighting -->
> <str name="f.name.hl.fragsize">0</str>
> <!-- instructs Solr to return the field itself if no query terms are
>           found -->
> <str name="f.name.hl.alternateField">oxtitle</str>
> <str name="f.text.hl.fragmenter">regex</str> <!-- defined below -->
> </lst>
> <arr name="last-components">
> <str>spellcheck</str>
> <str>elevator</str>
> </arr>
> </requestHandler>
>
>
> Thanks for any hint :)
> Stefan
>
> Am 21.09.2010 21:14, schrieb Stefan Moises:
>>  Sure, no problem, I'll submit a JIRA entry :)
>>
>> Am 21.09.2010 16:13, schrieb Robert Muir:
>>> I don't think you should get an error like this from 
>>> SynonymFilter... would
>>> you mind opening a JIRA issue?
>>>
>>> On Tue, Sep 21, 2010 at 9:49 AM, Stefan Moises<moises@shoptimax.de>  
>>> wrote:
>>>
>>>>   Hi again,
>>>>
>>>> well it turns out that it still doesn't work ...
>>>> Sometimes it works (i.e. for some cores), sometimes I still get the
>>>> nullpointer - e.g. if I create a new core and use the same settings 
>>>> as a
>>>> working one, but index different data, then I add a synonym (e.g. 
>>>> "foo =>
>>>> bar") and activate spellchecking, then search for "foo" -  boom :(
>>>> And I can't really tell where this error comes from... any idea 
>>>> where to
>>>> start looking?
>>>>
>>>> Thanks,
>>>> Stefan
>>>>
>>>> Am 01.09.2010 17:20, schrieb Stefan Moises:
>>>>
>>>>    doh, looks like I only forgot to add the spellcheck component to my
>>>>> edismax request handler... now it works with:
>>>>>
>>>>> ...
>>>>> <arr name="last-components">
>>>>> <str>spellcheck</str>
>>>>> <str>elevator</str>
>>>>> </arr>
>>>>>
>>>>> What's strange is that spellchecking seemed to work *without* that 
>>>>> entry,
>>>>> too....
>>>>>
>>>>> Cheers,
>>>>> Stefan
>>>>>
>>>>> Am 01.09.2010 13:33, schrieb Stefan Moises:
>>>>>
>>>>>>   Hi there,
>>>>>>
>>>>>> I am using Solr from SVN,
>>>>>> https://svn.apache.org/repos/asf/lucene/dev/trunk (my last 
>>>>>> update/build
>>>>>> on my dev server was in July I think)...
>>>>>>
>>>>>> I've encountered a strange problem when using the Spellcheck 
>>>>>> component in
>>>>>> combination with the SynonymFilter...
>>>>>> My "text" field is pretty standard, using the default 
>>>>>> synonyms.txt file:
>>>>>> <fieldType name="text" class="solr.TextField" 
>>>>>> positionIncrementGap="100">
>>>>>> <analyzer type="index">
>>>>>> <tokenizer class="solr.WhitespaceTokenizerFactory"/>
>>>>>> <!-- in this example, we will only use synonyms at query time
>>>>>> <filter class="solr.SynonymFilterFactory" 
>>>>>> synonyms="index_synonyms.txt"
>>>>>> ignoreCase="true" expand="false"/>
>>>>>>         -->
>>>>>> <filter class="solr.StopFilterFactory" ignoreCase="true"
>>>>>> words="stopwords.txt"/>
>>>>>> <filter class="solr.WordDelimiterFilterFactory" 
>>>>>> generateWordParts="1"
>>>>>> generateNumberParts="1" catenateWords="1" catenateNumbers="1"
>>>>>> catenateAll="0" splitOnCaseChange="1"/>
>>>>>> <filter class="solr.LowerCaseFilterFactory"/>
>>>>>> <!--<filter class="solr.EnglishPorterFilterFactory"
>>>>>> protected="protwords.txt"/>  -->
>>>>>> <filter class="solr.SnowballPorterFilterFactory"
>>>>>> protected="protwords.txt"/>
>>>>>> <filter class="solr.RemoveDuplicatesTokenFilterFactory"/>
>>>>>> </analyzer>
>>>>>> <analyzer type="query">
>>>>>> <tokenizer class="solr.WhitespaceTokenizerFactory"/>
>>>>>> <filter class="solr.SynonymFilterFactory" synonyms="synonyms.txt"
>>>>>> ignoreCase="true" expand="true"/>
>>>>>> <filter class="solr.StopFilterFactory" ignoreCase="true"
>>>>>> words="stopwords.txt"/>
>>>>>> <filter class="solr.WordDelimiterFilterFactory" 
>>>>>> generateWordParts="1"
>>>>>> generateNumberParts="1" catenateWords="0" catenateNumbers="0"
>>>>>> catenateAll="0" splitOnCaseChange="1"/>
>>>>>> <filter class="solr.LowerCaseFilterFactory"/>
>>>>>> <!--<filter class="solr.EnglishPorterFilterFactory"
>>>>>> protected="protwords.txt"/>  -->
>>>>>> <filter class="solr.SnowballPorterFilterFactory"
>>>>>> protected="protwords.txt"/>
>>>>>> <filter class="solr.RemoveDuplicatesTokenFilterFactory"/>
>>>>>> </analyzer>
>>>>>> </fieldType>
>>>>>>
>>>>>> I have only added some terms at the end of synonyms.txt:
>>>>>> ...
>>>>>> # Synonym mappings can be used for spelling correction too
>>>>>> pixima =>  pixma
>>>>>>
>>>>>> tekanne =>  teekanne
>>>>>> teekane =>  teekanne
>>>>>> flashen =>  flaschen
>>>>>> flasen =>  flaschen
>>>>>>
>>>>>> Here is my query and the exception... if I turn off spellcheck,
>>>>>> everything works as expected and the synonyms are found...
>>>>>>
>>>>>> INFO: [db] webapp=/solr path=/select
>>>>>> params={mlt.minwl=3&spellcheck=true&facet=true&mlt.fl=oxmanu_oxid,oxvendor_oxid,oxtags,oxsearchkeys&spellcheck.q=flasen&mlt.mintf=1&facet.limit=-1&mlt=true&

>>>>>>
>>>>>> json.nl=map&hl.fl=oxtitle&hl.fl=oxshortdesc&hl.fl=oxlongdesc&hl.fl=oxtags&hl.fl=seodesc&hl.fl=seokeywords&wt=json&hl=true&rows=10&version=1.2&mlt.mindf=1&debugQuery=true&facet.sort=lex&start=0&q=flasen&facet.field=oxcat_oxid&facet.field=oxcat_oxidtitle&facet.field=oxprice&facet.field=oxmanu_oxid&facet.field=oxmanu_oxidtitle&facet.field=oxvendor_oxid&facet.field=oxvendor_oxidtitle&facet.field=attrgroup_oxid&facet.field=attrgroup_oxidtitle&facet.field=attrgroup_oxidvalue&facet.field=attrvalue_oxid&facet.field=attrvalue_oxidtitle&facet.field=attr2attrgroup_oxidtitle&qt=dismax&spellcheck.build=false}

>>>>>>
>>>>>> hits=2 status=500 QTime=14
>>>>>> 01.09.2010 12:54:47 org.apache.solr.common.SolrException log
>>>>>> SCHWERWIEGEND: java.lang.NullPointerException
>>>>>>         at
>>>>>> org.apache.lucene.util.AttributeSource.cloneAttributes(AttributeSource.java:470)

>>>>>>
>>>>>>         at
>>>>>> org.apache.lucene.analysis.synonym.SynonymFilter.incrementToken(SynonymFilter.java:128)

>>>>>>
>>>>>>         at
>>>>>> org.apache.lucene.analysis.core.StopFilter.incrementToken(StopFilter.java:260)

>>>>>>
>>>>>>         at
>>>>>> org.apache.lucene.analysis.miscellaneous.WordDelimiterFilter.incrementToken(WordDelimiterFilter.java:336)

>>>>>>
>>>>>>         at
>>>>>> org.apache.lucene.analysis.core.LowerCaseFilter.incrementToken(LowerCaseFilter.java:62)

>>>>>>
>>>>>>         at
>>>>>> org.apache.solr.handler.component.SpellCheckComponent.getTokens(SpellCheckComponent.java:380)

>>>>>>
>>>>>>         at
>>>>>> org.apache.solr.handler.component.SpellCheckComponent.process(SpellCheckComponent.java:127)

>>>>>>
>>>>>>         at
>>>>>> org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:203)

>>>>>>
>>>>>>         at
>>>>>> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:131)

>>>>>>
>>>>>>         at org.apache.solr.core.SolrCore.execute(SolrCore.java:1323)
>>>>>>         at
>>>>>> org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:337)

>>>>>>
>>>>>>         at
>>>>>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:240)

>>>>>>
>>>>>>         at
>>>>>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235)

>>>>>>
>>>>>>         at
>>>>>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)

>>>>>>
>>>>>>         at
>>>>>> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)

>>>>>>
>>>>>>         at
>>>>>> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)

>>>>>>
>>>>>>         at
>>>>>> org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:465)

>>>>>>
>>>>>>         at
>>>>>> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)

>>>>>>
>>>>>>         at
>>>>>> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)

>>>>>>
>>>>>>         at
>>>>>> org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:555)

>>>>>>
>>>>>>         at
>>>>>> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)

>>>>>>
>>>>>>         at
>>>>>> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298)

>>>>>>
>>>>>>         at
>>>>>> org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:852)

>>>>>>
>>>>>>         at
>>>>>> org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:588)

>>>>>>
>>>>>>         at
>>>>>> org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)

>>>>>>
>>>>>>         at java.lang.Thread.run(Thread.java:619)
>>>>>>
>>>>>> Thanks for any hint what I may be doing wrong! :)
>>>>>> Stefan
>>>>>>
>>>>>>
>>>>>>
>>>> -- 
>>>> *******************************************
>>>> Stefan Moises
>>>> Senior Softwareentwickler
>>>>
>>>> shoptimax GmbH
>>>> Guntherstraße 45 a
>>>> 90461 Nürnberg
>>>> Amtsgericht Nürnberg HRB 21703
>>>> GF Friedrich Schreieck
>>>>
>>>> Tel.: 0911/25566-25
>>>> Fax:  0911/25566-29
>>>> moises@shoptimax.de
>>>> http://www.shoptimax.de
>>>> *******************************************
>>>>
>>>>
>>>
>>
>

-- 
*******************************************
Stefan Moises
Senior Softwareentwickler

shoptimax GmbH
Guntherstraße 45 a
90461 Nürnberg
Amtsgericht Nürnberg HRB 21703
GF Friedrich Schreieck

Tel.: 0911/25566-25
Fax:  0911/25566-29
moises@shoptimax.de
http://www.shoptimax.de
*******************************************


Mime
View raw message