Return-Path: 
 <solr-user-return-41358-apmail-lucene-solr-user-archive=lucene.apache.org@lucene.apache.org>
Delivered-To: apmail-lucene-solr-user-archive@minotaur.apache.org
Received: (qmail 72187 invoked from network); 22 Sep 2010 17:58:57 -0000
Received: from unknown (HELO mail.apache.org) (140.211.11.3)
  by 140.211.11.9 with SMTP; 22 Sep 2010 17:58:57 -0000
Received: (qmail 74615 invoked by uid 500); 22 Sep 2010 17:58:55 -0000
Delivered-To: apmail-lucene-solr-user-archive@lucene.apache.org
Received: (qmail 74465 invoked by uid 500); 22 Sep 2010 17:58:54 -0000
Mailing-List: contact solr-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:solr-user-help@lucene.apache.org>
List-Unsubscribe: <mailto:solr-user-unsubscribe@lucene.apache.org>
List-Post: <mailto:solr-user@lucene.apache.org>
List-Id: <solr-user.lucene.apache.org>
Reply-To: solr-user@lucene.apache.org
Delivered-To: mailing list solr-user@lucene.apache.org
Received: (qmail 74451 invoked by uid 99); 22 Sep 2010 17:58:54 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 22 Sep 2010 17:58:54 +0000
X-ASF-Spam-Status: No, hits=0.7 required=10.0
	tests=SPF_NEUTRAL
X-Spam-Check-By: apache.org
Received-SPF: neutral (nike.apache.org: local policy)
Received: from [212.114.153.196] (HELO mailrelay1.nefonline.de)
 (212.114.153.196)
    by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 22 Sep 2010 17:58:45 +0000
Received: from mail (gms1.nefonline.de [212.114.153.188])
	by mailrelay1.nefonline.de (NEFkom Mailservice) with ESMTP id o8MHwDSO012259
	for <solr-user@lucene.apache.org>; Wed, 22 Sep 2010 19:58:13 +0200
Received: from [87.175.143.187] by mail.shoptimax.de (GMS
 11.00.3335/NY4279.00.ab72d4bc) with ESMTP id fnywhaba for
 solr-user@lucene.apache.org; Wed, 22 Sep 2010 19:58:19 +0200
Message-ID: <4C9A43BE.8020508@shoptimax.de>
Date: Wed, 22 Sep 2010 19:58:22 +0200
From: Stefan Moises <moises@shoptimax.de>
Reply-To: moises@shoptimax.de
Organization: Shoptimax GmbH
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 6.1; de;
 rv:1.9.2.9) Gecko/20100915 Lightning/1.0b2 Thunderbird/3.1.4
MIME-Version: 1.0
To: solr-user@lucene.apache.org
Subject: Re: NullpointerException when combining spellcheck component and
 synonyms
References: <4C7E39F0.2080600@shoptimax.de> <4C7E6F4F.6010506@shoptimax.de>
 <4C98B803.50409@shoptimax.de>
 <AANLkTinh2NP61YfWZAtdD=gBmjct=tH0VEu+QPnDb4gj@mail.gmail.com>
 <4C990431.5000907@shoptimax.de>
In-Reply-To: <4C990431.5000907@shoptimax.de>
Content-Type: text/plain; charset=UTF-8; format=flowed
Content-Transfer-Encoding: 8bit
X-Originating-IP: [87.175.143.187]
X-AntiSpam: Checked for restricted content by Gordano's AntiSpam Software
X-Virus-Checked: Checked by ClamAV on apache.org

  Hi all,

wow, this is weird...
now before I file the JIRA issue - one thing I forgot to mention is that 
I am using the edismax query parser.
I've just done the following:

1) searched with edismax parser:
/select?indent=on&version=2.2&q=beffy&fq=&start=0&rows=10&fl=*%2Cscore&qt=dismax&wt=standard&debugQuery=on&explainOther=&hl.fl=&spellcheck=true
and got the NullPointer as usual

2) changed "dismax" to "standard", just to make sure - using the same query:
/select?indent=on&version=2.2&q=beffy&fq=&start=0&rows=10&fl=*%2Cscore&qt=standard&wt=standard&debugQuery=on&explainOther=&hl.fl=&spellcheck=true
and got 6 results

3) changed back to "dismax" and got the 6 results, again - no more 
NullPointer... !?

Is that something cache-related maybe? Because I've had other indexes 
which at first gave me the NullPointer exception using this setup and 
after a while the synonyms were working, even with spellchecking turned 
on and using edismax...

Just for reference, here is my handler config:
<requestHandler name="dismax" class="solr.SearchHandler" >
<lst name="defaults">
<str name="defType">edismax</str>
<str name="echoParams">explicit</str>
<float name="tie">0.05</float>
<str name="qf">
         oxtitle^136.9 oxartnum^20.9 oxartnum_exact^30 oxartnum_rev^30 
oxtags^20 seokeywords^25 seodesc^15 oxlongdesc^10 oxcattitle^80.4 
allcattitles_exact^55 allcattitles^50 allcattitles_rev^45 
allcattitles_preserve_rev^50 oxmanu_title^60.0 oxvendor_title^60.0
</str>
<str name="pf">
         oxtitle^136.9 oxartnum^20.9 oxartnum_exact^30 oxartnum_rev^30 
oxtags^20 seokeywords^25 seodesc^15 oxlongdesc^10 oxcattitle^80.4 
allcattitles_exact^55 allcattitles^50 allcattitles_rev^45 
allcattitles_preserve_rev^50 oxmanu_title^60.0 oxvendor_title^60.0
</str>
<str name="mm">
         2&lt;-1 5&lt;-2 6&lt;90%
</str>
<int name="ps">100</int>
<str name="q.alt">*:*</str>
<!-- example highlighter config, enable per-query with hl=true -->
<str name="hl.fl">oxcattitle oxtitle oxshortdesc oxlongdesc</str>
<!-- for this field, we want no fragmenting, just highlighting -->
<str name="f.name.hl.fragsize">0</str>
<!-- instructs Solr to return the field itself if no query terms are
           found -->
<str name="f.name.hl.alternateField">oxtitle</str>
<str name="f.text.hl.fragmenter">regex</str> <!-- defined below -->
</lst>
<arr name="last-components">
<str>spellcheck</str>
<str>elevator</str>
</arr>
</requestHandler>


Thanks for any hint :)
Stefan

Am 21.09.2010 21:14, schrieb Stefan Moises:
>  Sure, no problem, I'll submit a JIRA entry :)
>
> Am 21.09.2010 16:13, schrieb Robert Muir:
>> I don't think you should get an error like this from SynonymFilter... 
>> would
>> you mind opening a JIRA issue?
>>
>> On Tue, Sep 21, 2010 at 9:49 AM, Stefan Moises<moises@shoptimax.de>  
>> wrote:
>>
>>>   Hi again,
>>>
>>> well it turns out that it still doesn't work ...
>>> Sometimes it works (i.e. for some cores), sometimes I still get the
>>> nullpointer - e.g. if I create a new core and use the same settings 
>>> as a
>>> working one, but index different data, then I add a synonym (e.g. 
>>> "foo =>
>>> bar") and activate spellchecking, then search for "foo" -  boom :(
>>> And I can't really tell where this error comes from... any idea 
>>> where to
>>> start looking?
>>>
>>> Thanks,
>>> Stefan
>>>
>>> Am 01.09.2010 17:20, schrieb Stefan Moises:
>>>
>>>    doh, looks like I only forgot to add the spellcheck component to my
>>>> edismax request handler... now it works with:
>>>>
>>>> ...
>>>> <arr name="last-components">
>>>> <str>spellcheck</str>
>>>> <str>elevator</str>
>>>> </arr>
>>>>
>>>> What's strange is that spellchecking seemed to work *without* that 
>>>> entry,
>>>> too....
>>>>
>>>> Cheers,
>>>> Stefan
>>>>
>>>> Am 01.09.2010 13:33, schrieb Stefan Moises:
>>>>
>>>>>   Hi there,
>>>>>
>>>>> I am using Solr from SVN,
>>>>> https://svn.apache.org/repos/asf/lucene/dev/trunk (my last 
>>>>> update/build
>>>>> on my dev server was in July I think)...
>>>>>
>>>>> I've encountered a strange problem when using the Spellcheck 
>>>>> component in
>>>>> combination with the SynonymFilter...
>>>>> My "text" field is pretty standard, using the default synonyms.txt 
>>>>> file:
>>>>> <fieldType name="text" class="solr.TextField" 
>>>>> positionIncrementGap="100">
>>>>> <analyzer type="index">
>>>>> <tokenizer class="solr.WhitespaceTokenizerFactory"/>
>>>>> <!-- in this example, we will only use synonyms at query time
>>>>> <filter class="solr.SynonymFilterFactory" 
>>>>> synonyms="index_synonyms.txt"
>>>>> ignoreCase="true" expand="false"/>
>>>>>         -->
>>>>> <filter class="solr.StopFilterFactory" ignoreCase="true"
>>>>> words="stopwords.txt"/>
>>>>> <filter class="solr.WordDelimiterFilterFactory" generateWordParts="1"
>>>>> generateNumberParts="1" catenateWords="1" catenateNumbers="1"
>>>>> catenateAll="0" splitOnCaseChange="1"/>
>>>>> <filter class="solr.LowerCaseFilterFactory"/>
>>>>> <!--<filter class="solr.EnglishPorterFilterFactory"
>>>>> protected="protwords.txt"/>  -->
>>>>> <filter class="solr.SnowballPorterFilterFactory"
>>>>> protected="protwords.txt"/>
>>>>> <filter class="solr.RemoveDuplicatesTokenFilterFactory"/>
>>>>> </analyzer>
>>>>> <analyzer type="query">
>>>>> <tokenizer class="solr.WhitespaceTokenizerFactory"/>
>>>>> <filter class="solr.SynonymFilterFactory" synonyms="synonyms.txt"
>>>>> ignoreCase="true" expand="true"/>
>>>>> <filter class="solr.StopFilterFactory" ignoreCase="true"
>>>>> words="stopwords.txt"/>
>>>>> <filter class="solr.WordDelimiterFilterFactory" generateWordParts="1"
>>>>> generateNumberParts="1" catenateWords="0" catenateNumbers="0"
>>>>> catenateAll="0" splitOnCaseChange="1"/>
>>>>> <filter class="solr.LowerCaseFilterFactory"/>
>>>>> <!--<filter class="solr.EnglishPorterFilterFactory"
>>>>> protected="protwords.txt"/>  -->
>>>>> <filter class="solr.SnowballPorterFilterFactory"
>>>>> protected="protwords.txt"/>
>>>>> <filter class="solr.RemoveDuplicatesTokenFilterFactory"/>
>>>>> </analyzer>
>>>>> </fieldType>
>>>>>
>>>>> I have only added some terms at the end of synonyms.txt:
>>>>> ...
>>>>> # Synonym mappings can be used for spelling correction too
>>>>> pixima =>  pixma
>>>>>
>>>>> tekanne =>  teekanne
>>>>> teekane =>  teekanne
>>>>> flashen =>  flaschen
>>>>> flasen =>  flaschen
>>>>>
>>>>> Here is my query and the exception... if I turn off spellcheck,
>>>>> everything works as expected and the synonyms are found...
>>>>>
>>>>> INFO: [db] webapp=/solr path=/select
>>>>> params={mlt.minwl=3&spellcheck=true&facet=true&mlt.fl=oxmanu_oxid,oxvendor_oxid,oxtags,oxsearchkeys&spellcheck.q=flasen&mlt.mintf=1&facet.limit=-1&mlt=true& 
>>>>>
>>>>> json.nl=map&hl.fl=oxtitle&hl.fl=oxshortdesc&hl.fl=oxlongdesc&hl.fl=oxtags&hl.fl=seodesc&hl.fl=seokeywords&wt=json&hl=true&rows=10&version=1.2&mlt.mindf=1&debugQuery=true&facet.sort=lex&start=0&q=flasen&facet.field=oxcat_oxid&facet.field=oxcat_oxidtitle&facet.field=oxprice&facet.field=oxmanu_oxid&facet.field=oxmanu_oxidtitle&facet.field=oxvendor_oxid&facet.field=oxvendor_oxidtitle&facet.field=attrgroup_oxid&facet.field=attrgroup_oxidtitle&facet.field=attrgroup_oxidvalue&facet.field=attrvalue_oxid&facet.field=attrvalue_oxidtitle&facet.field=attr2attrgroup_oxidtitle&qt=dismax&spellcheck.build=false} 
>>>>>
>>>>> hits=2 status=500 QTime=14
>>>>> 01.09.2010 12:54:47 org.apache.solr.common.SolrException log
>>>>> SCHWERWIEGEND: java.lang.NullPointerException
>>>>>         at
>>>>> org.apache.lucene.util.AttributeSource.cloneAttributes(AttributeSource.java:470) 
>>>>>
>>>>>         at
>>>>> org.apache.lucene.analysis.synonym.SynonymFilter.incrementToken(SynonymFilter.java:128) 
>>>>>
>>>>>         at
>>>>> org.apache.lucene.analysis.core.StopFilter.incrementToken(StopFilter.java:260) 
>>>>>
>>>>>         at
>>>>> org.apache.lucene.analysis.miscellaneous.WordDelimiterFilter.incrementToken(WordDelimiterFilter.java:336) 
>>>>>
>>>>>         at
>>>>> org.apache.lucene.analysis.core.LowerCaseFilter.incrementToken(LowerCaseFilter.java:62) 
>>>>>
>>>>>         at
>>>>> org.apache.solr.handler.component.SpellCheckComponent.getTokens(SpellCheckComponent.java:380) 
>>>>>
>>>>>         at
>>>>> org.apache.solr.handler.component.SpellCheckComponent.process(SpellCheckComponent.java:127) 
>>>>>
>>>>>         at
>>>>> org.apache.solr.handler.component.SearchHandler.handleRequestBody(SearchHandler.java:203) 
>>>>>
>>>>>         at
>>>>> org.apache.solr.handler.RequestHandlerBase.handleRequest(RequestHandlerBase.java:131) 
>>>>>
>>>>>         at org.apache.solr.core.SolrCore.execute(SolrCore.java:1323)
>>>>>         at
>>>>> org.apache.solr.servlet.SolrDispatchFilter.execute(SolrDispatchFilter.java:337) 
>>>>>
>>>>>         at
>>>>> org.apache.solr.servlet.SolrDispatchFilter.doFilter(SolrDispatchFilter.java:240) 
>>>>>
>>>>>         at
>>>>> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:235) 
>>>>>
>>>>>         at
>>>>> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206) 
>>>>>
>>>>>         at
>>>>> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233) 
>>>>>
>>>>>         at
>>>>> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191) 
>>>>>
>>>>>         at
>>>>> org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:465) 
>>>>>
>>>>>         at
>>>>> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127) 
>>>>>
>>>>>         at
>>>>> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102) 
>>>>>
>>>>>         at
>>>>> org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:555) 
>>>>>
>>>>>         at
>>>>> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109) 
>>>>>
>>>>>         at
>>>>> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298) 
>>>>>
>>>>>         at
>>>>> org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:852) 
>>>>>
>>>>>         at
>>>>> org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:588) 
>>>>>
>>>>>         at
>>>>> org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489) 
>>>>>
>>>>>         at java.lang.Thread.run(Thread.java:619)
>>>>>
>>>>> Thanks for any hint what I may be doing wrong! :)
>>>>> Stefan
>>>>>
>>>>>
>>>>>
>>> -- 
>>> *******************************************
>>> Stefan Moises
>>> Senior Softwareentwickler
>>>
>>> shoptimax GmbH
>>> Guntherstraße 45 a
>>> 90461 Nürnberg
>>> Amtsgericht Nürnberg HRB 21703
>>> GF Friedrich Schreieck
>>>
>>> Tel.: 0911/25566-25
>>> Fax:  0911/25566-29
>>> moises@shoptimax.de
>>> http://www.shoptimax.de
>>> *******************************************
>>>
>>>
>>
>

-- 
*******************************************
Stefan Moises
Senior Softwareentwickler

shoptimax GmbH
Guntherstraße 45 a
90461 Nürnberg
Amtsgericht Nürnberg HRB 21703
GF Friedrich Schreieck

Tel.: 0911/25566-25
Fax:  0911/25566-29
moises@shoptimax.de
http://www.shoptimax.de
*******************************************