lucene-solr-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Markus Jelsma <markus.jel...@buyways.nl>
Subject RE: PatternReplaceFilterFactory creating empty string as a term
Date Tue, 05 Oct 2010 22:28:34 GMT
I'm not sure if this is the best approach but a LengthFilter will stop blank terms.

http://wiki.apache.org/solr/AnalyzersTokenizersTokenFilters#solr.LengthFilterFactory
 
-----Original message-----
From: Shawn Heisey <solr@elyograg.org>
Sent: Wed 06-10-2010 00:25
To: solr-user@lucene.apache.org; 
Subject: PatternReplaceFilterFactory creating empty string as a term

 I am developing a new schema. It has a pattern filter that trims 
leading and trailing punctuation from terms.

<filter class="solr.PatternReplaceFilterFactory"
pattern="^(\p{Punct}*)(.*?)(\p{Punct}*)$"
replacement="$2"
/>

It is resulting in empty terms, because there are situations in the 
analyzer stream where a term happens to be composed of nothing but 
punctuation. This problem is not happening in production. I want those 
terms removed.

This blank term makes the top of the list as far as term frequency. Out 
of 7.6 million documents, 4.8 million of them have it. From TermsComponent:

<response>
???
<lst name="responseHeader">
<int name="status">0</int>
<int name="QTime">19106</int>
</lst>
???
<lst name="terms">
???
<lst name="catchall">
<int name="">4830648</int>
<int name="usa">1863264</int>
<int name="photo">1743551</int>
<int name="new">1544314</int>
<int name="de">1455691</int>
<int name="during">1412551</int>
<int name="los">1408855</int>
<int name="united">1368594</int>
<int name="2009">1271103</int>
<int name="la">1204441</int>
</lst>
</lst>
</response>

Is there any existing way to remove empty terms during analysis? I tried 
TrimFilterFactory but that made no difference. Is this a bug in 
PatternReplaceFilterFactory?

Shawn


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message