Mailing-List: contact solr-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: solr-user@lucene.apache.org
From: "Teague James" <teaguej@insystechinc.com>
To: <solr-user@lucene.apache.org>
References: <058001d27ca7$bc019a50$3404cef0$@insystechinc.com> <CAN4YXvd77j-RA2R+hPV9BZc7ZiHSiWg=cdvmQ_5doNRpxK7+aQ@mail.gmail.com>
In-Reply-To: <CAN4YXvd77j-RA2R+hPV9BZc7ZiHSiWg=cdvmQ_5doNRpxK7+aQ@mail.gmail.com>
Subject: RE: Solr 6.0.0 Returns Blank Highlights for alpha-numeric combos
Date: Wed, 1 Feb 2017 15:23:21 -0500
Message-ID: <05aa01d27cc9$08e6cd60$1ab46820$@insystechinc.com>
MIME-Version: 1.0
Content-Type: text/plain;
	charset="utf-8"
Content-Transfer-Encoding: quoted-printable
Thread-Index: AQIdwhwhlz35fCpjdbJrraa110orWwIsxceIoKyhWNA=
Content-Language: en-us
archived-at: Wed, 01 Feb 2017 20:23:57 -0000

Hi Erick! Thanks for the reply. The goal is to get two character terms =
like 1a, 1b, 2a, 2b, 3a, etc. to get highlighted in the documents. =
Additional testing shows that any alpha-numeric combo returns a blank =
highlight, regardless of length. Thus, "pr0blem" will not highlight =
because of the zero in the middle of the term.

I came across a ServerFault article where it was suggested that the =
fieldType must be tokenized in order for highlighting to work correctly. =
Setting the field type to text_general was suggested as a solution. In =
my case the data is stored as a string fieldType, which is then copied =
using copyField to a field that has a fieldType of text_general, but I'm =
still not getting a good highlight on terms like "1a". Highlighting =
works for any other non-alpha-numeric term though.

Other articles pointed to termVectors and termOffsets, but none of these =
seemed to help. Here's  my config:

<field name=3D"contents" type=3D"string" indexed=3D"true" =
stored=3D"true" termPositions=3D"true" termVectors=3D"true" =
termOffsets=3D"true" />
<field name=3D"text" type=3D"text_general" indexed=3D"true" =
stored=3D"true" multiValued=3D"true"/>
<copyField source=3D"contents" dest=3D"text"/>

<fieldType name=3D"text_general" class=3D"solr.TextField" =
positionIncrementGap=3D"100">
	<analyzer type=3D"index">
		<tokenizer class=3D"solr.WhitespaceTokenizerFactory"/>
		<filter class=3D"solr.StopFilterFactory" ignoreCase=3D"true" =
words=3D"stopwords.txt" />
		<filter class=3D"solr.WordDelimiterFilterFactory" catenateAll=3D"1" =
preserveOriginal=3D"1" generateNumberParts=3D"0" generateWordParts=3D"0" =
/>
		<filter class=3D"solr.SynonymFilterFactory" =
synonyms=3D"index_synonyms.txt" ignoreCase=3D"true" expand=3D"true"/>
		<filter class=3D"solr.LowerCaseFilterFactory"/>
		<filter class=3D"solr.PorterStemFilterFactory"/>
		<filter class=3D"solr.ApostropheFilterFactory"/>
	</analyzer>
	<analyzer type=3D"query">
 		<tokenizer class=3D"solr.WhitespaceTokenizerFactory"/>
		<filter class=3D"solr.WordDelimiterFilterFactory" catenateAll=3D"1" =
preserveOriginal=3D"1" generateNumberParts=3D"0" generateWordParts=3D"0" =
/>
		<filter class=3D"solr.StopFilterFactory" ignoreCase=3D"true" =
words=3D"stopwords.txt" />
		<filter class=3D"solr.LowerCaseFilterFactory"/>
		<filter class=3D"solr.PorterStemFilterFactory"/>
		<filter class=3D"solr.ApostropheFilterFactory"/>
	</analyzer>
</fieldType>

In the solrconfig file highlighting is set to use the text field: <str =
name=3D"hl.fl">text</str>=20

Thoughts?

Appreciate the help! Thanks!

-Teague

-----Original Message-----
From: Erick Erickson [mailto:erickerickson@gmail.com]=20
Sent: Wednesday, February 1, 2017 2:49 PM
To: solr-user <solr-user@lucene.apache.org>
Subject: Re: Solr 6.0.0 Returns Blank Highlights for alpha-numeric =
combos

How far into the text field are these tokens? The highlighter defaults =
to the first 10K characters under control of hl.maxAnalyzedChars. It's =
vaguely possible that the values happen to be farther along in the text =
than that. Not likely, mind you but possible.

Best,
Erick

On Wed, Feb 1, 2017 at 8:24 AM, Teague James <teaguej@insystechinc.com> =
wrote:
> Hello everyone! I'm still stuck on this issue and could really use=20
> some help. I have a Solr 6.0.0 instance that is storing documents=20
> peppered with text like "1a", "2e", "4c", etc. If I search the=20
> documents for a word, "ms", "in", "the", etc., I get the correct=20
> number of hits and the results are highlighted correctly in the=20
> highlighting section. But when I search for "1a" or "2e" I get hits,=20
> but the highlights are blank. Further testing revealed that the=20
> highlighter fails to highlight any combination of alpha-numeric two =
character value, such a n0, b1, 1z, etc.:
> <result name=3D"response" numFound=3D"1" start=3D"0"> ...
> <lst name=3D"highlighting">
> <lst name=3D"8667"/>
>
> Where "8667" is the document ID of the record that had the hit, but no =

> highlight. Other searches, "ms" for example, return:
> <result name=3D"response" numFound=3D"1" start=3D"0"> ...
> <lst name=3D"highlighting">
>  <lst name=3D"8667"/>
>   <arr name=3D"text">
>    <str>
>     <em>MS</em>
>    </str>
>   </arr>
>  </lst>
> </lst>
>
> Why does highlighting fail for "1a" type searches? Any help is =
appreciated!
> Thanks!
>
> -Teague James
>