lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jack Krupansky (JIRA)" <>
Subject [jira] [Commented] (SOLR-3723) Improve OOTB behavior: English word-splitting should default to autoGeneratePhraseQueries=true
Date Thu, 09 Aug 2012 21:29:20 GMT


Jack Krupansky commented on SOLR-3723:

My current leaning is a mini-proposal as follows:

1. Leave text_general as is since it has a specific purpose: cross-language generality, even
if it does show English in a less than optimum light.
2. "Fix" text_en to have autoGeneratePhraseQueries=true. Maybe the same fix for some other
European langages, TBD, as well.
2a. Add word-splitting (WDF), Porter stemming and a populated stopwords_en.txt to the new
text_en type to enhance OOBE to show Lucene/Solr in a better light. Possibly add a commented-out
Edge-N-Gram filter as well - maybe even include by default.
3. Add a new text_default type which curent example schema fields will use instead of text_general.
Now the controversial part: text_default will be a copy of the new text_en but with clear
comments advising users to switch it to a copy of text_general for non-whitespace languages,
and maybe specific types if they are known.

An alternate proposal for #3 is simply to modify the existing example schema fields to directly
reference text_en, also with clear comments directing users to change text_en references to
text_general (or other language-specific types) if English is not the preferred language.
Personally, I prefer the text_default to emphasize that this is just the default and users
are expected to consider whether default is best for them.

> Improve OOTB behavior: English word-splitting should default to autoGeneratePhraseQueries=true
> ----------------------------------------------------------------------------------------------
>                 Key: SOLR-3723
>                 URL:
>             Project: Solr
>          Issue Type: Improvement
>          Components: Schema and Analysis
>    Affects Versions: 3.4, 3.5, 3.6, 4.0-ALPHA, 3.6.1
>            Reporter: Jack Krupansky
> Digging through the Jira and revision history, I discovered that back at the end of May
2011, a change was made to Solr that fairly significantly degrades the OOTB behavior for English
Solr queries, namely for word-splitting of terms with embedded punctuation, so that they end
up, by default, doing the OR of the sub-terms, rather than doing the obvious phrase query
of the sub-terms.
> Just a couple of examples:
> 1. CD-ROM => CD OR ROM rather than “CD ROM”
> 2. 1,000 => 1 OR 000 rather than “1 000” (when using the WordDelimiterFilter innocently
added to text_general or text_en)
> 3. out-of-the-box => out OR of OR the OR box rather than “out of the box”
> 4. 3.6 => 3 OR 6 rather than "3 6" (when using WordDelimiterFilter innocently added
to text_general or text_en)
> 5. docid-001 => docid OR 001 rather than "DOCID 001"
> All of those queries will give surprising and unexpected results.
> Note: The hyphen issue is present in StandardTokenizer, even if WDF is not used. Side
note: The full behavior of StandardTokenizer should be more fully documented on the Analyzers
> Back to the history of the change, there was a lot of lively discussion on SOLR-2015
- add a config hook for autoGeneratePhraseQueries.
> And the actual change to default to the behavior described above was SOLR-2519 - improve
defaults for text_* field types.
> (Consider the entire discussion in those two issues incorporated here for reference.
Anyone wishing to participate in discussion on this issue would be well-advised to study those
two issues first.)
> I gather that the original motivation was for non-European languages, and that even some
European languages might search better without auto-phrase generation, but the decision to
default English terms to NOT automatically generate phrase queries and to generate OR queries
instead is rather surprising and unexpected and outright undesirable, as my examples above
> I had been aware of the behavior for quite some time, but I had thought it was simply
a lingering bug so I paid little attention to it, until I stumbled across this autoGeneratePhraseQueries
"feature" while looking at the query parser code. I can understand the need to disable automatic
phrase queries for SOME languages, but to disable it by default for English seems rather bizarre,
as my simple use cases above show.
> Even if no action is taken on this Jira, I feel that it is important that there be a
wider awareness of the significant and unexpected impact from SOLR-2519, and that what had
seemed like buggy behavior was done intentionally.
> Unless there has been a change of heart since SOLR-2015/2519, I guess we are stuck with
the default TextField behavior, but at least we could improve the example schema in several
> 1. The English text field types should have autoGeneratePhraseQueries=true. If a user
innocently adds a word delimiter to text_en, for example, they need to know that autoGeneratePhraseQueries=true
is needed. Better to preempt that confusion and put the attribute in now. In fact, hyphenated
terms fail as I have noted above, so the addition is needed even if a WDF is not added.
> 2. Add commentary about the impact of autoGeneratePhraseQueries=true/false - in terms
of use case examples, as above. Specifically note the ones that will break with if the feature
is disabled.
> Another, more controversial change will be:
> 3. Change text_general to autoGeneratePhraseQueries=true so that English will be treated
reasonably by default. I suspect that most European languages will be at least "okay". A comment
will note that this field attribute should be removed or set to false for non-whitespace languages,
or that an alternative field type should be used. I suspect that the first thing any non-whitespace
language application will want to do is pick the text field type that has analysis that makes
the most sense for them, so I see no need to mess up English for no good reason.
> Make no mistake, #3 is the primary and only real goal of this OOTB 
> improvement. Maybe "text_general" could be kept as is for reference as the purported
"general" text field type (except that it doesn't work well for English, as shown above),
and maybe there should be a "text_default" that I would propose should be a literal copy of
text_en with commentary to direct users to the other choices for language.
> I would note that text_ja already has autoGeneratePhraseQueries=false, so I'm not sure
why the default in the TextField code had to be changed to false. Any languages for which
automatic phrase query generation is problematic should be attributed similarly. But, now
that it is wired into the schema defaults, we may be stuck with it.
> I was rather surprised that SOLR-2519 actually changed the default in TextField rather
than simply set the attribute as appropriate for the various text field types.
> There are probably also a couple of places in the wikis where the surprising behavior
should be noted. There is literally no wiki documentation for this important feature. There
are only two references to autoGeneratePhraseQueries, with no discussion of exactly what this
feature does or what the downside is if it is disabled.
> In the past, there was no need to document the treatment of embedded word delimiters
(well, okay, the poor handling for non-whitespace languages SHOULD have been documented),
but now there is no documentation of the degradation of what was a default and implicit feature
that a lot of people assume should be automatic.
> And, I would propose that the 4.0 CHANGES.TXT very clearly highlight the kinds of use
cases that unsuspecting users may not realize were BROKEN by the commit of SOLR-2519 that
is masked under the innocent phrasing of "improve defaults for text_* field types". How many
users seriously understood that a query with embedded dashes and commas behave differently
as a result of that change?
> I am contemplating whether to suggest that the WordDelimiterFilter should also be part
of the default text field type. Right now, it is hidden off in text_en_splitting.
> I think stemming should also be part of the default English field type. The whole point
of the "example" schema is to show-off the best of Lucene/Solr.
> I'm not quite ready to propose that English be the default language supported by the
example schema, but I am 99.999% certain that we should focus it on European, Roman, Latin
languages. Non-European languages are indeed important, and should probably have their own
schema. text_general was a good idea, but in hindsight it appears to have not been such a
great idea in light of the word-splitting problems I have highlighted above.
> Maybe I would propose that text_general be left as is, but that we add text_default which
is a copy of text_en (which would have WDF and stemming added) and fields use text_default
as their type. That way, it would be clear what is going on and users could sensibly see what
needs to happen if they wish to switch default languages.
> After discussion settles, a revised final proposal will be composed. And some specific
and non-controversial issues may be split into separate Jira issues.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:!default.jspa
For more information on JIRA, see:


To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message