lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Michael McCandless (JIRA)" <>
Subject [jira] [Commented] (SOLR-2519) Improve the defaults for the "text" field type in default schema.xml
Date Mon, 16 May 2011 17:59:47 GMT


Michael McCandless commented on SOLR-2519:

bq. I think maybe there's a misconception that the fieldType named "text" was meant to be
generic for all languages.

Regardless of what the original intention was, "text" today has become
the generic text fieldType new users use on starting with Solr.  I
mean, it has the perfect name for that :)

bq. As I said in the thread, if I had to do it over again, I would have named it "text_en"
because that's what it's purpose was.

Hindsight is 20/20... but, we can still fix this today.  We shouldn't
lock ourselves into poor defaults.

Especially, as things improve and we get better analyzers, etc., we
should be free to improve the defaults in schema.xml to take advantage
of these improvements.

bq. But at this point, it seems like the best way forward is to leave "text" as an english
fieldType and simply add other fieldTypes that can support other languages.

I think this is a dangerous approach -- the name (ie, missing _en if
in fact it has such English-specific configuration) is misleading and
traps new users.

Ideally, in the future, we wouldn't even have a "text" fieldType, only
text_XX per-language examples and then maybe something like
text_general, which you use if you cannot find your language.

Some downsides I see to this patch (i.e. trying to make the 'text' fieldType generic):

The current WordDelimiterFilter options the fieldType feel like a trap for non-whitespace-delimited
languages. WDF is configured to index catenations as well as splits... so all of the tokens
(words?) that are split out are also catenated together and indexed (which seems like it could
lead to some truly huge tokens erroneously being indexed.)
Ahh good point.  I think we should remove WDF altogether from the
generic "text" fieldType.

You left the english stemmer on the "text" fieldType... but if it's supposed to be generic,
couldn't this be bad for some other western languages where it could cause stemming collisions
of words not related to each other?

+1, we should remove the stemming too from "text".

bq. Taking into account all the existing users (and all the existing documentation, examples,
tutorial, etc), I favor a more conservative approach of adding new fieldTypes rather than
radically changing the behavior of existing ones.

Can you point to specific examples (docs, examples, tutorial)?  I'd
like to understand how much work it is to fix these...

My feeling is we should simply do the work here (I'll sign up to it)
and fix any places that actually rely on the specifics of "text"
fieldType, eg autophrase.

We shouldn't avoid fixing things well because it's gonna be more work
today, especially if someone (me) is signing up to do it.

Also: existing users would be unaffected by this?  They've already
copied over / edited their own schema.xml?  This is mainly about new

> Improve the defaults for the "text" field type in default schema.xml
> --------------------------------------------------------------------
>                 Key: SOLR-2519
>                 URL:
>             Project: Solr
>          Issue Type: Bug
>            Reporter: Michael McCandless
>            Assignee: Michael McCandless
>             Fix For: 3.2, 4.0
>         Attachments: SOLR-2519.patch
> Spinoff from:
> The text fieldType in schema.xml is unusable for non-whitespace
> languages, because it has the dangerous auto-phrase feature (of
> Lucene's QP -- see LUCENE-2458) enabled.
> Lucene leaves this off by default, as does ElasticSearch
> (http://
> Furthermore, the "text" fieldType uses WhitespaceTokenizer when
> StandardTokenizer is a better cross-language default.
> Until we have language specific field types, I think we should fix
> the "text" fieldType to work well for all languages, by:
>   * Switching from WhitespaceTokenizer to StandardTokenizer
>   * Turning off auto-phrase

This message is automatically generated by JIRA.
For more information on JIRA, see:

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message