Mailing-List: contact dev-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@lucene.apache.org
Date: Wed, 5 Dec 2012 22:47:58 +0000 (UTC)
From: "Robert Muir (JIRA)" <jira@apache.org>
To: dev@lucene.apache.org
Message-ID: <839609962.65500.1354747678501.JavaMail.jiratomcat@arcas>
In-Reply-To: <257789653.65270.1354745158758.JavaMail.jiratomcat@arcas>
Subject: [jira] [Commented] (SOLR-4148) WhiteSpaceTokenizer splits long
 no-whitespace sequences every 255 characters
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/SOLR-4148?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13510867#comment-13510867 ] 

Robert Muir commented on SOLR-4148:
-----------------------------------

We can probably give reasonable options (e.g. you can increase it up to 4k).

warning: ICUTokenizer has a limit really. its just larger (i think 4K).
                
> WhiteSpaceTokenizer splits long no-whitespace sequences every 255 characters
> ----------------------------------------------------------------------------
>
>                 Key: SOLR-4148
>                 URL: https://issues.apache.org/jira/browse/SOLR-4148
>             Project: Solr
>          Issue Type: Bug
>          Components: Schema and Analysis
>    Affects Versions: 3.5, 4.0
>         Environment: for branch_4x:
> Linux bigindy5 2.6.32-279.14.1.el6.centos.plus.x86_64 #1 SMP Wed Nov 7 00:40:45 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux
> java version "1.7.0_09"
> Java(TM) SE Runtime Environment (build 1.7.0_09-b05)
> Java HotSpot(TM) 64-Bit Server VM (build 23.5-b02, mixed mode)
> for 3.5:
> Linux idxa1 2.6.32-279.14.1.el6.centos.plus.x86_64 #1 SMP Wed Nov 7 00:40:45 UTC 2012 x86_64 x86_64 x86_64 GNU/Linux
> java version "1.6.0_29"
> Java(TM) SE Runtime Environment (build 1.6.0_29-b11)
> Java HotSpot(TM) 64-Bit Server VM (build 20.4-b02, mixed mode)
>            Reporter: Shawn Heisey
>             Fix For: 4.1, 5.0
>
>
> I have the following text input in a field with an analysis chain that starts with WhiteSpaceTokenizerFactory.
> Auto,DEU,Drogendezernat,Drogenfahnder,Drogenfahndung,Drug,Duesseldorf,Ermittler,Ermittlung,Fahnder,Fahndung,Fahrzeug,Germany,Innere,Kriminalitaet,Kriminalpolizei,Kriminalpolizisten,Kripo,Oeffentliche,Ordnung,Polizei,Polizeikelle,Polizist,Polizistin,Rauschgift,Rauschgiftdezernat,Sicherheit,Sicherheitsbehoerden,Sicherheitskraefte,Uniform,Verbrechen,Waffen,Zivilbeamte,Zivilfahnder,Zivilpolizisten,anhalten,car,crime,criminal,drug,enforcement,investigators,law,officers,police,policeman,policemen,policeofficer,public,saftey,security,squad,stop,stoppen,stopping,suspect,suspicious,ueberpruefen,verdaechtig
> This input has no whitespace.  WhiteSpaceTokenizer is taking this and breaking it into tokens after every 255th character.  It happens on both 3.5 and a recent branch_4x checkout.  I am using ICUTokenizerFactory on some fields in branch_4x, it doesn't have this problem.  I have not tried the ICUTokenizerFactory on 3.5.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org