Mailing-List: contact java-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-user@lucene.apache.org
Received-SPF: pass (nike.apache.org: local policy)
DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.com;
  h=Message-ID:X-YMail-OSG:Received:X-Mailer:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type;
  b=aL9dby+PPfYCIYMAxF4CiYX+g3H2ByYyHJkI5+z2L+kZ38fjmPGmXiWbpnqaX2bQNATHDYyLkaGrwevWZcTQIlbtV+ghOr4mfcvDzw+svKKN1Z6CDWZybasX1ob1cj+Ph56akA8xeSuqXQDRRwBaxFxz4SS5d43zrqIbqQJNypE=;
Message-ID: <470047.33619.qm@web52908.mail.re2.yahoo.com>
Date: Wed, 21 Apr 2010 06:50:15 -0700 (PDT)
From: Ahmet Arslan <iorixxx@yahoo.com>
Subject: Re: are long words split into up to 256 long tokens?
To: java-user@lucene.apache.org
In-Reply-To: <j2vf071061b1004210620k7bbf23cbi21ea63a89670735b@mail.gmail.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii

> Is 256 some inner maximum too
> in some
> lucene internal that causes this? What is happening is that
> the long
> word is split into smaller words up to 256 and then the min
> and max
> limit applied. Is that correct? I have removed LengthFilter
> and still
> see the splitting at 256 happen. I would like not to have
> this, and
> removed altogheter any word longer than max, wihtout
> decomposing into
> smaller ones. Is there a way to achieve this?
> 
> Using lucene 3.0.1


Assuming your Tokenizer extends CharTokenizer:

CharTokenizer.java has this field: 
private static final int MAX_WORD_LEN = 255;

you can modify CharTokenizer.java according to your needs.


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org