Return-Path: Delivered-To: apmail-lucene-java-commits-archive@www.apache.org Received: (qmail 23293 invoked from network); 29 Oct 2010 13:45:33 -0000 Received: from unknown (HELO mail.apache.org) (140.211.11.3) by 140.211.11.9 with SMTP; 29 Oct 2010 13:45:33 -0000 Received: (qmail 87588 invoked by uid 500); 29 Oct 2010 13:45:33 -0000 Delivered-To: apmail-lucene-java-commits-archive@lucene.apache.org Received: (qmail 87462 invoked by uid 500); 29 Oct 2010 13:45:31 -0000 Mailing-List: contact java-commits-help@lucene.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: java-dev@lucene.apache.org Delivered-To: mailing list java-commits@lucene.apache.org Received: (qmail 87451 invoked by uid 99); 29 Oct 2010 13:45:30 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 29 Oct 2010 13:45:30 +0000 X-ASF-Spam-Status: No, hits=-2000.0 required=10.0 tests=ALL_TRUSTED X-Spam-Check-By: apache.org Received: from [140.211.11.4] (HELO eris.apache.org) (140.211.11.4) by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 29 Oct 2010 13:45:28 +0000 Received: by eris.apache.org (Postfix, from userid 65534) id 190C92388A3D; Fri, 29 Oct 2010 13:44:31 +0000 (UTC) Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Subject: svn commit: r1028739 - in /lucene/java/branches/lucene_3_0: ./ src/java/org/apache/lucene/analysis/standard/ Date: Fri, 29 Oct 2010 13:44:30 -0000 To: java-commits@lucene.apache.org From: uschindler@apache.org X-Mailer: svnmailer-1.0.8 Message-Id: <20101029134431.190C92388A3D@eris.apache.org> X-Virus-Checked: Checked by ClamAV on apache.org Author: uschindler Date: Fri Oct 29 13:44:30 2010 New Revision: 1028739 URL: http://svn.apache.org/viewvc?rev=1028739&view=rev Log: LUCENE-2384: Reset zzBuffer in StandardTokenizerImpl when lexer is reset Modified: lucene/java/branches/lucene_3_0/CHANGES.txt lucene/java/branches/lucene_3_0/src/java/org/apache/lucene/analysis/standard/StandardTokenizer.java lucene/java/branches/lucene_3_0/src/java/org/apache/lucene/analysis/standard/StandardTokenizerImpl.java lucene/java/branches/lucene_3_0/src/java/org/apache/lucene/analysis/standard/StandardTokenizerImpl.jflex Modified: lucene/java/branches/lucene_3_0/CHANGES.txt URL: http://svn.apache.org/viewvc/lucene/java/branches/lucene_3_0/CHANGES.txt?rev=1028739&r1=1028738&r2=1028739&view=diff ============================================================================== --- lucene/java/branches/lucene_3_0/CHANGES.txt (original) +++ lucene/java/branches/lucene_3_0/CHANGES.txt Fri Oct 29 13:44:30 2010 @@ -48,6 +48,9 @@ Bug fixes * LUCENE-2650: Added extra safety to MMapIndexInput clones to prevent accessing an unmapped buffer if the input is closed (Mike McCandless, Uwe Schindler, Robert Muir) +* LUCENE-2384: Reset zzBuffer in StandardTokenizerImpl when lexer is reset. + (Ruben Laguna via Uwe Schindler, sub-issue of LUCENE-2074) + Documentation * LUCENE-2239: Documented limitations in NIOFSDirectory and MMapDirectory due Modified: lucene/java/branches/lucene_3_0/src/java/org/apache/lucene/analysis/standard/StandardTokenizer.java URL: http://svn.apache.org/viewvc/lucene/java/branches/lucene_3_0/src/java/org/apache/lucene/analysis/standard/StandardTokenizer.java?rev=1028739&r1=1028738&r2=1028739&view=diff ============================================================================== --- lucene/java/branches/lucene_3_0/src/java/org/apache/lucene/analysis/standard/StandardTokenizer.java (original) +++ lucene/java/branches/lucene_3_0/src/java/org/apache/lucene/analysis/standard/StandardTokenizer.java Fri Oct 29 13:44:30 2010 @@ -203,21 +203,10 @@ public final class StandardTokenizer ext offsetAtt.setOffset(finalOffset, finalOffset); } - /* - * (non-Javadoc) - * - * @see org.apache.lucene.analysis.TokenStream#reset() - */ - @Override - public void reset() throws IOException { - super.reset(); - scanner.yyreset(input); - } - @Override public void reset(Reader reader) throws IOException { super.reset(reader); - reset(); + scanner.reset(reader); } /** Modified: lucene/java/branches/lucene_3_0/src/java/org/apache/lucene/analysis/standard/StandardTokenizerImpl.java URL: http://svn.apache.org/viewvc/lucene/java/branches/lucene_3_0/src/java/org/apache/lucene/analysis/standard/StandardTokenizerImpl.java?rev=1028739&r1=1028738&r2=1028739&view=diff ============================================================================== --- lucene/java/branches/lucene_3_0/src/java/org/apache/lucene/analysis/standard/StandardTokenizerImpl.java (original) +++ lucene/java/branches/lucene_3_0/src/java/org/apache/lucene/analysis/standard/StandardTokenizerImpl.java Fri Oct 29 13:44:30 2010 @@ -364,6 +364,17 @@ public final int yychar() } /** + * Resets the Tokenizer to a new Reader. + */ +final void reset(java.io.Reader r) { + // reset to default buffer size, if buffer has grown + if (zzBuffer.length > ZZ_BUFFERSIZE) { + zzBuffer = new char[ZZ_BUFFERSIZE]; + } + yyreset(r); +} + +/** * Fills Lucene token with the current token text. */ final void getText(Token t) { Modified: lucene/java/branches/lucene_3_0/src/java/org/apache/lucene/analysis/standard/StandardTokenizerImpl.jflex URL: http://svn.apache.org/viewvc/lucene/java/branches/lucene_3_0/src/java/org/apache/lucene/analysis/standard/StandardTokenizerImpl.jflex?rev=1028739&r1=1028738&r2=1028739&view=diff ============================================================================== --- lucene/java/branches/lucene_3_0/src/java/org/apache/lucene/analysis/standard/StandardTokenizerImpl.jflex (original) +++ lucene/java/branches/lucene_3_0/src/java/org/apache/lucene/analysis/standard/StandardTokenizerImpl.jflex Fri Oct 29 13:44:30 2010 @@ -65,6 +65,17 @@ public final int yychar() } /** + * Resets the Tokenizer to a new Reader. + */ +final void reset(java.io.Reader r) { + // reset to default buffer size, if buffer has grown + if (zzBuffer.length > ZZ_BUFFERSIZE) { + zzBuffer = new char[ZZ_BUFFERSIZE]; + } + yyreset(r); +} + +/** * Fills Lucene token with the current token text. */ final void getText(Token t) {