lucene-java-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From uschind...@apache.org
Subject svn commit: r932163 - in /lucene/dev/trunk/lucene: ./ src/java/org/apache/lucene/analysis/standard/ src/test/org/apache/lucene/analysis/
Date Thu, 08 Apr 2010 22:56:24 GMT
Author: uschindler
Date: Thu Apr  8 22:56:23 2010
New Revision: 932163

URL: http://svn.apache.org/viewvc?rev=932163&view=rev
Log:
LUCENE-2074: Refactor StandardTokenizer to have separate JFlex files per matchVersion. Also
reset zzBuffer to initial size on reset(Reader) and clean up reset methods.

Added:
    lucene/dev/trunk/lucene/src/java/org/apache/lucene/analysis/standard/StandardTokenizerImpl31.java
      - copied, changed from r886325, lucene/java/trunk/src/java/org/apache/lucene/analysis/standard/StandardTokenizerImpl.java
    lucene/dev/trunk/lucene/src/java/org/apache/lucene/analysis/standard/StandardTokenizerImpl31.jflex
      - copied, changed from r886325, lucene/java/trunk/src/java/org/apache/lucene/analysis/standard/StandardTokenizerImpl.jflex
    lucene/dev/trunk/lucene/src/java/org/apache/lucene/analysis/standard/StandardTokenizerImplOrig.java
      - copied, changed from r886325, lucene/java/trunk/src/java/org/apache/lucene/analysis/standard/StandardTokenizerImpl.java
    lucene/dev/trunk/lucene/src/java/org/apache/lucene/analysis/standard/StandardTokenizerImplOrig.jflex
      - copied, changed from r886325, lucene/java/trunk/src/java/org/apache/lucene/analysis/standard/StandardTokenizerImpl.jflex
    lucene/dev/trunk/lucene/src/java/org/apache/lucene/analysis/standard/StandardTokenizerInterface.java
  (with props)
Removed:
    lucene/dev/trunk/lucene/src/java/org/apache/lucene/analysis/standard/StandardTokenizerImpl.java
    lucene/dev/trunk/lucene/src/java/org/apache/lucene/analysis/standard/StandardTokenizerImpl.jflex
Modified:
    lucene/dev/trunk/lucene/CHANGES.txt
    lucene/dev/trunk/lucene/build.xml
    lucene/dev/trunk/lucene/common-build.xml
    lucene/dev/trunk/lucene/src/java/org/apache/lucene/analysis/standard/READ_BEFORE_REGENERATING.txt
    lucene/dev/trunk/lucene/src/java/org/apache/lucene/analysis/standard/StandardFilter.java
    lucene/dev/trunk/lucene/src/java/org/apache/lucene/analysis/standard/StandardTokenizer.java
    lucene/dev/trunk/lucene/src/test/org/apache/lucene/analysis/TestStandardAnalyzer.java

Modified: lucene/dev/trunk/lucene/CHANGES.txt
URL: http://svn.apache.org/viewvc/lucene/dev/trunk/lucene/CHANGES.txt?rev=932163&r1=932162&r2=932163&view=diff
==============================================================================
--- lucene/dev/trunk/lucene/CHANGES.txt (original)
+++ lucene/dev/trunk/lucene/CHANGES.txt Thu Apr  8 22:56:23 2010
@@ -236,6 +236,9 @@ Bug fixes
 * LUCENE-2344: PostingsConsumer.merge was failing to call finishDoc,
   which caused corruption for sep codec.  Also fixed several tests to
   test all 4 core codecs.  (Renaud Delbru via Mike McCandless)
+
+* LUCENE-2074: Reduce buffer size of lexer back to default on reset.
+  (Ruben Laguna, Shai Erera via Uwe Schindler)
   
 New features
 
@@ -347,6 +350,9 @@ New features
   can be used to prevent commits from ever getting deleted from the index.
   (Shai Erera)
   
+* LUCENE-2074: Make StandardTokenizer fit for Unicode 4.0, if the
+  matchVersion parameter is Version.LUCENE_31. (Uwe Schindler)
+
 Optimizations
 
 * LUCENE-2075: Terms dict cache is now shared across threads instead
@@ -433,9 +439,12 @@ Build
    into core, and moved the ICU-based collation support into contrib/icu.  
    (Robert Muir)
 
-* LUCENE-2326: Removed SVN checkouts for backwards tests. The backwards branch
-  is now included in the svn repository using "svn copy" after release.
-  (Uwe Schindler)
+* LUCENE-2326: Removed SVN checkouts for backwards tests. The backwards
+  branch is now included in the svn repository using "svn copy"
+  after release. (Uwe Schindler)
+
+* LUCENE-2074: Regenerating StandardTokenizerImpl files now needs
+  JFlex 1.5 (currently only available on SVN). (Uwe Schindler)
 
 Test Cases
 

Modified: lucene/dev/trunk/lucene/build.xml
URL: http://svn.apache.org/viewvc/lucene/dev/trunk/lucene/build.xml?rev=932163&r1=932162&r2=932163&view=diff
==============================================================================
--- lucene/dev/trunk/lucene/build.xml (original)
+++ lucene/dev/trunk/lucene/build.xml Thu Apr  8 22:56:23 2010
@@ -698,11 +698,14 @@ The source distribution does not contain
   <target name="jflex" depends="clean-jflex,jflex-StandardAnalyzer" />
 
   <target name="jflex-StandardAnalyzer" depends="init,jflex-check" if="jflex.present">
-    <taskdef classname="JFlex.anttask.JFlexTask" name="jflex">
-      <classpath location="${jflex.home}/lib/JFlex.jar" />
+    <taskdef classname="jflex.anttask.JFlexTask" name="jflex">
+			<classpath refid="jflex.classpath"/>
     </taskdef>
 
-    <jflex file="src/java/org/apache/lucene/analysis/standard/StandardTokenizerImpl.jflex"
+    <jflex file="src/java/org/apache/lucene/analysis/standard/StandardTokenizerImplOrig.jflex"
+           outdir="src/java/org/apache/lucene/analysis/standard"
+           nobak="on" />
+    <jflex file="src/java/org/apache/lucene/analysis/standard/StandardTokenizerImpl31.jflex"
            outdir="src/java/org/apache/lucene/analysis/standard"
            nobak="on" />
   </target>

Modified: lucene/dev/trunk/lucene/common-build.xml
URL: http://svn.apache.org/viewvc/lucene/dev/trunk/lucene/common-build.xml?rev=932163&r1=932162&r2=932163&view=diff
==============================================================================
--- lucene/dev/trunk/lucene/common-build.xml (original)
+++ lucene/dev/trunk/lucene/common-build.xml Thu Apr  8 22:56:23 2010
@@ -92,6 +92,13 @@
   <property name="javacc.home" location="${common.dir}"/>
   <property name="jflex.home" location="${common.dir}"/>
 
+	<path id="jflex.classpath">
+    <fileset dir="${jflex.home}/">
+      <include name="**/target/*.jar"/>
+      <include name="**/lib/*.jar"/>
+    </fileset>
+	</path>
+	
   <property name="backwards.dir" location="backwards"/>
   <property name="build.dir.backwards" location="${build.dir}/backwards"/>
 
@@ -151,11 +158,9 @@
     classpath="${javacc.home}/bin/lib/javacc.jar"
     />
 
-   <available
-    property="jflex.present"
-    classname="JFlex.anttask.JFlexTask"
-    classpath="${jflex.home}/lib/JFlex.jar"
-    />
+  <available property="jflex.present" classname="jflex.anttask.JFlexTask">
+		<classpath refid="jflex.classpath"/>
+	</available>
 
   <available
     property="maven.ant.tasks.present"
@@ -237,17 +242,17 @@
       JFlex not found.
       JFlex Home: ${jflex.home}
 
-      Please download and install JFlex from:
+      Please install the jFlex 1.5 version (currently not released)
+      from its SVN repository:
 
-      &lt;http://jflex.de/download.html&gt;
+       svn co http://jflex.svn.sourceforge.net/svnroot/jflex/trunk jflex
+       cd jflex
+       mvn install
 
       Then, create a build.properties file either in your home
       directory, or within the Lucene directory and set the jflex.home
-      property to the path where JFlex is installed. For example,
-      if you installed JFlex in /usr/local/java/jflex-1.4.1, then set the
-      jflex.home property to:
-
-      jflex.home=/usr/local/java/jflex-1.4.1
+      property to the path where the JFlex trunk checkout is located
+      (in the above example its the directory called "jflex").
 
       ##################################################################
     </fail>

Modified: lucene/dev/trunk/lucene/src/java/org/apache/lucene/analysis/standard/READ_BEFORE_REGENERATING.txt
URL: http://svn.apache.org/viewvc/lucene/dev/trunk/lucene/src/java/org/apache/lucene/analysis/standard/READ_BEFORE_REGENERATING.txt?rev=932163&r1=932162&r2=932163&view=diff
==============================================================================
--- lucene/dev/trunk/lucene/src/java/org/apache/lucene/analysis/standard/READ_BEFORE_REGENERATING.txt
(original)
+++ lucene/dev/trunk/lucene/src/java/org/apache/lucene/analysis/standard/READ_BEFORE_REGENERATING.txt
Thu Apr  8 22:56:23 2010
@@ -16,10 +16,5 @@
 */
 
 
-WARNING: if you change StandardTokenizerImpl.jflex and need to regenerate
-      the tokenizer, only use Java 1.4 !!!
-      This grammar currently uses constructs (eg :digit:, :letter:) whose 
-      meaning can vary according to the JRE used to run jflex.  See
-      https://issues.apache.org/jira/browse/LUCENE-1126 for details.
-      For current backwards compatibility it is needed to support
-      only Java 1.4 - this will change in Lucene 3.1.
+WARNING: if you change StandardTokenizerImpl*.jflex and need to regenerate
+      the tokenizer, only use the trunk version of JFlex 1.5 at the moment!

Modified: lucene/dev/trunk/lucene/src/java/org/apache/lucene/analysis/standard/StandardFilter.java
URL: http://svn.apache.org/viewvc/lucene/dev/trunk/lucene/src/java/org/apache/lucene/analysis/standard/StandardFilter.java?rev=932163&r1=932162&r2=932163&view=diff
==============================================================================
--- lucene/dev/trunk/lucene/src/java/org/apache/lucene/analysis/standard/StandardFilter.java
(original)
+++ lucene/dev/trunk/lucene/src/java/org/apache/lucene/analysis/standard/StandardFilter.java
Thu Apr  8 22:56:23 2010
@@ -34,12 +34,12 @@ public final class StandardFilter extend
     typeAtt = addAttribute(TypeAttribute.class);
   }
 
-  private static final String APOSTROPHE_TYPE = StandardTokenizerImpl.TOKEN_TYPES[StandardTokenizerImpl.APOSTROPHE];
-  private static final String ACRONYM_TYPE = StandardTokenizerImpl.TOKEN_TYPES[StandardTokenizerImpl.ACRONYM];
+  private static final String APOSTROPHE_TYPE = StandardTokenizer.TOKEN_TYPES[StandardTokenizer.APOSTROPHE];
+  private static final String ACRONYM_TYPE = StandardTokenizer.TOKEN_TYPES[StandardTokenizer.ACRONYM];
 
   // this filters uses attribute type
-  private TypeAttribute typeAtt;
-  private TermAttribute termAtt;
+  private final TypeAttribute typeAtt;
+  private final TermAttribute termAtt;
   
   /** Returns the next token in the stream, or null at EOS.
    * <p>Removes <tt>'s</tt> from the end of words.

Modified: lucene/dev/trunk/lucene/src/java/org/apache/lucene/analysis/standard/StandardTokenizer.java
URL: http://svn.apache.org/viewvc/lucene/dev/trunk/lucene/src/java/org/apache/lucene/analysis/standard/StandardTokenizer.java?rev=932163&r1=932162&r2=932163&view=diff
==============================================================================
--- lucene/dev/trunk/lucene/src/java/org/apache/lucene/analysis/standard/StandardTokenizer.java
(original)
+++ lucene/dev/trunk/lucene/src/java/org/apache/lucene/analysis/standard/StandardTokenizer.java
Thu Apr  8 22:56:23 2010
@@ -23,7 +23,7 @@ import java.io.Reader;
 import org.apache.lucene.analysis.Tokenizer;
 import org.apache.lucene.analysis.tokenattributes.OffsetAttribute;
 import org.apache.lucene.analysis.tokenattributes.PositionIncrementAttribute;
-import org.apache.lucene.analysis.tokenattributes.TermAttribute;
+import org.apache.lucene.analysis.tokenattributes.CharTermAttribute;
 import org.apache.lucene.analysis.tokenattributes.TypeAttribute;
 import org.apache.lucene.util.AttributeSource;
 import org.apache.lucene.util.Version;
@@ -55,7 +55,7 @@ import org.apache.lucene.util.Version;
 
 public final class StandardTokenizer extends Tokenizer {
   /** A private instance of the JFlex-constructed scanner */
-  private final StandardTokenizerImpl scanner;
+  private StandardTokenizerInterface scanner;
 
   public static final int ALPHANUM          = 0;
   public static final int APOSTROPHE        = 1;
@@ -111,7 +111,6 @@ public final class StandardTokenizer ext
    */
   public StandardTokenizer(Version matchVersion, Reader input) {
     super();
-    this.scanner = new StandardTokenizerImpl(input);
     init(input, matchVersion);
   }
 
@@ -120,7 +119,6 @@ public final class StandardTokenizer ext
    */
   public StandardTokenizer(Version matchVersion, AttributeSource source, Reader input) {
     super(source);
-    this.scanner = new StandardTokenizerImpl(input);
     init(input, matchVersion);
   }
 
@@ -129,29 +127,26 @@ public final class StandardTokenizer ext
    */
   public StandardTokenizer(Version matchVersion, AttributeFactory factory, Reader input)
{
     super(factory);
-    this.scanner = new StandardTokenizerImpl(input);
     init(input, matchVersion);
   }
 
-  private void init(Reader input, Version matchVersion) {
+  private final void init(Reader input, Version matchVersion) {
+    this.scanner = matchVersion.onOrAfter(Version.LUCENE_31) ?
+      new StandardTokenizerImpl31(input) : new StandardTokenizerImplOrig(input);
     if (matchVersion.onOrAfter(Version.LUCENE_24)) {
       replaceInvalidAcronym = true;
     } else {
       replaceInvalidAcronym = false;
     }
     this.input = input;    
-    termAtt = addAttribute(TermAttribute.class);
-    offsetAtt = addAttribute(OffsetAttribute.class);
-    posIncrAtt = addAttribute(PositionIncrementAttribute.class);
-    typeAtt = addAttribute(TypeAttribute.class);
   }
 
   // this tokenizer generates three attributes:
-  // offset, positionIncrement and type
-  private TermAttribute termAtt;
-  private OffsetAttribute offsetAtt;
-  private PositionIncrementAttribute posIncrAtt;
-  private TypeAttribute typeAtt;
+  // term offset, positionIncrement and type
+  private final CharTermAttribute termAtt = addAttribute(CharTermAttribute.class);
+  private final OffsetAttribute offsetAtt = addAttribute(OffsetAttribute.class);
+  private final PositionIncrementAttribute posIncrAtt = addAttribute(PositionIncrementAttribute.class);
+  private final TypeAttribute typeAtt = addAttribute(TypeAttribute.class);
 
   /*
    * (non-Javadoc)
@@ -166,7 +161,7 @@ public final class StandardTokenizer ext
     while(true) {
       int tokenType = scanner.getNextToken();
 
-      if (tokenType == StandardTokenizerImpl.YYEOF) {
+      if (tokenType == StandardTokenizerInterface.YYEOF) {
         return false;
       }
 
@@ -174,19 +169,19 @@ public final class StandardTokenizer ext
         posIncrAtt.setPositionIncrement(posIncr);
         scanner.getText(termAtt);
         final int start = scanner.yychar();
-        offsetAtt.setOffset(correctOffset(start), correctOffset(start+termAtt.termLength()));
+        offsetAtt.setOffset(correctOffset(start), correctOffset(start+termAtt.length()));
         // This 'if' should be removed in the next release. For now, it converts
         // invalid acronyms to HOST. When removed, only the 'else' part should
         // remain.
-        if (tokenType == StandardTokenizerImpl.ACRONYM_DEP) {
+        if (tokenType == StandardTokenizer.ACRONYM_DEP) {
           if (replaceInvalidAcronym) {
-            typeAtt.setType(StandardTokenizerImpl.TOKEN_TYPES[StandardTokenizerImpl.HOST]);
-            termAtt.setTermLength(termAtt.termLength() - 1); // remove extra '.'
+            typeAtt.setType(StandardTokenizer.TOKEN_TYPES[StandardTokenizer.HOST]);
+            termAtt.setLength(termAtt.length() - 1); // remove extra '.'
           } else {
-            typeAtt.setType(StandardTokenizerImpl.TOKEN_TYPES[StandardTokenizerImpl.ACRONYM]);
+            typeAtt.setType(StandardTokenizer.TOKEN_TYPES[StandardTokenizer.ACRONYM]);
           }
         } else {
-          typeAtt.setType(StandardTokenizerImpl.TOKEN_TYPES[tokenType]);
+          typeAtt.setType(StandardTokenizer.TOKEN_TYPES[tokenType]);
         }
         return true;
       } else
@@ -203,21 +198,10 @@ public final class StandardTokenizer ext
     offsetAtt.setOffset(finalOffset, finalOffset);
   }
 
-  /*
-   * (non-Javadoc)
-   *
-   * @see org.apache.lucene.analysis.TokenStream#reset()
-   */
-  @Override
-  public void reset() throws IOException {
-    super.reset();
-    scanner.yyreset(input);
-  }
-
   @Override
   public void reset(Reader reader) throws IOException {
     super.reset(reader);
-    reset();
+    scanner.reset(reader);
   }
 
   /**

Copied: lucene/dev/trunk/lucene/src/java/org/apache/lucene/analysis/standard/StandardTokenizerImpl31.java
(from r886325, lucene/java/trunk/src/java/org/apache/lucene/analysis/standard/StandardTokenizerImpl.java)
URL: http://svn.apache.org/viewvc/lucene/dev/trunk/lucene/src/java/org/apache/lucene/analysis/standard/StandardTokenizerImpl31.java?p2=lucene/dev/trunk/lucene/src/java/org/apache/lucene/analysis/standard/StandardTokenizerImpl31.java&p1=lucene/java/trunk/src/java/org/apache/lucene/analysis/standard/StandardTokenizerImpl.java&r1=886325&r2=932163&rev=932163&view=diff
==============================================================================
--- lucene/java/trunk/src/java/org/apache/lucene/analysis/standard/StandardTokenizerImpl.java
(original)
+++ lucene/dev/trunk/lucene/src/java/org/apache/lucene/analysis/standard/StandardTokenizerImpl31.java
Thu Apr  8 22:56:23 2010
@@ -1,4 +1,4 @@
-/* The following code was generated by JFlex 1.4.1 on 9/4/08 6:49 PM */
+/* The following code was generated by JFlex 1.5.0-SNAPSHOT on 09.04.10 00:10 */
 
 package org.apache.lucene.analysis.standard;
 
@@ -21,27 +21,22 @@ package org.apache.lucene.analysis.stand
 
 /*
 
-WARNING: if you change StandardTokenizerImpl.jflex and need to regenerate
-      the tokenizer, only use Java 1.4 !!!
-      This grammar currently uses constructs (eg :digit:, :letter:) whose 
-      meaning can vary according to the JRE used to run jflex.  See
-      https://issues.apache.org/jira/browse/LUCENE-1126 for details.
-      For current backwards compatibility it is needed to support
-      only Java 1.4 - this will change in Lucene 3.1.
+WARNING: if you change StandardTokenizerImpl*.jflex and need to regenerate
+      the tokenizer, only use the trunk version of JFlex 1.5 at the moment!
 
 */
 
-import org.apache.lucene.analysis.Token;
-import org.apache.lucene.analysis.tokenattributes.TermAttribute;
+import java.io.Reader;
+import org.apache.lucene.analysis.tokenattributes.CharTermAttribute;
 
 
 /**
  * This class is a scanner generated by 
- * <a href="http://www.jflex.de/">JFlex</a> 1.4.1
- * on 9/4/08 6:49 PM from the specification file
- * <tt>/tango/mike/src/lucene.standarddigit/src/java/org/apache/lucene/analysis/standard/StandardTokenizerImpl.jflex</tt>
+ * <a href="http://www.jflex.de/">JFlex</a> 1.5.0-SNAPSHOT
+ * on 09.04.10 00:10 from the specification file
+ * <tt>C:/Users/Uwe Schindler/Projects/lucene/trunk-full1/src/java/org/apache/lucene/analysis/standard/StandardTokenizerImpl31.jflex</tt>
  */
-class StandardTokenizerImpl {
+class StandardTokenizerImpl31 implements StandardTokenizerInterface {
 
   /** This character denotes the end of file */
   public static final int YYEOF = -1;
@@ -52,6 +47,16 @@ class StandardTokenizerImpl {
   /** lexical states */
   public static final int YYINITIAL = 0;
 
+  /**
+   * ZZ_LEXSTATE[l] is the state in the DFA for the lexical state l
+   * ZZ_LEXSTATE[l+1] is the state in the DFA for the lexical state l
+   *                  at the beginning of a line
+   * l is of the form l = 2*k, k a non negative integer
+   */
+  private static final int ZZ_LEXSTATE[] = { 
+     0, 0
+  };
+
   /** 
    * Translates characters to character classes
    */
@@ -59,59 +64,63 @@ class StandardTokenizerImpl {
     "\11\0\1\0\1\15\1\0\1\0\1\14\22\0\1\0\5\0\1\5"+
     "\1\3\4\0\1\11\1\7\1\4\1\11\12\2\6\0\1\6\32\12"+
     "\4\0\1\10\1\0\32\12\57\0\1\12\12\0\1\12\4\0\1\12"+
-    "\5\0\27\12\1\0\37\12\1\0\u0128\12\2\0\22\12\34\0\136\12"+
-    "\2\0\11\12\2\0\7\12\16\0\2\12\16\0\5\12\11\0\1\12"+
-    "\213\0\1\12\13\0\1\12\1\0\3\12\1\0\1\12\1\0\24\12"+
-    "\1\0\54\12\1\0\10\12\2\0\32\12\14\0\202\12\12\0\71\12"+
-    "\2\0\2\12\2\0\2\12\3\0\46\12\2\0\2\12\67\0\46\12"+
-    "\2\0\1\12\7\0\47\12\110\0\33\12\5\0\3\12\56\0\32\12"+
-    "\5\0\13\12\25\0\12\2\7\0\143\12\1\0\1\12\17\0\2\12"+
-    "\11\0\12\2\3\12\23\0\1\12\1\0\33\12\123\0\46\12\u015f\0"+
-    "\65\12\3\0\1\12\22\0\1\12\7\0\12\12\4\0\12\2\25\0"+
-    "\10\12\2\0\2\12\2\0\26\12\1\0\7\12\1\0\1\12\3\0"+
-    "\4\12\42\0\2\12\1\0\3\12\4\0\12\2\2\12\23\0\6\12"+
-    "\4\0\2\12\2\0\26\12\1\0\7\12\1\0\2\12\1\0\2\12"+
-    "\1\0\2\12\37\0\4\12\1\0\1\12\7\0\12\2\2\0\3\12"+
-    "\20\0\7\12\1\0\1\12\1\0\3\12\1\0\26\12\1\0\7\12"+
-    "\1\0\2\12\1\0\5\12\3\0\1\12\22\0\1\12\17\0\1\12"+
-    "\5\0\12\2\25\0\10\12\2\0\2\12\2\0\26\12\1\0\7\12"+
-    "\1\0\2\12\2\0\4\12\3\0\1\12\36\0\2\12\1\0\3\12"+
-    "\4\0\12\2\25\0\6\12\3\0\3\12\1\0\4\12\3\0\2\12"+
-    "\1\0\1\12\1\0\2\12\3\0\2\12\3\0\3\12\3\0\10\12"+
-    "\1\0\3\12\55\0\11\2\25\0\10\12\1\0\3\12\1\0\27\12"+
-    "\1\0\12\12\1\0\5\12\46\0\2\12\4\0\12\2\25\0\10\12"+
-    "\1\0\3\12\1\0\27\12\1\0\12\12\1\0\5\12\44\0\1\12"+
-    "\1\0\2\12\4\0\12\2\25\0\10\12\1\0\3\12\1\0\27\12"+
-    "\1\0\20\12\46\0\2\12\4\0\12\2\25\0\22\12\3\0\30\12"+
-    "\1\0\11\12\1\0\1\12\2\0\7\12\71\0\1\1\60\12\1\1"+
-    "\2\12\14\1\7\12\11\1\12\2\47\0\2\12\1\0\1\12\2\0"+
-    "\2\12\1\0\1\12\2\0\1\12\6\0\4\12\1\0\7\12\1\0"+
-    "\3\12\1\0\1\12\1\0\1\12\2\0\2\12\1\0\4\12\1\0"+
-    "\2\12\11\0\1\12\2\0\5\12\1\0\1\12\11\0\12\2\2\0"+
-    "\2\12\42\0\1\12\37\0\12\2\26\0\10\12\1\0\42\12\35\0"+
-    "\4\12\164\0\42\12\1\0\5\12\1\0\2\12\25\0\12\2\6\0"+
-    "\6\12\112\0\46\12\12\0\47\12\11\0\132\12\5\0\104\12\5\0"+
-    "\122\12\6\0\7\12\1\0\77\12\1\0\1\12\1\0\4\12\2\0"+
-    "\7\12\1\0\1\12\1\0\4\12\2\0\47\12\1\0\1\12\1\0"+
-    "\4\12\2\0\37\12\1\0\1\12\1\0\4\12\2\0\7\12\1\0"+
-    "\1\12\1\0\4\12\2\0\7\12\1\0\7\12\1\0\27\12\1\0"+
-    "\37\12\1\0\1\12\1\0\4\12\2\0\7\12\1\0\47\12\1\0"+
-    "\23\12\16\0\11\2\56\0\125\12\14\0\u026c\12\2\0\10\12\12\0"+
-    "\32\12\5\0\113\12\225\0\64\12\54\0\12\2\46\0\12\2\6\0"+
-    "\130\12\10\0\51\12\u0557\0\234\12\4\0\132\12\6\0\26\12\2\0"+
-    "\6\12\2\0\46\12\2\0\6\12\2\0\10\12\1\0\1\12\1\0"+
-    "\1\12\1\0\1\12\1\0\37\12\2\0\65\12\1\0\7\12\1\0"+
-    "\1\12\3\0\3\12\1\0\7\12\3\0\4\12\2\0\6\12\4\0"+
-    "\15\12\5\0\3\12\1\0\7\12\202\0\1\12\202\0\1\12\4\0"+
-    "\1\12\2\0\12\12\1\0\1\12\3\0\5\12\6\0\1\12\1\0"+
-    "\1\12\1\0\1\12\1\0\4\12\1\0\3\12\1\0\7\12\u0ecb\0"+
-    "\2\12\52\0\5\12\12\0\1\13\124\13\10\13\2\13\2\13\132\13"+
-    "\1\13\3\13\6\13\50\13\3\13\1\0\136\12\21\0\30\12\70\0"+
+    "\5\0\27\12\1\0\37\12\1\0\u013f\12\31\0\162\12\4\0\14\12"+
+    "\16\0\5\12\11\0\1\12\213\0\1\12\13\0\1\12\1\0\3\12"+
+    "\1\0\1\12\1\0\24\12\1\0\54\12\1\0\46\12\1\0\5\12"+
+    "\4\0\202\12\10\0\105\12\1\0\46\12\2\0\2\12\6\0\20\12"+
+    "\41\0\46\12\2\0\1\12\7\0\47\12\110\0\33\12\5\0\3\12"+
+    "\56\0\32\12\5\0\13\12\25\0\12\2\4\0\2\12\1\0\143\12"+
+    "\1\0\1\12\17\0\2\12\7\0\2\12\12\2\3\12\2\0\1\12"+
+    "\20\0\1\12\1\0\36\12\35\0\3\12\60\0\46\12\13\0\1\12"+
+    "\u0152\0\66\12\3\0\1\12\22\0\1\12\7\0\12\12\4\0\12\2"+
+    "\25\0\10\12\2\0\2\12\2\0\26\12\1\0\7\12\1\0\1\12"+
+    "\3\0\4\12\3\0\1\12\36\0\2\12\1\0\3\12\4\0\12\2"+
+    "\2\12\23\0\6\12\4\0\2\12\2\0\26\12\1\0\7\12\1\0"+
+    "\2\12\1\0\2\12\1\0\2\12\37\0\4\12\1\0\1\12\7\0"+
+    "\12\2\2\0\3\12\20\0\11\12\1\0\3\12\1\0\26\12\1\0"+
+    "\7\12\1\0\2\12\1\0\5\12\3\0\1\12\22\0\1\12\17\0"+
+    "\2\12\4\0\12\2\25\0\10\12\2\0\2\12\2\0\26\12\1\0"+
+    "\7\12\1\0\2\12\1\0\5\12\3\0\1\12\36\0\2\12\1\0"+
+    "\3\12\4\0\12\2\1\0\1\12\21\0\1\12\1\0\6\12\3\0"+
+    "\3\12\1\0\4\12\3\0\2\12\1\0\1\12\1\0\2\12\3\0"+
+    "\2\12\3\0\3\12\3\0\10\12\1\0\3\12\55\0\11\2\25\0"+
+    "\10\12\1\0\3\12\1\0\27\12\1\0\12\12\1\0\5\12\46\0"+
+    "\2\12\4\0\12\2\25\0\10\12\1\0\3\12\1\0\27\12\1\0"+
+    "\12\12\1\0\5\12\3\0\1\12\40\0\1\12\1\0\2\12\4\0"+
+    "\12\2\25\0\10\12\1\0\3\12\1\0\27\12\1\0\20\12\46\0"+
+    "\2\12\4\0\12\2\25\0\22\12\3\0\30\12\1\0\11\12\1\0"+
+    "\1\12\2\0\7\12\71\0\1\1\60\12\1\1\2\12\14\1\7\12"+
+    "\11\1\12\2\47\0\2\12\1\0\1\12\2\0\2\12\1\0\1\12"+
+    "\2\0\1\12\6\0\4\12\1\0\7\12\1\0\3\12\1\0\1\12"+
+    "\1\0\1\12\2\0\2\12\1\0\4\12\1\0\2\12\11\0\1\12"+
+    "\2\0\5\12\1\0\1\12\11\0\12\2\2\0\2\12\42\0\1\12"+
+    "\37\0\12\2\26\0\10\12\1\0\42\12\35\0\4\12\164\0\42\12"+
+    "\1\0\5\12\1\0\2\12\25\0\12\2\6\0\6\12\112\0\46\12"+
+    "\12\0\51\12\7\0\132\12\5\0\104\12\5\0\122\12\6\0\7\12"+
+    "\1\0\77\12\1\0\1\12\1\0\4\12\2\0\7\12\1\0\1\12"+
+    "\1\0\4\12\2\0\47\12\1\0\1\12\1\0\4\12\2\0\37\12"+
+    "\1\0\1\12\1\0\4\12\2\0\7\12\1\0\1\12\1\0\4\12"+
+    "\2\0\7\12\1\0\7\12\1\0\27\12\1\0\37\12\1\0\1\12"+
+    "\1\0\4\12\2\0\7\12\1\0\47\12\1\0\23\12\16\0\11\2"+
+    "\56\0\125\12\14\0\u026c\12\2\0\10\12\12\0\32\12\5\0\113\12"+
+    "\25\0\15\12\1\0\4\12\16\0\22\12\16\0\22\12\16\0\15\12"+
+    "\1\0\3\12\17\0\64\12\43\0\1\12\4\0\1\12\3\0\12\2"+
+    "\46\0\12\2\6\0\130\12\10\0\51\12\127\0\35\12\51\0\12\2"+
+    "\36\12\2\0\5\12\u038b\0\154\12\224\0\234\12\4\0\132\12\6\0"+
+    "\26\12\2\0\6\12\2\0\46\12\2\0\6\12\2\0\10\12\1\0"+
+    "\1\12\1\0\1\12\1\0\1\12\1\0\37\12\2\0\65\12\1\0"+
+    "\7\12\1\0\1\12\3\0\3\12\1\0\7\12\3\0\4\12\2\0"+
+    "\6\12\4\0\15\12\5\0\3\12\1\0\7\12\164\0\1\12\15\0"+
+    "\1\12\202\0\1\12\4\0\1\12\2\0\12\12\1\0\1\12\3\0"+
+    "\5\12\6\0\1\12\1\0\1\12\1\0\1\12\1\0\4\12\1\0"+
+    "\3\12\1\0\7\12\3\0\3\12\5\0\5\12\u0ebb\0\2\12\52\0"+
+    "\5\12\5\0\2\12\3\0\1\13\126\13\6\13\3\13\1\13\132\13"+
+    "\1\13\4\13\5\13\50\13\3\13\1\0\136\12\21\0\30\12\70\0"+
     "\20\13\u0100\0\200\13\200\0\u19b6\13\12\13\100\0\u51a6\13\132\13\u048d\12"+
-    "\u0773\0\u2ba4\12\u215c\0\u012e\13\322\13\7\12\14\0\5\12\5\0\1\12"+
-    "\1\0\12\12\1\0\15\12\1\0\5\12\1\0\1\12\1\0\2\12"+
-    "\1\0\2\12\1\0\154\12\41\0\u016b\12\22\0\100\12\2\0\66\12"+
-    "\50\0\14\12\164\0\3\12\1\0\1\12\1\0\207\12\23\0\12\2"+
+    "\u0773\0\u2ba4\12\u215c\0\u012e\13\2\13\73\13\225\13\7\12\14\0\5\12"+
+    "\5\0\1\12\1\0\12\12\1\0\15\12\1\0\5\12\1\0\1\12"+
+    "\1\0\2\12\1\0\2\12\1\0\154\12\41\0\u016b\12\22\0\100\12"+
+    "\2\0\66\12\50\0\14\12\164\0\5\12\1\0\207\12\23\0\12\2"+
     "\7\0\32\12\6\0\32\12\12\0\1\13\72\13\37\12\3\0\6\12"+
     "\2\0\6\12\2\0\6\12\2\0\3\12\43\0";
 
@@ -307,9 +316,6 @@ class StandardTokenizerImpl {
   /** the textposition at the last accepting state */
   private int zzMarkedPos;
 
-  /** the textposition at the last state to be included in yytext */
-  private int zzPushbackPos;
-
   /** the current text position in the buffer */
   private int zzCurrentPos;
 
@@ -340,6 +346,9 @@ class StandardTokenizerImpl {
   /** zzAtEOF == true <=> the scanner is at the EOF */
   private boolean zzAtEOF;
 
+  /** denotes if the user-EOF-code has already been executed */
+  private boolean zzEOFDone;
+
   /* user code: */
 
 public static final int ALPHANUM          = StandardTokenizer.ALPHANUM;
@@ -364,27 +373,32 @@ public final int yychar()
 }
 
 /**
- * Fills Lucene token with the current token text.
+ * Fills CharTermAttribute with the current token text.
  */
-final void getText(Token t) {
-  t.setTermBuffer(zzBuffer, zzStartRead, zzMarkedPos-zzStartRead);
+public final void getText(CharTermAttribute t) {
+  t.copyBuffer(zzBuffer, zzStartRead, zzMarkedPos-zzStartRead);
 }
 
 /**
- * Fills TermAttribute with the current token text.
+ * Resets the Tokenizer to a new Reader.
  */
-final void getText(TermAttribute t) {
-  t.setTermBuffer(zzBuffer, zzStartRead, zzMarkedPos-zzStartRead);
+public final void reset(Reader r) {
+  // reset to default buffer size, if buffer has grown
+  if (zzBuffer.length > ZZ_BUFFERSIZE) {
+    zzBuffer = new char[ZZ_BUFFERSIZE];
+  }
+  yyreset(r);
 }
 
 
+
   /**
    * Creates a new scanner
    * There is also a java.io.InputStream version of this constructor.
    *
    * @param   in  the java.io.Reader to read input from.
    */
-  StandardTokenizerImpl(java.io.Reader in) {
+  StandardTokenizerImpl31(java.io.Reader in) {
     this.zzReader = in;
   }
 
@@ -394,7 +408,7 @@ final void getText(TermAttribute t) {
    *
    * @param   in  the java.io.Inputstream to read input from.
    */
-  StandardTokenizerImpl(java.io.InputStream in) {
+  StandardTokenizerImpl31(java.io.InputStream in) {
     this(new java.io.InputStreamReader(in));
   }
 
@@ -408,7 +422,7 @@ final void getText(TermAttribute t) {
     char [] map = new char[0x10000];
     int i = 0;  /* index in packed string  */
     int j = 0;  /* index in unpacked array */
-    while (i < 1154) {
+    while (i < 1234) {
       int  count = packed.charAt(i++);
       char value = packed.charAt(i++);
       do map[j++] = value; while (--count > 0);
@@ -436,7 +450,6 @@ final void getText(TermAttribute t) {
       zzEndRead-= zzStartRead;
       zzCurrentPos-= zzStartRead;
       zzMarkedPos-= zzStartRead;
-      zzPushbackPos-= zzStartRead;
       zzStartRead = 0;
     }
 
@@ -452,13 +465,23 @@ final void getText(TermAttribute t) {
     int numRead = zzReader.read(zzBuffer, zzEndRead,
                                             zzBuffer.length-zzEndRead);
 
-    if (numRead < 0) {
-      return true;
-    }
-    else {
+    if (numRead > 0) {
       zzEndRead+= numRead;
       return false;
     }
+    // unlikely but not impossible: read 0 characters, but not at end of stream    
+    if (numRead == 0) {
+      int c = zzReader.read();
+      if (c == -1) {
+        return true;
+      } else {
+        zzBuffer[zzEndRead++] = (char) c;
+        return false;
+      }     
+    }
+
+	// numRead < 0
+    return true;
   }
 
     
@@ -488,8 +511,9 @@ final void getText(TermAttribute t) {
     zzReader = reader;
     zzAtBOL  = true;
     zzAtEOF  = false;
+    zzEOFDone = false;
     zzEndRead = zzStartRead = 0;
-    zzCurrentPos = zzMarkedPos = zzPushbackPos = 0;
+    zzCurrentPos = zzMarkedPos = 0;
     yyline = yychar = yycolumn = 0;
     zzLexicalState = YYINITIAL;
   }
@@ -619,7 +643,7 @@ final void getText(TermAttribute t) {
 
       zzCurrentPosL = zzCurrentPos = zzStartRead = zzMarkedPosL;
   
-      zzState = zzLexicalState;
+      zzState = ZZ_LEXSTATE[zzLexicalState];
 
 
       zzForAction: {
@@ -667,44 +691,44 @@ final void getText(TermAttribute t) {
       zzMarkedPos = zzMarkedPosL;
 
       switch (zzAction < 0 ? zzAction : ZZ_ACTION[zzAction]) {
-        case 4: 
-          { return HOST;
+        case 5: 
+          { return NUM;
           }
         case 11: break;
         case 9: 
           { return ACRONYM;
           }
         case 12: break;
-        case 8: 
-          { return ACRONYM_DEP;
+        case 7: 
+          { return COMPANY;
           }
         case 13: break;
-        case 1: 
-          { /* ignore */
+        case 10: 
+          { return EMAIL;
           }
         case 14: break;
-        case 5: 
-          { return NUM;
+        case 1: 
+          { /* ignore */
           }
         case 15: break;
-        case 3: 
-          { return CJ;
+        case 6: 
+          { return APOSTROPHE;
           }
         case 16: break;
-        case 2: 
-          { return ALPHANUM;
+        case 3: 
+          { return CJ;
           }
         case 17: break;
-        case 7: 
-          { return COMPANY;
+        case 8: 
+          { return ACRONYM_DEP;
           }
         case 18: break;
-        case 6: 
-          { return APOSTROPHE;
+        case 2: 
+          { return ALPHANUM;
           }
         case 19: break;
-        case 10: 
-          { return EMAIL;
+        case 4: 
+          { return HOST;
           }
         case 20: break;
         default: 

Copied: lucene/dev/trunk/lucene/src/java/org/apache/lucene/analysis/standard/StandardTokenizerImpl31.jflex
(from r886325, lucene/java/trunk/src/java/org/apache/lucene/analysis/standard/StandardTokenizerImpl.jflex)
URL: http://svn.apache.org/viewvc/lucene/dev/trunk/lucene/src/java/org/apache/lucene/analysis/standard/StandardTokenizerImpl31.jflex?p2=lucene/dev/trunk/lucene/src/java/org/apache/lucene/analysis/standard/StandardTokenizerImpl31.jflex&p1=lucene/java/trunk/src/java/org/apache/lucene/analysis/standard/StandardTokenizerImpl.jflex&r1=886325&r2=932163&rev=932163&view=diff
==============================================================================
--- lucene/java/trunk/src/java/org/apache/lucene/analysis/standard/StandardTokenizerImpl.jflex
(original)
+++ lucene/dev/trunk/lucene/src/java/org/apache/lucene/analysis/standard/StandardTokenizerImpl31.jflex
Thu Apr  8 22:56:23 2010
@@ -19,23 +19,19 @@ package org.apache.lucene.analysis.stand
 
 /*
 
-WARNING: if you change StandardTokenizerImpl.jflex and need to regenerate
-      the tokenizer, only use Java 1.4 !!!
-      This grammar currently uses constructs (eg :digit:, :letter:) whose 
-      meaning can vary according to the JRE used to run jflex.  See
-      https://issues.apache.org/jira/browse/LUCENE-1126 for details.
-      For current backwards compatibility it is needed to support
-      only Java 1.4 - this will change in Lucene 3.1.
+WARNING: if you change StandardTokenizerImpl*.jflex and need to regenerate
+      the tokenizer, only use the trunk version of JFlex 1.5 at the moment!
 
 */
 
-import org.apache.lucene.analysis.Token;
-import org.apache.lucene.analysis.tokenattributes.TermAttribute;
+import java.io.Reader;
+import org.apache.lucene.analysis.tokenattributes.CharTermAttribute;
 
 %%
 
-%class StandardTokenizerImpl
-%unicode
+%class StandardTokenizerImpl31
+%implements StandardTokenizerInterface
+%unicode 4.0
 %integer
 %function getNextToken
 %pack
@@ -65,17 +61,21 @@ public final int yychar()
 }
 
 /**
- * Fills Lucene token with the current token text.
+ * Fills CharTermAttribute with the current token text.
  */
-final void getText(Token t) {
-  t.setTermBuffer(zzBuffer, zzStartRead, zzMarkedPos-zzStartRead);
+public final void getText(CharTermAttribute t) {
+  t.copyBuffer(zzBuffer, zzStartRead, zzMarkedPos-zzStartRead);
 }
 
 /**
- * Fills TermAttribute with the current token text.
+ * Resets the Tokenizer to a new Reader.
  */
-final void getText(TermAttribute t) {
-  t.setTermBuffer(zzBuffer, zzStartRead, zzMarkedPos-zzStartRead);
+public final void reset(Reader r) {
+  // reset to default buffer size, if buffer has grown
+  if (zzBuffer.length > ZZ_BUFFERSIZE) {
+    zzBuffer = new char[ZZ_BUFFERSIZE];
+  }
+  yyreset(r);
 }
 
 %}

Copied: lucene/dev/trunk/lucene/src/java/org/apache/lucene/analysis/standard/StandardTokenizerImplOrig.java
(from r886325, lucene/java/trunk/src/java/org/apache/lucene/analysis/standard/StandardTokenizerImpl.java)
URL: http://svn.apache.org/viewvc/lucene/dev/trunk/lucene/src/java/org/apache/lucene/analysis/standard/StandardTokenizerImplOrig.java?p2=lucene/dev/trunk/lucene/src/java/org/apache/lucene/analysis/standard/StandardTokenizerImplOrig.java&p1=lucene/java/trunk/src/java/org/apache/lucene/analysis/standard/StandardTokenizerImpl.java&r1=886325&r2=932163&rev=932163&view=diff
==============================================================================
--- lucene/java/trunk/src/java/org/apache/lucene/analysis/standard/StandardTokenizerImpl.java
(original)
+++ lucene/dev/trunk/lucene/src/java/org/apache/lucene/analysis/standard/StandardTokenizerImplOrig.java
Thu Apr  8 22:56:23 2010
@@ -1,4 +1,4 @@
-/* The following code was generated by JFlex 1.4.1 on 9/4/08 6:49 PM */
+/* The following code was generated by JFlex 1.5.0-SNAPSHOT on 09.04.10 00:10 */
 
 package org.apache.lucene.analysis.standard;
 
@@ -21,27 +21,22 @@ package org.apache.lucene.analysis.stand
 
 /*
 
-WARNING: if you change StandardTokenizerImpl.jflex and need to regenerate
-      the tokenizer, only use Java 1.4 !!!
-      This grammar currently uses constructs (eg :digit:, :letter:) whose 
-      meaning can vary according to the JRE used to run jflex.  See
-      https://issues.apache.org/jira/browse/LUCENE-1126 for details.
-      For current backwards compatibility it is needed to support
-      only Java 1.4 - this will change in Lucene 3.1.
+WARNING: if you change StandardTokenizerImpl*.jflex and need to regenerate
+      the tokenizer, only use the trunk version of JFlex 1.5 at the moment!
 
 */
 
-import org.apache.lucene.analysis.Token;
-import org.apache.lucene.analysis.tokenattributes.TermAttribute;
+import java.io.Reader;
+import org.apache.lucene.analysis.tokenattributes.CharTermAttribute;
 
 
 /**
  * This class is a scanner generated by 
- * <a href="http://www.jflex.de/">JFlex</a> 1.4.1
- * on 9/4/08 6:49 PM from the specification file
- * <tt>/tango/mike/src/lucene.standarddigit/src/java/org/apache/lucene/analysis/standard/StandardTokenizerImpl.jflex</tt>
+ * <a href="http://www.jflex.de/">JFlex</a> 1.5.0-SNAPSHOT
+ * on 09.04.10 00:10 from the specification file
+ * <tt>C:/Users/Uwe Schindler/Projects/lucene/trunk-full1/src/java/org/apache/lucene/analysis/standard/StandardTokenizerImplOrig.jflex</tt>
  */
-class StandardTokenizerImpl {
+class StandardTokenizerImplOrig implements StandardTokenizerInterface {
 
   /** This character denotes the end of file */
   public static final int YYEOF = -1;
@@ -52,6 +47,16 @@ class StandardTokenizerImpl {
   /** lexical states */
   public static final int YYINITIAL = 0;
 
+  /**
+   * ZZ_LEXSTATE[l] is the state in the DFA for the lexical state l
+   * ZZ_LEXSTATE[l+1] is the state in the DFA for the lexical state l
+   *                  at the beginning of a line
+   * l is of the form l = 2*k, k a non negative integer
+   */
+  private static final int ZZ_LEXSTATE[] = { 
+     0, 0
+  };
+
   /** 
    * Translates characters to character classes
    */
@@ -307,9 +312,6 @@ class StandardTokenizerImpl {
   /** the textposition at the last accepting state */
   private int zzMarkedPos;
 
-  /** the textposition at the last state to be included in yytext */
-  private int zzPushbackPos;
-
   /** the current text position in the buffer */
   private int zzCurrentPos;
 
@@ -340,6 +342,9 @@ class StandardTokenizerImpl {
   /** zzAtEOF == true <=> the scanner is at the EOF */
   private boolean zzAtEOF;
 
+  /** denotes if the user-EOF-code has already been executed */
+  private boolean zzEOFDone;
+
   /* user code: */
 
 public static final int ALPHANUM          = StandardTokenizer.ALPHANUM;
@@ -364,27 +369,32 @@ public final int yychar()
 }
 
 /**
- * Fills Lucene token with the current token text.
+ * Fills CharTermAttribute with the current token text.
  */
-final void getText(Token t) {
-  t.setTermBuffer(zzBuffer, zzStartRead, zzMarkedPos-zzStartRead);
+public final void getText(CharTermAttribute t) {
+  t.copyBuffer(zzBuffer, zzStartRead, zzMarkedPos-zzStartRead);
 }
 
 /**
- * Fills TermAttribute with the current token text.
+ * Resets the Tokenizer to a new Reader.
  */
-final void getText(TermAttribute t) {
-  t.setTermBuffer(zzBuffer, zzStartRead, zzMarkedPos-zzStartRead);
+public final void reset(Reader r) {
+  // reset to default buffer size, if buffer has grown
+  if (zzBuffer.length > ZZ_BUFFERSIZE) {
+    zzBuffer = new char[ZZ_BUFFERSIZE];
+  }
+  yyreset(r);
 }
 
 
+
   /**
    * Creates a new scanner
    * There is also a java.io.InputStream version of this constructor.
    *
    * @param   in  the java.io.Reader to read input from.
    */
-  StandardTokenizerImpl(java.io.Reader in) {
+  StandardTokenizerImplOrig(java.io.Reader in) {
     this.zzReader = in;
   }
 
@@ -394,7 +404,7 @@ final void getText(TermAttribute t) {
    *
    * @param   in  the java.io.Inputstream to read input from.
    */
-  StandardTokenizerImpl(java.io.InputStream in) {
+  StandardTokenizerImplOrig(java.io.InputStream in) {
     this(new java.io.InputStreamReader(in));
   }
 
@@ -436,7 +446,6 @@ final void getText(TermAttribute t) {
       zzEndRead-= zzStartRead;
       zzCurrentPos-= zzStartRead;
       zzMarkedPos-= zzStartRead;
-      zzPushbackPos-= zzStartRead;
       zzStartRead = 0;
     }
 
@@ -452,13 +461,23 @@ final void getText(TermAttribute t) {
     int numRead = zzReader.read(zzBuffer, zzEndRead,
                                             zzBuffer.length-zzEndRead);
 
-    if (numRead < 0) {
-      return true;
-    }
-    else {
+    if (numRead > 0) {
       zzEndRead+= numRead;
       return false;
     }
+    // unlikely but not impossible: read 0 characters, but not at end of stream    
+    if (numRead == 0) {
+      int c = zzReader.read();
+      if (c == -1) {
+        return true;
+      } else {
+        zzBuffer[zzEndRead++] = (char) c;
+        return false;
+      }     
+    }
+
+	// numRead < 0
+    return true;
   }
 
     
@@ -488,8 +507,9 @@ final void getText(TermAttribute t) {
     zzReader = reader;
     zzAtBOL  = true;
     zzAtEOF  = false;
+    zzEOFDone = false;
     zzEndRead = zzStartRead = 0;
-    zzCurrentPos = zzMarkedPos = zzPushbackPos = 0;
+    zzCurrentPos = zzMarkedPos = 0;
     yyline = yychar = yycolumn = 0;
     zzLexicalState = YYINITIAL;
   }
@@ -619,7 +639,7 @@ final void getText(TermAttribute t) {
 
       zzCurrentPosL = zzCurrentPos = zzStartRead = zzMarkedPosL;
   
-      zzState = zzLexicalState;
+      zzState = ZZ_LEXSTATE[zzLexicalState];
 
 
       zzForAction: {
@@ -667,44 +687,44 @@ final void getText(TermAttribute t) {
       zzMarkedPos = zzMarkedPosL;
 
       switch (zzAction < 0 ? zzAction : ZZ_ACTION[zzAction]) {
-        case 4: 
-          { return HOST;
+        case 5: 
+          { return NUM;
           }
         case 11: break;
         case 9: 
           { return ACRONYM;
           }
         case 12: break;
-        case 8: 
-          { return ACRONYM_DEP;
+        case 7: 
+          { return COMPANY;
           }
         case 13: break;
-        case 1: 
-          { /* ignore */
+        case 10: 
+          { return EMAIL;
           }
         case 14: break;
-        case 5: 
-          { return NUM;
+        case 1: 
+          { /* ignore */
           }
         case 15: break;
-        case 3: 
-          { return CJ;
+        case 6: 
+          { return APOSTROPHE;
           }
         case 16: break;
-        case 2: 
-          { return ALPHANUM;
+        case 3: 
+          { return CJ;
           }
         case 17: break;
-        case 7: 
-          { return COMPANY;
+        case 8: 
+          { return ACRONYM_DEP;
           }
         case 18: break;
-        case 6: 
-          { return APOSTROPHE;
+        case 2: 
+          { return ALPHANUM;
           }
         case 19: break;
-        case 10: 
-          { return EMAIL;
+        case 4: 
+          { return HOST;
           }
         case 20: break;
         default: 

Copied: lucene/dev/trunk/lucene/src/java/org/apache/lucene/analysis/standard/StandardTokenizerImplOrig.jflex
(from r886325, lucene/java/trunk/src/java/org/apache/lucene/analysis/standard/StandardTokenizerImpl.jflex)
URL: http://svn.apache.org/viewvc/lucene/dev/trunk/lucene/src/java/org/apache/lucene/analysis/standard/StandardTokenizerImplOrig.jflex?p2=lucene/dev/trunk/lucene/src/java/org/apache/lucene/analysis/standard/StandardTokenizerImplOrig.jflex&p1=lucene/java/trunk/src/java/org/apache/lucene/analysis/standard/StandardTokenizerImpl.jflex&r1=886325&r2=932163&rev=932163&view=diff
==============================================================================
--- lucene/java/trunk/src/java/org/apache/lucene/analysis/standard/StandardTokenizerImpl.jflex
(original)
+++ lucene/dev/trunk/lucene/src/java/org/apache/lucene/analysis/standard/StandardTokenizerImplOrig.jflex
Thu Apr  8 22:56:23 2010
@@ -19,23 +19,19 @@ package org.apache.lucene.analysis.stand
 
 /*
 
-WARNING: if you change StandardTokenizerImpl.jflex and need to regenerate
-      the tokenizer, only use Java 1.4 !!!
-      This grammar currently uses constructs (eg :digit:, :letter:) whose 
-      meaning can vary according to the JRE used to run jflex.  See
-      https://issues.apache.org/jira/browse/LUCENE-1126 for details.
-      For current backwards compatibility it is needed to support
-      only Java 1.4 - this will change in Lucene 3.1.
+WARNING: if you change StandardTokenizerImpl*.jflex and need to regenerate
+      the tokenizer, only use the trunk version of JFlex 1.5 at the moment!
 
 */
 
-import org.apache.lucene.analysis.Token;
-import org.apache.lucene.analysis.tokenattributes.TermAttribute;
+import java.io.Reader;
+import org.apache.lucene.analysis.tokenattributes.CharTermAttribute;
 
 %%
 
-%class StandardTokenizerImpl
-%unicode
+%class StandardTokenizerImplOrig
+%implements StandardTokenizerInterface
+%unicode 3.0
 %integer
 %function getNextToken
 %pack
@@ -65,17 +61,21 @@ public final int yychar()
 }
 
 /**
- * Fills Lucene token with the current token text.
+ * Fills CharTermAttribute with the current token text.
  */
-final void getText(Token t) {
-  t.setTermBuffer(zzBuffer, zzStartRead, zzMarkedPos-zzStartRead);
+public final void getText(CharTermAttribute t) {
+  t.copyBuffer(zzBuffer, zzStartRead, zzMarkedPos-zzStartRead);
 }
 
 /**
- * Fills TermAttribute with the current token text.
+ * Resets the Tokenizer to a new Reader.
  */
-final void getText(TermAttribute t) {
-  t.setTermBuffer(zzBuffer, zzStartRead, zzMarkedPos-zzStartRead);
+public final void reset(Reader r) {
+  // reset to default buffer size, if buffer has grown
+  if (zzBuffer.length > ZZ_BUFFERSIZE) {
+    zzBuffer = new char[ZZ_BUFFERSIZE];
+  }
+  yyreset(r);
 }
 
 %}

Added: lucene/dev/trunk/lucene/src/java/org/apache/lucene/analysis/standard/StandardTokenizerInterface.java
URL: http://svn.apache.org/viewvc/lucene/dev/trunk/lucene/src/java/org/apache/lucene/analysis/standard/StandardTokenizerInterface.java?rev=932163&view=auto
==============================================================================
--- lucene/dev/trunk/lucene/src/java/org/apache/lucene/analysis/standard/StandardTokenizerInterface.java
(added)
+++ lucene/dev/trunk/lucene/src/java/org/apache/lucene/analysis/standard/StandardTokenizerInterface.java
Thu Apr  8 22:56:23 2010
@@ -0,0 +1,66 @@
+package org.apache.lucene.analysis.standard;
+
+/**
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements.  See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License.  You may obtain a copy of the License at
+ *
+ *     http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+import org.apache.lucene.analysis.tokenattributes.CharTermAttribute;
+
+import java.io.Reader;
+import java.io.IOException;
+
+interface StandardTokenizerInterface {
+
+  /** This character denotes the end of file */
+  public static final int YYEOF = -1;
+
+  /**
+   * Copies the matched text into the CharTermAttribute
+   */
+  void getText(CharTermAttribute t);
+
+  /**
+   * Returns the current position.
+   */
+  int yychar();
+
+  /**
+   * Resets the scanner to read from a new input stream.
+   * Does not close the old reader.
+   *
+   * All internal variables are reset, the old input stream 
+   * <b>cannot</b> be reused (internal buffer is discarded and lost).
+   * Lexical state is set to <tt>ZZ_INITIAL</tt>.
+   *
+   * @param reader   the new input stream 
+   */
+  void reset(Reader reader);
+
+  /**
+   * Returns the length of the matched text region.
+   */
+  int yylength();
+
+  /**
+   * Resumes scanning until the next regular expression is matched,
+   * the end of input is encountered or an I/O-Error occurs.
+   *
+   * @return      the next token, {@link #YYEOF} on end of stream
+   * @exception   IOException  if any I/O-Error occurs
+   */
+  int getNextToken() throws IOException;
+
+}

Propchange: lucene/dev/trunk/lucene/src/java/org/apache/lucene/analysis/standard/StandardTokenizerInterface.java
------------------------------------------------------------------------------
    svn:eol-style = native

Propchange: lucene/dev/trunk/lucene/src/java/org/apache/lucene/analysis/standard/StandardTokenizerInterface.java
------------------------------------------------------------------------------
    svn:keywords = Date Author Id Revision HeadURL

Modified: lucene/dev/trunk/lucene/src/test/org/apache/lucene/analysis/TestStandardAnalyzer.java
URL: http://svn.apache.org/viewvc/lucene/dev/trunk/lucene/src/test/org/apache/lucene/analysis/TestStandardAnalyzer.java?rev=932163&r1=932162&r2=932163&view=diff
==============================================================================
--- lucene/dev/trunk/lucene/src/test/org/apache/lucene/analysis/TestStandardAnalyzer.java
(original)
+++ lucene/dev/trunk/lucene/src/test/org/apache/lucene/analysis/TestStandardAnalyzer.java
Thu Apr  8 22:56:23 2010
@@ -224,4 +224,12 @@ public class TestStandardAnalyzer extend
                     "<ALPHANUM>", "<NUM>", "<HOST>", "<NUM>", "<ALPHANUM>",
                     "<ALPHANUM>", "<HOST>"});
   }
+
+  public void testJava14BWCompatibility() throws Exception {
+    StandardAnalyzer sa = new StandardAnalyzer(Version.LUCENE_30);
+    assertAnalyzesTo(sa, "test\u02C6test", new String[] { "test", "test" });
+    sa = new StandardAnalyzer(Version.LUCENE_31);
+    assertAnalyzesTo(sa, "test\u02C6test", new String[] { "test\u02C6test" });
+  }
+
 }



Mime
View raw message