Mailing-List: contact java-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-user@lucene.apache.org
Received-SPF: pass (nike.apache.org: local policy)
DomainKey-Signature: a=rsa-sha1; q=dns; c=nofws;
  s=s1024; d=yahoo.com;
  h=Message-ID:X-YMail-OSG:Received:X-Mailer:References:Date:From:Subject:To:In-Reply-To:MIME-Version:Content-Type;
  b=QO664fF9jZQc0u9YwaM/6WJQ1A5gK9exRYk/3WO/zm1vgoU2qQlqtihGU7A18R63tm91yJ1r000o5pDWr0AxIi9lbZcF67pYtpjqScsEIB0IdGS+a5bp0/9nDybLIk9ue/7XyXkFgxO3ppHnwJK9kpgqvhMK9R94KFjoTV+LLss=;
Message-ID: <111835.48056.qm@web32903.mail.mud.yahoo.com>
References: <43985.50280.qm@web32907.mail.mud.yahoo.com>
 <h2w8f0ad1f31004240703t8f0b12a9j98eaf1f48e3434c7@mail.gmail.com>
Date: Mon, 26 Apr 2010 16:15:56 -0700 (PDT)
From: Justin <crynax@yahoo.com>
Subject: Re: HTMLStripReader, HTMLStripCharFilter
To: java-user@lucene.apache.org
In-Reply-To: <h2w8f0ad1f31004240703t8f0b12a9j98eaf1f48e3434c7@mail.gmail.com>
MIME-Version: 1.0
Content-Type: text/plain; charset=us-ascii

Thanks for the update!  I appreciate the hard work.

Perhaps someone can help me with the use of HTMLStripCharFilter...


I get an exception (3.1-dev) similar to the one reported here (2.9):

https://issues.apache.org/jira/browse/LUCENE-1695


With the following code:

    Analyzer htmlStripAnalyzer = new ReusableAnalyzerBase() {
        @Override
        protected TokenStreamComponents createComponents(
                final String fieldName, final Reader reader) {
            return new TokenStreamComponents(new StandardTokenizer(Version.LUCENE_30,
                    new HTMLStripCharFilter(CharReader.get(reader))));
        }
    };
    String content = reader.document(id, fieldSelector).get(field);
    TokenStream ts = htmlStripAnalyzer.tokenStream(field, new StringReader(content));
    String best = highlighter.getBestFragments(ts, content,
      DEFAULT_EXCERPT_FRAGS, DEFAULT_EXCERPT_SEPARATOR);
    OffsetAttribute off = ts.addAttribute(OffsetAttribute.class);
    ts.reset();
    ts.incrementToken();


java.io.IOException: Stream closed
        at java.io.StringReader.ensureOpen(StringReader.java:39)
        at java.io.StringReader.read(StringReader.java:73)
        at org.apache.lucene.analysis.CharReader.read(CharReader.java:54)
        at java.io.Reader.read(Reader.java:104)
        at org.apache.solr.analysis.HTMLStripCharFilter.next(HTMLStripCharFilter.java:92)
        at org.apache.solr.analysis.HTMLStripCharFilter.read(HTMLStripCharFilter.java:690)
        at org.apache.solr.analysis.HTMLStripCharFilter.read(HTMLStripCharFilter.java:748)
        at org.apache.lucene.analysis.standard.StandardTokenizerImpl.zzRefill(StandardTokenizerImpl.java:453)
        at org.apache.lucene.analysis.standard.StandardTokenizerImpl.getNextToken(StandardTokenizerImpl.java:639)
        at org.apache.lucene.analysis.standard.StandardTokenizer.incrementToken(StandardTokenizer.java:167)


Looking at the source, I wonder if Tokenizer should override reset():

  public void reset() throws IOException {
    if (input != null) input.reset(); // would reset CharReader, StringReader
  }


----- Original Message ----
From: Robert Muir <rcmuir@gmail.com>
To: java-user@lucene.apache.org
Sent: Sat, April 24, 2010 9:03:02 AM
Subject: Re: HTMLStripReader, HTMLStripCharFilter

On Fri, Apr 23, 2010 at 4:48 PM, Justin <crynax@yahoo.com> wrote:

> Just out of curiousity, why does LUCENE-1377 have a minor priorty?
>
> https://issues.apache.org/jira/browse/LUCENE-1377
>
> Don't people index, filter, search HTML, perhaps more than any other
> format?
>
>
Rest assured we are working on this... but it unfortunately won't happen
overnight. First of all, the development of Lucene and Solr was merged such
that there is now one team working on this stuff. This way, both Solr and
Lucene developers can maintain this stuff.

There is now the practical issue to combine all Lucene and Solr analyzers
(not just the two components listed on that issue) into one package that can
then be used by both Lucene and Solr users:
https://issues.apache.org/jira/browse/LUCENE-2413

-- 
Robert Muir
rcmuir@gmail.com


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org