lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Uwe Schindler (JIRA)" <>
Subject [jira] Commented: (LUCENE-2731) HyphenationCompoundWordTokenFilter fails to load DTD in Crimson parser (JDK 1.4)
Date Sun, 31 Oct 2010 10:05:23 GMT


Uwe Schindler commented on LUCENE-2731:

By the way, the whole loading in the TokenFilter is broken! XML files should never be load
by a Reaer, always by InputStream. The charset detection is part of the XML spec and the default
is UTF-8 if not overridden in the XML file. The HyphenationCompoundWordTokenFilter supplies
ISO-8859-1 as fixed for the reader, so it is never possible to load other xml files with different
charsets. This is another issue, I will open for 3.x and trunk (as it needs API change).

> HyphenationCompoundWordTokenFilter fails to load DTD in Crimson parser (JDK 1.4)
> --------------------------------------------------------------------------------
>                 Key: LUCENE-2731
>                 URL:
>             Project: Lucene - Java
>          Issue Type: Bug
>          Components: contrib/analyzers
>            Reporter: Uwe Schindler
>            Assignee: Uwe Schindler
>             Fix For: 2.9.4
>         Attachments: LUCENE-2731.patch
> HyphenationCompoundWordTokenFilter loads the DTD in its XML parser from memory by supplying
EntityResolver. In Java 1.4 (affects Lucene 2.9, but also later versions if not Apache Xerces
is used as XML parser) this does not work, because Cromson does not even ask the entity resolver,
if no base URI is known. As the hyphenation file is loaded from Reader/InputStream no base
URI is known. Crimson needs at least a non-null systemId to proceed.
> This patch (Lucene 2.9 only)  fakes this by supplying a fake systemId to the InputSource.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message