lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Keith Sprochi (JIRA)" <j...@apache.org>
Subject [jira] Created: (LUCENE-1704) org.apache.lucene.ant.HtmlDocument added Tidy config file passthrough availability
Date Fri, 19 Jun 2009 18:54:07 GMT
org.apache.lucene.ant.HtmlDocument added Tidy config file passthrough availability
----------------------------------------------------------------------------------

                 Key: LUCENE-1704
                 URL: https://issues.apache.org/jira/browse/LUCENE-1704
             Project: Lucene - Java
          Issue Type: Improvement
          Components: contrib/*
    Affects Versions: 2.4.1
            Reporter: Keith Sprochi
            Priority: Trivial


Parsing HTML documents using the org.apache.lucene.ant.HtmlDocument.Document method resulted
in many error messages such as this:

    line 152 column 725 - Error: <as-html> is not recognized!
    This document has errors that must be fixed before
    using HTML Tidy to generate a tidied up version.

The solution is to configure Tidy to accept these abnormal tags by adding the tag name to
the "new-inline-tags" option in the Tidy config file (or the command line which does not make
sense in this context), like so:

    new-inline-tags: as-html

Tidy needs to know where the configuration file is, so a new constructor and Document method
can be added.  Here is the code:


    /**                                                                                  
                                                                                         
               
     *  Constructs an <code>HtmlDocument</code> from a {@link                
                                                                                         
                           
     *  java.io.File}.                                                                   
                                                                                         
               
     *                                                                                   
                                                                                         
               
     *@param  file             the <code>File</code> containing the          
                                                                                         
                           
     *      HTML to parse                                                                
                                                                                         
               
     *@param  tidyConfigFile   the <code>String</code> containing            
                                                                                         
                           
     *      the full path to the Tidy config file                                        
                                                                                         
               
     *@exception  IOException  if an I/O exception occurs                                
                                                                                         
               
     */
    public HtmlDocument(File file, String tidyConfigFile) throws IOException {
        Tidy tidy = new Tidy();
        tidy.setConfigurationFromFile(tidyConfigFile);
        tidy.setQuiet(true);
        tidy.setShowWarnings(false);
        org.w3c.dom.Document root =
                tidy.parseDOM(new FileInputStream(file), null);
        rawDoc = root.getDocumentElement();
    }

    /**                                                                                  
                                                                                         
               
     *  Creates a Lucene <code>Document</code> from a {@link                 
                                                                                         
                           
     *  java.io.File}.                                                                   
                                                                                         
               
     *                                                                                   
                                                                                         
               
     *@param  file                                                                       
                                                                                         
               
     *@param  tidyConfigFile the full path to the Tidy config file                       
                                                                                         
               
     *@exception  IOException                                                            
                                                                                         
               
     */
    public static org.apache.lucene.document.Document
        Document(File file, String tidyConfigFile) throws IOException {

        HtmlDocument htmlDoc = new HtmlDocument(file, tidyConfigFile);

        org.apache.lucene.document.Document luceneDoc = new org.apache.lucene.document.Document();

        luceneDoc.add(new Field("title", htmlDoc.getTitle(), Field.Store.YES, Field.Index.ANALYZED));
        luceneDoc.add(new Field("contents", htmlDoc.getBody(), Field.Store.YES, Field.Index.ANALYZED));

        String contents = null;
        BufferedReader br =
            new BufferedReader(new FileReader(file));
        StringWriter sw = new StringWriter();
        String line = br.readLine();
        while (line != null) {
            sw.write(line);
            line = br.readLine();
        }
        br.close();
        contents = sw.toString();
        sw.close();

        luceneDoc.add(new Field("rawcontents", contents, Field.Store.YES, Field.Index.NO));

        return luceneDoc;
    }


I am using this now and it is working fine.  The configuration file is being passed to Tidy
and now I am able to index thousands of HTML pages with no more Tidy tag errors.



-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org


Mime
View raw message