Return-Path: X-Original-To: apmail-manifoldcf-dev-archive@www.apache.org Delivered-To: apmail-manifoldcf-dev-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 371C617296 for ; Tue, 28 Oct 2014 11:40:34 +0000 (UTC) Received: (qmail 89205 invoked by uid 500); 28 Oct 2014 11:40:34 -0000 Delivered-To: apmail-manifoldcf-dev-archive@manifoldcf.apache.org Received: (qmail 89151 invoked by uid 500); 28 Oct 2014 11:40:34 -0000 Mailing-List: contact dev-help@manifoldcf.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@manifoldcf.apache.org Delivered-To: mailing list dev@manifoldcf.apache.org Received: (qmail 89129 invoked by uid 99); 28 Oct 2014 11:40:34 -0000 Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 28 Oct 2014 11:40:34 +0000 Date: Tue, 28 Oct 2014 11:40:34 +0000 (UTC) From: "Karl Wright (JIRA)" To: dev@manifoldcf.apache.org Message-ID: In-Reply-To: References: Subject: [jira] [Commented] (CONNECTORS-1088) Augment Tika extractor to allow full use of boilerpipe content extraction MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: 7bit X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394 [ https://issues.apache.org/jira/browse/CONNECTORS-1088?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14186713#comment-14186713 ] Karl Wright commented on CONNECTORS-1088: ----------------------------------------- r1634850 (trunk) : first part (functionality, not UI yet) r1634851 (dev_1x) > Augment Tika extractor to allow full use of boilerpipe content extraction > ------------------------------------------------------------------------- > > Key: CONNECTORS-1088 > URL: https://issues.apache.org/jira/browse/CONNECTORS-1088 > Project: ManifoldCF > Issue Type: Improvement > Components: Tika extractor > Affects Versions: ManifoldCF 1.8, ManifoldCF 2.0 > Reporter: Karl Wright > Assignee: Karl Wright > Fix For: ManifoldCF 1.8, ManifoldCF 2.0 > > > Boilerpipe has the ability to process content further than our current Tika extractor implementation allows. Specifically, we should be allowing a user to specify a BoilerPipe extractor class, from within the following package (or other places too, one expects): > http://boilerpipe.googlecode.com/svn/trunk/boilerpipe-core/javadoc/1.0/de/l3s/boilerpipe/extractors/package-summary.html > If the extractor is specified, then our ContentHandler creation code in the Tika extractor changes from: > {code} > ContentHandler handler = new BodyContentHandler(w); > {code} > to: > {code} > ContentHandler handler = new BodyContentHandler(w); > boilerpipe = "de.l3s.boilerpipe.extractors." + boilerpipe; > try { > ClassLoader loader = BoilerpipeExtractor.class.getClassLoader(); > Class extractorClass = loader.loadClass(boilerpipe); > BoilerpipeExtractor boilerpipeExtractor = (BoilerpipeExtractor)extractorClass.newInstance(); > handler = new BoilerpipeContentHandler(handler, boilerpipeExtractor); > } catch (ClassNotFoundException e) { > log.warn("BoilerpipeExtractor " + boilerpipe + " not found!"); > } catch (InstantiationException e) { > log.warn("Could not instantiate " + boilerpipe); > } catch (Exception e) { > log.warn(e.toString()); > } > {code} -- This message was sent by Atlassian JIRA (v6.3.4#6332)