Mailing-List: contact user-help@manifoldcf.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@manifoldcf.apache.org
Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Mac OS X Mail 6.5 \(1508\))
Subject: Re: Extracting Content from Web Crawler using the new PipeLine
From: Shinichiro Abe <shinichiro.abe.1@gmail.com>
In-Reply-To: 
 <CAJn2xr-q1du079stziP_S4jc0P_RhCs8icrCAmHd79j1h0k_DA@mail.gmail.com>
Date: Thu, 23 Oct 2014 15:03:41 +0900
Reply-To: "user@manifoldcf.apache.org" <user@manifoldcf.apache.org>
Content-Transfer-Encoding: quoted-printable
Message-Id: <E69C5E0B-A2A3-4136-B22B-CD1AE5823262@gmail.com>
References: 
 <CAJn2xr-q1du079stziP_S4jc0P_RhCs8icrCAmHd79j1h0k_DA@mail.gmail.com>
To: user@manifoldcf.apache.org

Hi Arcadius,

> - use Tika's BoilerPipe to get cleaner content from web sites?
Yes, Tika extractor will remove tags in html
and send content and metadata to downstream pipeline/output connection.

> - What about extracting specific HTML tags such as all h1 or h2 and =
map them to a Solr field?
No, currently it can map only metadata which is extracted by Tika to =
Solr field.
For h1, h2, p tags etc,  Tika extractor doesn't capture them and doesn't =
treat them as metadata.
Currently when capturing these tags and map them to fields,=20
we have to use Solr's ExtractingRequestHandler(CAPTURE_ELEMENTS param).

Regards,
Shinichiro Abe

On 2014/10/23, at 10:21, Arcadius Ahouansou <arcadius@menelic.com> =
wrote:

>=20
> Hello.
>=20
> Given that we now have pipelines in ManifoldCF, How feasible  is it =
to:
>=20
> - use Tika's BoilerPipe to get cleaner content from web sites?
> - What about extracting specific HTML tags such as all h1 or h2 and =
map them to a Solr field?
>=20
> Thank you very much.
>=20
> Arcadius.
>  =20