manifoldcf-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Karl Wright (JIRA)" <>
Subject [jira] [Commented] (CONNECTORS-962) Support multiple output connections for a single job
Date Thu, 12 Jun 2014 10:25:02 GMT


Karl Wright commented on CONNECTORS-962:

Hi Rafa,
Our problem is that pipelining in ManifoldCF must operate in a manner consistent with database
storage and with incremental crawling.  It's not just a pipeline.

I've looked at what needs to be done to support multiple output connectors and it is actually
relatively straightforward.  Since I've already disrupted things by introducing the first
order of pipeline, I might as well finish the job now.  If something like Camel is suited
to this picture, it would be as an implementation detail, and hidden in the internals anyway.

> Support multiple output connections for a single job
> ----------------------------------------------------
>                 Key: CONNECTORS-962
>                 URL:
>             Project: ManifoldCF
>          Issue Type: Improvement
>          Components: Framework crawler agent
>    Affects Versions: ManifoldCF 1.7
>            Reporter: Karl Wright
>            Assignee: Karl Wright
>             Fix For: ManifoldCF 1.7
> Zaizi has a requirement to support multiple outputs for a single job.  In theory this
requirement can be met by doing the following:
> - Allow multiple output connections, and multiple pipelines, per job
> - Keep a distinct ingeststatus record for each document/output combination
> - Modify WorkerThread to call IncrementalIndexer multiple times for every document fetched
> Places where different things need to happen are:
> - RepositoryDocument - because one binary stream will not do for multiple outputs
> - UI, obviously, because there will need to be multiple pipelines, not just one, and
in addition it would be probably important to be able to "split" the pipeline at arbitrary

This message was sent by Atlassian JIRA

View raw message