manifoldcf-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Karl Wright (JIRA)" <>
Subject [jira] [Created] (CONNECTORS-962) Support multiple output connections for a single job
Date Wed, 11 Jun 2014 16:29:01 GMT
Karl Wright created CONNECTORS-962:

             Summary: Support multiple output connections for a single job
                 Key: CONNECTORS-962
             Project: ManifoldCF
          Issue Type: Improvement
          Components: Framework crawler agent
    Affects Versions: ManifoldCF 1.7
            Reporter: Karl Wright
            Assignee: Karl Wright
             Fix For: ManifoldCF 1.7

Zaizi has a requirement to support multiple outputs for a single job.  In theory this requirement
can be met by doing the following:

- Allow multiple output connections, and multiple pipelines, per job
- Keep a distinct ingeststatus record for each document/output combination
- Modify WorkerThread to call IncrementalIndexer multiple times for every document fetched

Places where different things need to happen are:
- RepositoryDocument - because one binary stream will not do for multiple outputs
- UI, obviously, because there will need to be multiple pipelines, not just one, and in addition
it would be probably important to be able to "split" the pipeline at arbitrary points

This message was sent by Atlassian JIRA

View raw message