Mailing-List: contact java-dev-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-dev@lucene.apache.org
Message-ID: <1482465679.1257338132439.JavaMail.jira@brutus>
Date: Wed, 4 Nov 2009 12:35:32 +0000 (UTC)
From: "Michael McCandless (JIRA)" <jira@apache.org>
To: java-dev@lucene.apache.org
Subject: [jira] Commented: (LUCENE-1879) Parallel incremental indexing
In-Reply-To: <1995931318.1251705701427.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/LUCENE-1879?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12773466#action_12773466 ] 

Michael McCandless commented on LUCENE-1879:
--------------------------------------------

I wonder if we could change Lucene's index format to make this feature
simpler to implement...

Ie, you're having to go to great lengths (since this is built
"outside" of Lucene's core) to force multiple separate indexes to
share everything but the postings files (merge choices, flush,
deletions files, segments files, turning off the stores, etc.).

What if we could invert this approach, so that we use only single
index/IndexWriter, but we allow "partitioned postings", where sets of
fields are mapped to different postings files in the segment?

Whenever a doc is indexed, postings from the fields are then written
according to this partition.  Eg if I map "body" to partition 1, and
"title" to partition 2, then I'd have two sets of postings files for
each segment.

Could something like this work?

> Parallel incremental indexing
> -----------------------------
>
>                 Key: LUCENE-1879
>                 URL: https://issues.apache.org/jira/browse/LUCENE-1879
>             Project: Lucene - Java
>          Issue Type: New Feature
>          Components: Index
>            Reporter: Michael Busch
>            Assignee: Michael Busch
>             Fix For: 3.1
>
>         Attachments: parallel_incremental_indexing.tar
>
>
> A new feature that allows building parallel indexes and keeping them in sync on a docID level, independent of the choice of the MergePolicy/MergeScheduler.
> Find details on the wiki page for this feature:
> http://wiki.apache.org/lucene-java/ParallelIncrementalIndexing 
> Discussion on java-dev:
> http://markmail.org/thread/ql3oxzkob7aqf3jd

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


---------------------------------------------------------------------
To unsubscribe, e-mail: java-dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-dev-help@lucene.apache.org