lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Michael Busch (JIRA)" <>
Subject [jira] Commented: (LUCENE-1879) Parallel incremental indexing
Date Fri, 06 Nov 2009 17:43:32 GMT


Michael Busch commented on LUCENE-1879:

This sounds great! In fact your proposal for a ParallelSegmentWriter
is just like what I'm picturing - making the switching "down low"
instead of "up high" (above Lucene). This'd be more generic than just
the postings files, since all index files can be separately written. 

Right.  The goal should it be to be able to use this for updating Lucene internal things (like
norms, column-stride fields), but also giving advanced users APIs, so that they can partition
their data into parallel indexes according to their update requirements (which the current
"above Lucene" approach allows).

t'd then a low-level question of whether ParallelSegmentWriter stores
its files in different Directories, or, a single directory with
different file names (or maybe sub-directories within a directory, or,
something else). It could even use FileSwitchDirectory, eg to direct
certain segment files to an SSD (another way to achieve your example).

Exactly! We should also keep the distributed indexing use case in mind here. It could make
sense for systems like Katta to not only shard in the document direction.

This should also fit well into LUCENE-1458

Sounds great!

> Parallel incremental indexing
> -----------------------------
>                 Key: LUCENE-1879
>                 URL:
>             Project: Lucene - Java
>          Issue Type: New Feature
>          Components: Index
>            Reporter: Michael Busch
>            Assignee: Michael Busch
>             Fix For: 3.1
>         Attachments: parallel_incremental_indexing.tar
> A new feature that allows building parallel indexes and keeping them in sync on a docID
level, independent of the choice of the MergePolicy/MergeScheduler.
> Find details on the wiki page for this feature:
> Discussion on java-dev:

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message