Mailing-List: contact java-user-help@lucene.apache.org; run by ezmlm
Precedence: bulk
Reply-To: java-user@lucene.apache.org
Received-SPF: pass (athena.apache.org: domain of adb@teamware.com designates
 212.226.92.15 as permitted sender)
Message-ID: <48043BD9.3080009@teamware.com>
Date: Tue, 15 Apr 2008 15:23:37 +1000
From: Antony Bowesman <adb@teamware.com>
Organization: Teamware Group
User-Agent: Thunderbird 2.0.0.12 (Windows/20080213)
MIME-Version: 1.0
To: java-user@lucene.apache.org
Subject: Re: Using Lucene partly as DB and 'joining' search results.
References: <47FF3B35.4000705@teamware.com>
 <Pine.LNX.4.62.0804141625570.24515@radix.cryptio.net>
 <48042808.1010600@teamware.com>
 <Pine.LNX.4.62.0804142136280.1081@radix.cryptio.net>
In-Reply-To: <Pine.LNX.4.62.0804142136280.1081@radix.cryptio.net>
Content-Type: text/plain; charset=ISO-8859-1; format=flowed
Content-Transfer-Encoding: 7bit

Chris Hostetter wrote:
> you can't ... that's why i said you'd need to rebuild the smaller index 
> completley on a periodic basis (going in the same order as the docs in the 

Mmm, the annotations would only be stored in the index.  It would be possible to 
store them elsewhere, so I can investigate that, in which case the rebuild would 
be possible.

> i can also imagine a situation where you break both indexes up into lots 
> of pieces (shards) and use a MultiReader over lots of ParallelReaders ... 
> that way you have much smaller "small" indexes to rebuild when someone 
> annotates an email -- and if hte shards are organized by date, you're less 
> likely to ever need to rebuild many of them since people will tend to 

Data will be 'sharded' anyway, by date of some granularity.  Looking at the 
source for MultiReader/MultiSearcher, they are single threaded.  Is there a 
performance trade off between single-thread/many small indexes and 
single-thread/some large indexes.  Can a MultiReader work with one..n reader per 
thread, something like a thread pool of IndexReaders.  I expect it would be 
faster to run the searches in parallel?

> Disclaimer: all of this is purely brainstorming, i've never actually tried 
> anything like this, it may be more trouble then it's worth.

:) Thanks for the sounding board - it's always useful to get new ideas!
Antony


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org