lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From ariel goldberg <>
Subject One index per user or one index per day?
Date Mon, 26 Feb 2007 15:25:51 GMT



I'm creating an application that 
requires the indexing of millions of documents on behalf of a large group of 
users, and was hoping to get an opinion on whether I should use one index per 
user or one index per day.


My application will have to handle 
the following:


- the indexing of about 1 million 5K 
documents per day, with each document containing about 5 

- expiration of documents, since 
after a while, my hard drive would run out of 

- queries that consist of boolean 
expressions (e.g., the body field contains "a" AND "b", and the title field 
contains "c"), as well as ranges (e.g., the document needs to have been indexed 
between 2/25/07 10:00 am and 2/28/07 9:00 pm)

- permissions; in other words, user 
A might be able to search on documents X and Y, but user B might be able to 
search on documents Y and Z.

- up to 1,000 


So, I was considering the 


1) Using one index per 


This would entail creating and using 
up to 1,000 indices.  Document Y in the example above would have to be 
duplicated.  Expiration is performed via IndexWriter.deleteDocuments.  The 
advantage here is that querying should be reasonably quick, because each index 
would only contain tens of thousands of documents, instead of millions.  The 
disadvantages: I'm concerned about the "too many open files" error, and I'm also 
concerned about the performance of 


2) Using one index per 


Each day, I create a new index.  
Again, document Y in the example above would have to be duplicated (is there any 
way around this?)  The advantage here is that expiring documents means simply 
deleting the index corresponding to a particular day.  The disadvantage is the 
query performance, since the queries, which are already very complex, would have 
to be performed using MultiSearcher (if expiration is after 10 days, that's 10 
indices to search across).


Tough to know for sure which option 
is better without testing, but does anyone have a gut reaction?  Any advice 
would be greatly appreciated!




Need Mail bonding?
Go to the Yahoo! Mail Q&A for great tips from Yahoo! Answers users.

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message