lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Uwe Schindler" <...@thetaphi.de>
Subject RE: Separating the document dataset and the index dataset
Date Tue, 11 Dec 2012 09:44:29 GMT
You can use Lucene 4.1 nightly builds from http://goo.gl/jZ6YD - it is not yet released, but
upgrading from Lucene 4.0 is easy. If you are not yet on Lucene 4.0, there is more work to
do, in that case a solution to your problem would be to save the stored fields in a separate
database/whatever and only add *one* stored field to your index, containing the document ID
inside this external database.

-----
Uwe Schindler
H.-H.-Meier-Allee 63, D-28213 Bremen
http://www.thetaphi.de
eMail: uwe@thetaphi.de


> -----Original Message-----
> From: Ramprakash Ramamoorthy [mailto:youngestachiever@gmail.com]
> Sent: Tuesday, December 11, 2012 10:32 AM
> To: java-user@lucene.apache.org
> Subject: Re: Separating the document dataset and the index dataset
> 
> On Fri, Dec 7, 2012 at 1:11 PM, Jain Rahul <jainr@ivycomptech.com> wrote:
> 
> > If you are using lucene 4.0 and afford to compress your document
> > dataset while indexing, it will be a huge savings in terms of disk
> > space and also in IO (resulting in indexing throughput).
> >
> > In our case, it has helped us a lot as compressed data size was
> > roughly 3 times less than  of original document data set size.
> >
> > You may want to check  the below  link.
> >
> >
> > http://blog.jpountz.net/post/33247161884/efficient-compressed-stored-f
> > ields-with-lucene
> >
> > Regards,
> > Rahul
> >
> 
> Thank you Rahul. That indeed seems promising. Just one doubt, how do I
> plug this  CompressingStoredFieldsFormat into my app, as in I tried bundling
> it in a codec, but not sure if I am proceeding in the right path. Any pointers
> would be of great help!
> 
> >
> >
> > -----Original Message-----
> > From: Ramprakash Ramamoorthy [mailto:youngestachiever@gmail.com]
> > Sent: 07 December 2012 13:03
> > To: java-user@lucene.apache.org
> > Subject: Separating the document dataset and the index dataset
> >
> > Greetings,
> >
> >          We are using lucene in our log analysis tool. We get data
> > around 35Gb a day and we have this practice of zipping week old
> > indices and then unzip when need arises.
> >
> >            Though the compression offers a huge saving with respect to
> > disk space, the decompression becomes an overhead. At times it takes
> > around
> > 10 minutes (de-compression takes 95% of the time) to search across a
> > month long set of logs. We need to unzip fully atleast to get the
> > total count from the index.
> >
> >            My question is, we are setting Index.Store to true. Is
> > there a way where we can split the index dataset and the document
> > dataset. In my understanding, if at all separation is possible, the
> > document dataset can alone be zipped leaving the index dataset on
> > disk? Will it be tangible to do this? Any pointers?
> >
> >            Or is adding more disks the only solution? Thanks in advance!
> >
> > --
> > With Thanks and Regards,
> > Ramprakash Ramamoorthy,
> > +91 9626975420
> > This email and any attachments are confidential, and may be legally
> > privileged and protected by copyright. If you are not the intended
> > recipient dissemination or copying of this email is prohibited. If you
> > have received this in error, please notify the sender by replying by
> > email and then delete the email completely from your system. Any views
> > or opinions are solely those of the sender. This communication is not
> > intended to form a binding contract unless expressly indicated to the
> > contrary and properly authorised. Any actions taken on the basis of
> > this email are at the recipient's own risk.
> >
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: java-user-help@lucene.apache.org
> >
> >
> 
> 
> --
> With Thanks and Regards,
> Ramprakash Ramamoorthy,
> Engineer Trainee,
> Zoho Corporation.
> +91 9626975420


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message