lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Suraj Parida <parida.su...@gmail.com>
Subject Re: Searching compressed text using CompressionTools
Date Wed, 03 Feb 2010 10:59:53 GMT

Ian,

Thanks for solving my previous problems.


Now i tested the compression with 100 docs and found:
  1. With Compression size of FS directory (on disk)= 10.8 KB
  2. Without Compression size of FS directory (on disk) = 12.0 MB

and with 500 docs:
  1. With Compression size of FS directory (on disk) = 45.9 KB
  2. Without Compression size of FS directory (on disk)  = 56.8 MB

I mean do the compression will increase my disk usage ? if so will 50,000
docs take around 6000 MB?
or please tell if am i doing wrong somewhere because i thought compression
will reduce space usage.

Regards,
Suraj 





Ian Lea wrote:
> 
> Please read Uwe's answers again.  I think he has already answered all
> your questions.
> 
> The javadocs for 2.9.1 are very useful when upgrading to 3.0.0.  Read
> the entry for Field.Store.COMPRESS.
> 
> 
> --
> Ian.
> 
> 
> 
> On Mon, Feb 1, 2010 at 12:04 PM, Suraj Parida <parida.suraj@gmail.com>
> wrote:
>>
>> Uwe,
>>
>> Thanks for the reply.
>>
>> I am confused with
>>    document.add(new Field(key, value, Field.Store.COMPRESS,
>> Field.Index.ANALYZED));
>>
>>  My requirement is also the same, but how can i do it in 3.0 ?
>>  i thought CompressionTools would be used for compression.
>>
>> Basically i need to compress the text which content of a document and
>> there
>> are around 50,000 documents and still increasing in number.  If possible
>> please send a code or hint as an example.
>>
>> Regards,
>> Suraj
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> Uwe Schindler wrote:
>>>
>>> Compression is only used for *stored* fields. For indexing there is no
>>> compression available (how should that work). You must clearly
>>> differentiate between stored and indexed fields!
>>>
>>> -----
>>> Uwe Schindler
>>> H.-H.-Meier-Allee 63, D-28213 Bremen
>>> http://www.thetaphi.de
>>> eMail: uwe@thetaphi.de
>>>
>>>> -----Original Message-----
>>>> From: Suraj Parida [mailto:parida.suraj@gmail.com]
>>>> Sent: Monday, February 01, 2010 12:44 PM
>>>> To: java-user@lucene.apache.org
>>>> Subject: Searching compressed text using CompressionTools
>>>>
>>>>
>>>> Hi,
>>>>
>>>> I want to compress a text field (due to its large size and spaces),
>>>> during
>>>> indexing.
>>>>
>>>> I am unable to get the same also want to search.
>>>>
>>>>
>>>> My code during compressing is as follows:
>>>>                                 String value = "Some large
text ......
>>>> ";
>>>>                              byte[] valuesbyte =
>>>> CompressionTools.compress(value.getBytes());
>>>>                              final Field f = new Field(key,
valuesbyte,
>>>> Field.Store.YES);
>>>>                              f.setOmitTermFreqAndPositions(true);
>>>>                              f.setOmitNorms(true);
>>>>                                 document.add(f);
>>>>
>>>> Please tell me how to search and display this value.
>>>>
>>>> Regards
>>>> Suraj
>>>> --
>>>> View this message in context: http://old.nabble.com/Searching-
>>>> compressed-text-using-CompressionTools-tp27402945p27402945.html
>>>> Sent from the Lucene - Java Users mailing list archive at Nabble.com.
>>>>
>>>>
>>>> ---------------------------------------------------------------------
>>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>
>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>>
>>>
>>>
>>
>> --
>> View this message in context:
>> http://old.nabble.com/Searching-compressed-text-using-CompressionTools-tp27402945p27403169.html
>> Sent from the Lucene - Java Users mailing list archive at Nabble.com.
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
> 
> 
> 

-- 
View this message in context: http://old.nabble.com/Searching-compressed-text-using-CompressionTools-tp27402945p27434620.html
Sent from the Lucene - Java Users mailing list archive at Nabble.com.


---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message