lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Danil ŢORIN <torin...@gmail.com>
Subject Re: index large size file
Date Wed, 11 Mar 2009 08:16:50 GMT
The problem you may face that for such large documents,is that there
is a high probability that most of terms will be present in all
documents.

So on search you'll receive a lot of documents (if you need to
retrieve full text, it will take a while), but the bigger problem is
usability: what a user will do when instead of 100 000 documents the
search will narrow to 90 000 and adding more terms to search will go
down to 50 000?

On Tue, Mar 10, 2009 at 17:03, Amy Zhou <amy.zhou@systemware.com> wrote:
> Thanks Eric for your quick response and useful information. I'll give a try to bump up
the MaxFieldLength and check the performance. It seems the quickest way to handle the issue.
>
> Amy
>
> -----Original Message-----
> From: Erick Erickson [mailto:erickerickson@gmail.com]
> Sent: Tuesday, March 10, 2009 9:56 AM
> To: java-user@lucene.apache.org
> Subject: Re: index large size file
>
> Sure there are other options. You could decide to index in chunks
> rather then entire  documents. You could decide many things.
> None of which we can recommend unless we have a clue what
> you're really trying to accomplish or whether you're encountering
> a specific problem.
>
> I can say that we've indexed 7,000 *page* documents by bumping the
> MaxFieldLength. The performance is fine. I didn't measure indexing
> performance, but it ran acceptably quickly. Search performance seems
> unaffected, it's mostly dependent upon the overall index size and
> number of unique tokens as far as I can tell.
>
> I suggest you just try it and measure, that's the only way to determine
> whether *your* situation is adversely affected, since nobody can answer
> such a general question without considerably more specifics, and even
> then the answer is a qualified guess.
>
> But if you're *really* asking whether bumping MaxFieldLength does
> something like reserve that much space for every document whether
> or not it needs to, the answer is "no". A MaxFieldLength of 1,000,000,000
> won't use noticeably more resources for a file with 10 tokens than if the
> MaxFieldLength were 100. As far as I know.
>
> Best
> Erick
>
> On Tue, Mar 10, 2009 at 10:47 AM, Amy Zhou <amy.zhou@systemware.com> wrote:
>
>> My issue here is that large file is truncated with default MaxFieldLength
>> 10,000 during indexing. The file size I index could be 10mb or larger.
>>
>> My questions are:
>>
>> 1) If I chose MaxFieldLength as UNLIMITED instead of 100,000, what the
>> performance could be?
>> 2) Any other options?
>>
>>
>> -----Original Message-----
>> From: Mark Miller [mailto:markrmiller@gmail.com]
>> Sent: Tuesday, March 10, 2009 9:37 AM
>> To: java-user@lucene.apache.org
>> Subject: Re: index large size file
>>
>> Amy Zhou wrote:
>> > Hi,
>> >
>> > I'm having a couple of questions about indexing large size file. As my
>> understanding, the default MaxFieldLength 100,000. In Lucene 2.4, we can set
>> the MaxFieldLength during constructor. My questions are:
>> >
>> The default is 10,000.
>> > 1) How's the performance if MaxFieldLength is set to UNLIMITED?
>> >
>> It depends on how long your documents are. Its simply a cutoff -
>> documents longer than n (10,000 by default) terms will be truncated.
>> > 2) Any other options for indexing large size file?
>> >
>> What is the problem you are trying to address? Are you having trouble
>> indexing a very large file? Can you share more details?
>>
>>
>> --
>> - Mark
>>
>> http://www.lucidimagination.com
>>
>>
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>> ---------------------------------------------------------------------
>> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
>> For additional commands, e-mail: java-user-help@lucene.apache.org
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
>
>

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message