lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Paul Elschot <paul.elsc...@xs4all.nl>
Subject Re: How to index a lot of fields (without FileNotFoundException: Too many open files)
Date Tue, 01 May 2007 07:31:31 GMT
On Tuesday 01 May 2007 00:22, pbm-rico@gmx.net wrote:
> I thought about using ulimit, but it does not scale. In the scenario that 
the app has to support, client applications could create hundreds of 
thousands of unique properties, which would result in this many indexable 
fields.
> 
> Based on previous answers, the way out of this problem while still being 
scalable would be to use 2.1 where there is only 1 norm file.
> 
> However, it does not look like upgrading is an option, so I wonder if my 
current approach of mapping a property that a client app creates to one field 
name is workable at all. Maybe I have to introduce some sort of mapping of 
client properties to a fixed number of indexable fields.
> 
> ...or modify the 1.4.3 code myself to get rid of norm files.

Do you need the idf computations of lucene for each of these client fields
separately?
When not, another way is to move the names of the client fields into the
term value, and use a single special field for these.
When you do that, you'll also have to move the client field names
in the queries from the field name to the term. This can easily be done by 
overriding one of the methods in QueryParser.

Regards,
Paul Elschot


> 
> -Rico
> 
> -------- Original-Nachricht --------
> Datum: Mon, 30 Apr 2007 15:08:14 -0700
> Von: "Mike Klaas" <mike.klaas@gmail.com>
> An: java-user@lucene.apache.org
> Betreff: Re: Re: How to index a lot of fields (without 
FileNotFoundException: Too many open files)
> 
> > On 4/30/07, pbm-rico@gmx.net <pbm-rico@gmx.net> wrote:
> > > Thanks for you reply.
> > >
> > > We are still using Lucene v1.4.3 and I'm not sure if upgrading is an
> > option. Is there another way of disabling length normalization/document 
boosts
> > to get rid of those files?
> > 
> > Why not raise the limit of open files on your system?  (man ulimit)
> > 
> > -Mike
> > 
> > ---------------------------------------------------------------------
> > To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> > For additional commands, e-mail: java-user-help@lucene.apache.org
> 
> -- 
> "Feel free" - 10 GB Mailbox, 100 FreeSMS/Monat ...
> Jetzt GMX TopMail testen: http://www.gmx.net/de/go/topmail
> 
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
> For additional commands, e-mail: java-user-help@lucene.apache.org
> 
> 
> 

---------------------------------------------------------------------
To unsubscribe, e-mail: java-user-unsubscribe@lucene.apache.org
For additional commands, e-mail: java-user-help@lucene.apache.org


Mime
View raw message