lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From petite_abeille <>
Subject Re: Homogeneous vs Heterogeneous indexes (was: FileNotFoundException)
Date Tue, 30 Apr 2002 07:03:38 GMT
On Tuesday, April 30, 2002, at 01:57 AM, Steven J. Owens wrote:

> Just be glad you aren't doing this on Solaris with JDK 1.1.6

I know... In fact I'm looking forward to port my stuff to 1.4... As my 
app is very much IO bond I'm really excited by this nio madness... :-)

> Yes and no.  Setting ulimit to a reasonable number of open files is not 
> only not a patch, it's the "right" way to do it.

Of course... Nothing is really black or white... What I wanted to say is 
that -as a first strike- *I* prefer not to mess around with system 

>  I understand where you're coming from, really, and in a certain way, 
> it makes sense

Thanks. I already feel less alone... ;-)

> BUT... sometimes the impulse for clean, good design takes you too far 
> down a blind alley.

Sure. At the end of the day, everything is a tradeoff...

>  Sometimes there is no elegant solution. Sometimes there is no "best" 
> way, only one of a limited set of options with different tradeoffs.


> Most serious applications have to have some sort of OS variable 
> tweaking, you're just used to having it done invisibly and painlessly.

Agree. In fact that's my first desktop application for nearly a decade. 
I usually work on large scale system. And let me tell you, it's a very 
different pair of sleeves... ;-)

>  You could figure out the "right" way to set the system configuration 
> on install or launch.

One of my design "goal" is to try to avoid these sort of tweakings as 
much as I can.

>  You could look at the alternative techniques for indexing in Lucene

That's another one of those nasty tradeoffs... ;-) Memory is even more 
precious than file descriptors in my situation... Specially with a jvm 
that have this funky notion of constraining your memory usage...

> if there's anything you're doing wrong (perhaps opening files and not 
> closing them, and leaving them for the garbage collector to eventually 
> get around to closing?)

Sure. I went through all those sanity checks. Also, in my case, the 
garbage collector is my friend as I'm using the java.lang.ref API 

>  or if you have a pessimal usage pattern that exacerbates the situation.

Ummmm...?!? You lost me here... What's a "pessimal usage pattern"?

> if you can come up with a scheme to run Lucene indexing with modified 
> code for keeping track of file resources.

Sure, there are many thing that one could do... However, I have to 
balance how much time I want to invest into any one of those allays. One 
thing I really like about Lucene is it very simple API and usage. So far 
it has worked out pretty well for me as I'm using it pretty extensively. 
And I seem to have found -at last- a good balance between the different 
constrains I'm operating under.

> an anomalous situation (use on a client/desktop machine)

"Anomalous situation"?!?! Ummm... Lucene is just an API... Hopefully 
it's not bundled with some "dogma" attached to it... However, I'm kind 
of starting to wander about that considering some of the -very 
defensive- responses I got to my postings... Oh, well... I will just go 
back to my cave... :-(

> could configure lucene to be careful about how many files it keeps open 
> at any given time.

That will be great! On a somewhat related note, I have decided to stick 
with the com.lucene package for the time being.... I was pretty excited 
when the rc stuff came out, but it just didn't work out for me. My 
resources problem just went from bad to worse. And also, I have two 
issues with the release candidate: locking and reference counting.

Locking. I don't have anything against locking per see. However, I 
really don't like how it's implemented in the rc. Using files just do 
not work for me. It creates too many problems when something goes wrong 
(eg the app is killed without warning and I have to clean up all those 
locks by myself). What about using sockets or something to rendez-vous 
on an index? Or at a bare minimum, be able to disable the locking all 
together. I understand that most people are using Lucene under a very 
different setup that I do, but nevertheless it should not hurt to make 
it configurable. Anyway, it does not work for older jvm as noted in the 
source code. Last, but not least, I'm always get very scared when I see 
some "platform" dependent code somewhere (eg "if version 1 then ") ;-)

Reference counting. Well, as noted in a comment in the source code, the 
reference API is really the way to go... And trying to be backward 
compatible to version 0.9 is somehow missing the forest for the tree... 
Just my two cents in any case. And yes, I'm well aware that I can fix 
all these issue by myself... And start to contribute to Lucene instead 
of just ranting left and right... But also keep in mind that I'm just a 
humble Lucene user. And there seem to be a very clear distinction 
between "user" and "developer" in Lucene's world... ;-)

Thanks for your response in any case. I hope I didn't "offend" too many 
people with my ramblings ;-)


To unsubscribe, e-mail:   <>
For additional commands, e-mail: <>

View raw message