lucene-java-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From David Spencer <>
Subject Re: SubstringQuery -- Re: Leading Wild Card Search
Date Wed, 18 Feb 2004 18:54:20 GMT
Doug Cutting wrote:

> David Spencer wrote:
>> 2 files attached, SubstringQuery (which you'll use) and 
>> SubstringTermEnum ( used by the former to be
>> consistent w/ other Query code).
>> I find this kind of query useful to have and think that the query 
>> parser should allow it in spite of the perception
>> of this being slow, however I think the debate is the "user centric 
>> view" (say mine, allow substring queries)
>> vs the "protect the engines performance" view which says not to allow 
>> expensive queries.
> I think the argument is more complex.

Thanks for elaboration, a few notes below.

> One issue is cost of execution: very slow queries can be used to 
> implement a denial-of-service attack.  Maybe that's an overstatement, 
> but in a web server setting, once a few of slow searches are running, 
> no others may complete.  When folks hit "Stop" in their browser the 
> server does not stop processing the query.  If they hit "Reload" then 
> another new search is started.  So these can be very problematic.

I guess this is defendable right? The search engine could allow  only 
one query at a time per session.

> This is real.  Lots of folks have deployed Lucene with large indexes 
> and then found that their server randomly crashes.  Closer scrutiny 
> shows that they were permitting operators that are too slow for their 
> combination of index size and query traffic.  The 
> BooleanQuery.TooManyClauses exception was added to address this, but 
> it can still be too late, if the problem is caused before the query is 
> built, e.g., while enumerating all terms.

Yeah, this is tricky, as really the TooManyClauses is not necessarily 
the cure, and every query can't defend itself against expense as these 
heuristics (e.g. max # of clauses) are not always right (depends on 
speed of system/disk, size of index, etc).
It's too bad the Java VM doesn't have any kind of execution time quotas 
for threads or something like that as that seems like in the ideal world 
to get to the root of the issue. Going back in time I believe the 
Telescript VM
from General Magic...

> A releated issue is that users (and even most developers) don't 
> understand the relative costs of different query operators.

I'm coming at this from the point of view of the smaller personal/small 
enterprise search server, where
you "know" you need a certain kind of possibly expensive query and don't 
want your hands tied.

> Some things are fast, others are surprisingly slow.  That's not a 
> great user experience, and triggers problems like those described above.

Reminds me of a human factors paper I saw in some ACM pub years ago (um, 
prob the 80's). Said more or less what you're saying, that humans prefer 
a constant response time even if a varying response time has a lower 
average. Example is, say, for a compiler, if it always takes 10sec to 
compile a file they users are happy, but
if it takes 1sec 99% of the time and a minute the other 1% then users 
are less happy. What was most interesting
about the paper was they suggested something like this:

goal = 10; // 10 sec
t1 = time(); // time in seconds
search(); // execute search, don't display results
dt = time() - t1; // elapsed
if ( elapsed < goal)
   sleep( goal- elapsed); // the shocker

i.e. artifically delay(!) if less than some threshold, thus making the 
user happier (!) to get a more constant response time.

> People think that the rare slow cases are network problems or 
> something, and hit "Reload".
> I have no problem with including slow operators with Lucene, but they 
> should be well documented as such, at least for developers.  Perhaps 
> we should make a pass through the existing Query classes, in 
> particular those which expand into other queries, and add some 
> performance notes, so that folks don't blindly start using things 
> which may bite them.  By default I think it would be safest if the 
> QueryParser only permitted operators which are efficient.  Folks can 
> then, at their own risk, enable other operators.
> In summary, removing operators can be user-centric, if it removes 
> unpredictablity.  And the reason for protecting engine performance is 
> not miserly, it's to guarantee availablility.  And finally, an issue 
> dear to me, a predicatble search engine results in fewer spurious bug 
> reports, saving developer time for real bugs.

Great, thx..

> Doug
> ---------------------------------------------------------------------
> To unsubscribe, e-mail:
> For additional commands, e-mail:

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message