lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jack Krupansky" <j...@basetechnology.com>
Subject Re: FuzzyQuery vs SlowFuzsyQuery docs? -- was: Re: [jira] [Commented] (LUCENE-2667) Fix FuzzyQuery's defaults, so its fast.
Date Sun, 11 Nov 2012 15:18:32 GMT
Okay, so maybe this is simply a case where “an adjustment” was made to Lucene and Solr
did not make a corresponding “adustment” to compensate to “preserve” functionality.
Solr users cannot easily override factory methods, but of course the Solr query parser can
and probably should.

I’m not able to determine for sure whether Mark’s comments were directed strictly at Lucene
or Solr. I mean, for a lot of people there isn’t really supposed to be a distinction between
the two.

So, maybe we need a Solr Jira to the effect of adding a parameter such as “fuzzy.maxEdit”
which defaults to “2”, but could be set higher to access “slow” fuzzy query. And,
maybe that default should also be keyed off of the schema version or Lucene version so that
an “old” Solr app would see no loss of function, although there is the issue that old
queries used a float correlation factor rather than an int edit distance.

Is there a reliable formula for converting from the old correlation factor and the new edit
distance or at least a reasonable approximation? That should be in the Javadoc. I suppose
it could be approximated, but there are edge cases where the new approach provides more discrimination
that the old approach, such as for very short terms.

-- Jack Krupansky

From: Uwe Schindler 
Sent: Sunday, November 11, 2012 2:51 AM
To: dev@lucene.apache.org 
Subject: RE: FuzzyQuery vs SlowFuzsyQuery docs? -- was: Re: [jira] [Commented] (LUCENE-2667)
Fix FuzzyQuery's defaults, so its fast.

Yes, if you override the factory method in the classic QueryParser. The functionality is still
available, you have to just use the classes directly.

 

-----

Uwe Schindler

H.-H.-Meier-Allee 63, D-28213 Bremen

http://www.thetaphi.de

eMail: uwe@thetaphi.de

 

From: Jack Krupansky [mailto:jack@basetechnology.com] 
Sent: Sunday, November 11, 2012 6:14 AM
To: dev@lucene.apache.org
Subject: Re: FuzzyQuery vs SlowFuzsyQuery docs? -- was: Re: [jira] [Commented] (LUCENE-2667)
Fix FuzzyQuery's defaults, so its fast.

 

“we did not remove functionality”

 

Are you saying that full-featured “classic” fuzzy query is still available in the Lucene
query parser? By default? Or via what option?


-- Jack Krupansky

 

From: Uwe Schindler 

Sent: Saturday, November 10, 2012 1:30 PM

To: dev@lucene.apache.org 

Subject: RE: FuzzyQuery vs SlowFuzsyQuery docs? -- was: Re: [jira] [Commented] (LUCENE-2667)
Fix FuzzyQuery's defaults, so its fast.

 

Just use SlowFuzzyQuery from contrib, I really don’t understand where the issue is? The
code is available in the sandbox/query module and is available in Lucene 4.0. There is no
reason to complain here, we did not remove functionality.

 

-----

Uwe Schindler

H.-H.-Meier-Allee 63, D-28213 Bremen

http://www.thetaphi.de

eMail: uwe@thetaphi.de

 

From: mbennett.ideaeng@gmail.com [mailto:mbennett.ideaeng@gmail.com] On Behalf Of Mark Bennett
Sent: Saturday, November 10, 2012 10:18 PM
To: dev@lucene.apache.org
Subject: Re: FuzzyQuery vs SlowFuzsyQuery docs? -- was: Re: [jira] [Commented] (LUCENE-2667)
Fix FuzzyQuery's defaults, so its fast.

 

Hi guys,

Not expecting to change minds, but found Robert's last email helpful, so wanted to try one
more round.

On Fri, Nov 9, 2012 at 5:32 PM, Robert Muir <rcmuir@gmail.com> wrote:

...
This is some analysis chain configuration issue.

 

Interesting, so you would expect that the seed term *would* go through analysis before it
finds the variants in the index?  If it's supposed to work that way then I can recheck my
config.  (it wasn't just lowercase, that was just an example)
 

  If it doesn't work with 100M documents, i don't want it in lucene.

 

Ah, this is very illuminating.  For scalability, big data, etc, that certainly makes sense.

But there are many important Intranet search applications that have far less than 100M docs,
but still need the fine-grained control of solr/lucene.  Intranet projects in the 35k to 2M
doc range often have even more precise indexing, filtering and faceting requirements, and
solr/lucene provides that fine blade.

Wouldn't it be more constructive to pick some number, say 100M, and give that the "big data"
moniker.  Then, perhaps for things are not that scalable, have some separate area/label but
still retain them.  Discarding all use cases < 100M seems draconian.

 


  I would have the same opinion if someone wanted unscalable solutions
  for scoring w/ language models (e.g. not happy with smoothing for
  unknown probabilities), or if someone claimed that spatial queries
  should do slow things because they don't currently support
  interplanetary distances, and so on.


  On Fri, Nov 9, 2012 at 7:52 PM, Mark Bennett <mbennett@ideaeng.com> wrote:
  > Hi Robert,
  >
  > I acknowledge your "-1" vote, and I'm guessing that your objection is maybe
  > 70% "scalability", and only 30% use-case?
  >
  > The older Levenstein stuff has been around for a long time, scalable or not,
  > and already in real systems.
  >
  > You seem to have a very "binary" on code being "in" or "out".  Is there any
  > room in your world-view of code for "gray code", unsupported, incubator,
  > what-have-you?  Maybe analagous to people who jailbreak their iPhones or
  > something?
  >
  > You're an important part of the community, and working at Lucid, etc., and
  > clearly concerned about software quality.  When smart folks like you have
  > such sharp opinions I do try to ponder them against my own circumstances.
  >
  > And on the quality of the old code, was it just the scalability, or were
  > there other concerns such as stability, coding style, or possibly
  > inconsistent results?
  >
  > Isn't the sandbox and admonished reference in Java docs sufficient?
  >
  > I'm harping on this because I'm really between a rock and hard place, and
  > also posted another question.
  >
  > Just trying to understand your very strong opinions, and I thank you for
  > your patience in this matter.  This issue is either going to fix or break my
  > weekend / next-deliverble.
  >
  > Sincere thanks,
  > Mark
  >
  >
  > --
  > Mark Bennett / New Idea Engineering, Inc. / mbennett@ideaeng.com
  > Direct: 408-733-0387 / Main: 866-IDEA-ENG / Cell: 408-829-6513
  >
  >
  > On Fri, Nov 9, 2012 at 4:37 PM, Robert Muir <rcmuir@gmail.com> wrote:
  >>
  >> I'm -1 for having unscalable shit in lucene's core. This query should
  >> have never been added.
  >>
  >> I don't care if a few people complain because they aren't using
  >> lowercasefilter or some other insanity. Fix your analysis chain. I
  >> don't have any sympathy.
  >>
  >> On Fri, Nov 9, 2012 at 7:35 PM, Jack Krupansky <jack@basetechnology.com>
  >> wrote:
  >> > +1 for permitting a choice of fuzzy query implementation.
  >> >
  >> > I agree that we want a super-fast fuzzy query for simple variations, but
  >> > I
  >> > also agree that we should have the option to trade off speed for
  >> > function.
  >> >
  >> > But I am also sympathetic to assuring that any core Lucene features be
  >> > as
  >> > performant as possible.
  >> >
  >> > Ultimately, if there was a single fuzzy query implementation that did
  >> > everything for everybody all of the time, that would be the way to go,
  >> > but
  >> > if choices need to be made to satisfy competing goals, we should support
  >> > going that route.
  >> >
  >> > -- Jack Krupansky
  >> >
  >> > From: Mark Bennett
  >> > Sent: Friday, November 09, 2012 3:48 PM
  >> > To: dev@lucene.apache.org
  >> > Subject: Re: FuzzyQuery vs SlowFuzsyQuery docs? -- was: Re: [jira]
  >> > [Commented] (LUCENE-2667) Fix FuzzyQuery's defaults, so its fast.
  >> >
  >> > Hi Robert,
  >> >
  >> > On Thu, Sep 13, 2012 at 7:39 PM, Robert Muir <rcmuir@gmail.com> wrote:
  >> >>
  >> >> ...
  >> >> ... I'm strongly against having this
  >> >> unscalable garbage in lucene's core.
  >> >>
  >> >> There is no use case for ed > 2, thats just crazy.
  >> >
  >> >
  >> > I promise you there ARE use cases for edit distances > 2, especially
  >> > with
  >> > longer words.  Due to NDA I can't go into details.
  >> >
  >> > Also ed>2 can be useful when COMBINING that low-quality part of the
  >> > search
  >> > with other sub-queries, or additional business rules.  Maybe instead of
  >> > boiling an ocean this lets you just boil the sea.  ;-)
  >> >
  >> > I won't comment on the quality of the older Levenstein code, or the
  >> > likely
  >> > very slow performance, nor where the code should live, etc.
  >> >
  >> > But your statement about "no use case for ed > 2" is simply not true.
  >> > (whether you'd agree with any of them or not is certainly another
  >> > matter)
  >> >
  >> > I understand your concerns about not having it be the default.  (or
  >> > maybe
  >> > having a giant warning message or something, whatever)
  >> >
  >> >> --
  >> >> lucidworks.com
  >> >>
  >> >> ---------------------------------------------------------------------
  >> >> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
  >> >> For additional commands, e-mail: dev-help@lucene.apache.org
  >> >>
  >> >
  >>
  >> ---------------------------------------------------------------------
  >> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
  >> For additional commands, e-mail: dev-help@lucene.apache.org
  >>
  >

  ---------------------------------------------------------------------
  To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
  For additional commands, e-mail: dev-help@lucene.apache.org

 

Mime
View raw message