Return-Path: X-Original-To: apmail-lucene-dev-archive@www.apache.org Delivered-To: apmail-lucene-dev-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 373B3D696 for ; Sun, 11 Nov 2012 05:14:57 +0000 (UTC) Received: (qmail 76875 invoked by uid 500); 11 Nov 2012 05:14:56 -0000 Delivered-To: apmail-lucene-dev-archive@lucene.apache.org Received: (qmail 76702 invoked by uid 500); 11 Nov 2012 05:14:55 -0000 Mailing-List: contact dev-help@lucene.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@lucene.apache.org Delivered-To: mailing list dev@lucene.apache.org Received: (qmail 76675 invoked by uid 99); 11 Nov 2012 05:14:54 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 11 Nov 2012 05:14:54 +0000 X-ASF-Spam-Status: No, hits=2.2 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_NONE,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of SRS0=ZLAgSg=JH=basetechnology.com=jack@yourhostingaccount.com designates 65.254.253.93 as permitted sender) Received: from [65.254.253.93] (HELO mailout11.yourhostingaccount.com) (65.254.253.93) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 11 Nov 2012 05:14:47 +0000 Received: from mailscan06.yourhostingaccount.com ([10.1.15.6] helo=mailscan06.yourhostingaccount.com) by mailout11.yourhostingaccount.com with esmtp (Exim) id 1TXPsI-0004BW-EB for dev@lucene.apache.org; Sun, 11 Nov 2012 00:14:26 -0500 Received: from impout01.yourhostingaccount.com ([10.1.55.1] helo=impout01.yourhostingaccount.com) by mailscan06.yourhostingaccount.com with esmtp (Exim) id 1TXPsI-0002Zc-6o for dev@lucene.apache.org; Sun, 11 Nov 2012 00:14:26 -0500 Received: from authsmtp01.yourhostingaccount.com ([10.1.18.1]) by impout01.yourhostingaccount.com with NO UCE id N5ES1k00301P85W015ESrH; Sun, 11 Nov 2012 00:14:26 -0500 X-Authority-Analysis: v=2.0 cv=IICA+3TG c=1 sm=1 a=531gI4P/BMH+/0jtZHoXtg==:17 a=aQzbgH187woA:10 a=Sd47zMRaCqoA:10 a=3jZET7lWBKwA:10 a=jvYhGVW7AAAA:8 a=fSEKCpUa5NAA:10 a=mV9VRH-2AAAA:8 a=qeREMo8yAAAA:8 a=pGLkceISAAAA:8 a=On6MY0FVAAAA:8 a=1RZpZjfYAAAA:8 a=02imkTy2nbFv733ENpAA:9 a=QEXdDO2ut3YA:10 a=0dVgO9N7na4A:10 a=vUAIPad2b3MA:10 a=yvvDrRkhPVoA:10 a=EXfkSGV2Do4A:10 a=m4aRD8sWmd4A:10 a=MSl-tDqOz04A:10 a=PoQXmyKPbPEA:10 a=EMlJoiak7gQA:10 a=NwxXq7e0TMNfMriM:21 a=DVM8EuJvduWk6_qX:21 a=yMhMjlubAAAA:8 a=gZEHubgnOzL1pvO6mrgA:9 a=gKO2Hq4RSVkA:10 a=UiCQ7L4-1S4A:10 a=hTZeC7Yk6K0A:10 a=frz4AuCg-hUA:10 a=tXsnliwV7b4A:10 a=FERfFkWt-AwA:10 a=bvcwM067-hYA:10 a=NWVoK91CQyQA:10 a=K-Uh-l01Dstqg9-K:21 a=ilymawf/5WNU8sTmGLp1gQ==:117 X-EN-OrigOutIP: 10.1.18.1 X-EN-IMPSID: N5ES1k00301P85W015ESrH Received: from ip-64-134-228-203.public.wayport.net ([64.134.228.203] helo=JackKrupansky) by authsmtp01.yourhostingaccount.com with esmtpa (Exim) id 1TXPsH-00028a-4z for dev@lucene.apache.org; Sun, 11 Nov 2012 00:14:25 -0500 Message-ID: <99C11B58493E4B59B8A8C61774C725D5@JackKrupansky> From: "Jack Krupansky" To: References: <1574581308.74146.1347527827817.JavaMail.jiratomcat@arcas> <7C09B40B9AF7446190E18140406F076D@JackKrupansky> <007001cdbf8a$ab0891b0$0119b510$@thetaphi.de> In-Reply-To: <007001cdbf8a$ab0891b0$0119b510$@thetaphi.de> Subject: Re: FuzzyQuery vs SlowFuzsyQuery docs? -- was: Re: [jira] [Commented] (LUCENE-2667) Fix FuzzyQuery's defaults, so its fast. Date: Sat, 10 Nov 2012 21:14:19 -0800 MIME-Version: 1.0 Content-Type: multipart/alternative; boundary="----=_NextPart_000_001E_01CDBF88.59264470" X-Priority: 3 X-MSMail-Priority: Normal Importance: Normal X-Mailer: Microsoft Windows Live Mail 15.4.3555.308 X-MimeOLE: Produced By Microsoft MimeOLE V15.4.3555.308 X-EN-UserInfo: e0a4b55451ed9f27313ebf02e3d4348d:fc4a93e1349e680c52bdd723c0ab3ef6 X-EN-AuthUser: jack@basetechnology.com Sender: "Jack Krupansky" X-EN-OrigIP: 64.134.228.203 X-EN-OrigHost: ip-64-134-228-203.public.wayport.net X-Virus-Checked: Checked by ClamAV on apache.org ------=_NextPart_000_001E_01CDBF88.59264470 Content-Type: text/plain; charset="utf-8" Content-Transfer-Encoding: quoted-printable =E2=80=9Cwe did not remove functionality=E2=80=9D Are you saying that full-featured =E2=80=9Cclassic=E2=80=9D fuzzy query = is still available in the Lucene query parser? By default? Or via what = option? -- Jack Krupansky From: Uwe Schindler=20 Sent: Saturday, November 10, 2012 1:30 PM To: dev@lucene.apache.org=20 Subject: RE: FuzzyQuery vs SlowFuzsyQuery docs? -- was: Re: [jira] = [Commented] (LUCENE-2667) Fix FuzzyQuery's defaults, so its fast. Just use SlowFuzzyQuery from contrib, I really don=E2=80=99t understand = where the issue is? The code is available in the sandbox/query module = and is available in Lucene 4.0. There is no reason to complain here, we = did not remove functionality. =20 ----- Uwe Schindler H.-H.-Meier-Allee 63, D-28213 Bremen http://www.thetaphi.de eMail: uwe@thetaphi.de =20 From: mbennett.ideaeng@gmail.com [mailto:mbennett.ideaeng@gmail.com] On = Behalf Of Mark Bennett Sent: Saturday, November 10, 2012 10:18 PM To: dev@lucene.apache.org Subject: Re: FuzzyQuery vs SlowFuzsyQuery docs? -- was: Re: [jira] = [Commented] (LUCENE-2667) Fix FuzzyQuery's defaults, so its fast. =20 Hi guys, Not expecting to change minds, but found Robert's last email helpful, so = wanted to try one more round. On Fri, Nov 9, 2012 at 5:32 PM, Robert Muir wrote: ... This is some analysis chain configuration issue. =20 Interesting, so you would expect that the seed term *would* go through = analysis before it finds the variants in the index? If it's supposed to = work that way then I can recheck my config. (it wasn't just lowercase, = that was just an example) =20 If it doesn't work with 100M documents, i don't want it in lucene. =20 Ah, this is very illuminating. For scalability, big data, etc, that = certainly makes sense. But there are many important Intranet search applications that have far = less than 100M docs, but still need the fine-grained control of = solr/lucene. Intranet projects in the 35k to 2M doc range often have = even more precise indexing, filtering and faceting requirements, and = solr/lucene provides that fine blade. Wouldn't it be more constructive to pick some number, say 100M, and give = that the "big data" moniker. Then, perhaps for things are not that = scalable, have some separate area/label but still retain them. = Discarding all use cases < 100M seems draconian. =20 I would have the same opinion if someone wanted unscalable solutions for scoring w/ language models (e.g. not happy with smoothing for unknown probabilities), or if someone claimed that spatial queries should do slow things because they don't currently support interplanetary distances, and so on. On Fri, Nov 9, 2012 at 7:52 PM, Mark Bennett = wrote: > Hi Robert, > > I acknowledge your "-1" vote, and I'm guessing that your objection = is maybe > 70% "scalability", and only 30% use-case? > > The older Levenstein stuff has been around for a long time, scalable = or not, > and already in real systems. > > You seem to have a very "binary" on code being "in" or "out". Is = there any > room in your world-view of code for "gray code", unsupported, = incubator, > what-have-you? Maybe analagous to people who jailbreak their = iPhones or > something? > > You're an important part of the community, and working at Lucid, = etc., and > clearly concerned about software quality. When smart folks like you = have > such sharp opinions I do try to ponder them against my own = circumstances. > > And on the quality of the old code, was it just the scalability, or = were > there other concerns such as stability, coding style, or possibly > inconsistent results? > > Isn't the sandbox and admonished reference in Java docs sufficient? > > I'm harping on this because I'm really between a rock and hard = place, and > also posted another question. > > Just trying to understand your very strong opinions, and I thank you = for > your patience in this matter. This issue is either going to fix or = break my > weekend / next-deliverble. > > Sincere thanks, > Mark > > > -- > Mark Bennett / New Idea Engineering, Inc. / mbennett@ideaeng.com > Direct: 408-733-0387 / Main: 866-IDEA-ENG / Cell: 408-829-6513 > > > On Fri, Nov 9, 2012 at 4:37 PM, Robert Muir = wrote: >> >> I'm -1 for having unscalable shit in lucene's core. This query = should >> have never been added. >> >> I don't care if a few people complain because they aren't using >> lowercasefilter or some other insanity. Fix your analysis chain. I >> don't have any sympathy. >> >> On Fri, Nov 9, 2012 at 7:35 PM, Jack Krupansky = >> wrote: >> > +1 for permitting a choice of fuzzy query implementation. >> > >> > I agree that we want a super-fast fuzzy query for simple = variations, but >> > I >> > also agree that we should have the option to trade off speed for >> > function. >> > >> > But I am also sympathetic to assuring that any core Lucene = features be >> > as >> > performant as possible. >> > >> > Ultimately, if there was a single fuzzy query implementation that = did >> > everything for everybody all of the time, that would be the way = to go, >> > but >> > if choices need to be made to satisfy competing goals, we should = support >> > going that route. >> > >> > -- Jack Krupansky >> > >> > From: Mark Bennett >> > Sent: Friday, November 09, 2012 3:48 PM >> > To: dev@lucene.apache.org >> > Subject: Re: FuzzyQuery vs SlowFuzsyQuery docs? -- was: Re: = [jira] >> > [Commented] (LUCENE-2667) Fix FuzzyQuery's defaults, so its fast. >> > >> > Hi Robert, >> > >> > On Thu, Sep 13, 2012 at 7:39 PM, Robert Muir = wrote: >> >> >> >> ... >> >> ... I'm strongly against having this >> >> unscalable garbage in lucene's core. >> >> >> >> There is no use case for ed > 2, thats just crazy. >> > >> > >> > I promise you there ARE use cases for edit distances > 2, = especially >> > with >> > longer words. Due to NDA I can't go into details. >> > >> > Also ed>2 can be useful when COMBINING that low-quality part of = the >> > search >> > with other sub-queries, or additional business rules. Maybe = instead of >> > boiling an ocean this lets you just boil the sea. ;-) >> > >> > I won't comment on the quality of the older Levenstein code, or = the >> > likely >> > very slow performance, nor where the code should live, etc. >> > >> > But your statement about "no use case for ed > 2" is simply not = true. >> > (whether you'd agree with any of them or not is certainly another >> > matter) >> > >> > I understand your concerns about not having it be the default. = (or >> > maybe >> > having a giant warning message or something, whatever) >> > >> >> -- >> >> lucidworks.com >> >> >> >> = --------------------------------------------------------------------- >> >> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org >> >> For additional commands, e-mail: dev-help@lucene.apache.org >> >> >> > >> >> = --------------------------------------------------------------------- >> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org >> For additional commands, e-mail: dev-help@lucene.apache.org >> > --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org For additional commands, e-mail: dev-help@lucene.apache.org =20 ------=_NextPart_000_001E_01CDBF88.59264470 Content-Type: text/html; charset="utf-8" Content-Transfer-Encoding: quoted-printable
=E2=80=9Cwe did not = remove=20 functionality=E2=80=9D
 
Are you saying that full-featured =E2=80=9Cclassic=E2=80=9D fuzzy = query is still available=20 in the Lucene query parser? By default? Or via what option?

-- Jack=20 Krupansky
 
Sent: Saturday, November 10, 2012 1:30 PM
Subject: RE: FuzzyQuery vs SlowFuzsyQuery docs? -- was: Re: = [jira]=20 [Commented] (LUCENE-2667) Fix FuzzyQuery's defaults, so its=20 fast.
 

Just use SlowFuzzyQuery from contrib, I really = don=E2=80=99t understand where=20 the issue is? The code is available in the sandbox/query module and is = available=20 in Lucene 4.0. There is no reason to complain here, we did not remove=20 functionality.

 

-----

Uwe=20 Schindler

H.-H.-Meier-Allee=20 63, D-28213 Bremen

http://www.thetaphi.de

eMail:=20 uwe@thetaphi.de

 

From:=20 mbennett.ideaeng@gmail.com [mailto:mbennett.ideaeng@gmail.com] On = Behalf Of=20 Mark Bennett
Sent: Saturday, November 10, 2012 10:18=20 PM
To: dev@lucene.apache.org
Subject: Re: FuzzyQuery = vs=20 SlowFuzsyQuery docs? -- was: Re: [jira] [Commented] (LUCENE-2667) Fix=20 FuzzyQuery's defaults, so its fast.

 

Hi guys,

Not = expecting to=20 change minds, but found Robert's last email helpful, so wanted to try = one more=20 round.

On Fri, Nov 9, 2012 at 5:32 PM, Robert Muir <rcmuir@gmail.com>=20 wrote:

...
This is some analysis chain configuration=20 issue.

 

Interesting, so you would expect that the seed term = *would*=20 go through analysis before it finds the variants in the index?  If = it's=20 supposed to work that way then I can recheck my config.  (it wasn't = just=20 lowercase, that was just an example)
 

If it doesn't work with 100M documents, i don't = want it in=20 lucene.

 

Ah, this is very illuminating.  For = scalability, big=20 data, etc, that certainly makes sense.

But there are many = important=20 Intranet search applications that have far less than 100M docs, but = still need=20 the fine-grained control of solr/lucene.  Intranet projects in the = 35k to=20 2M doc range often have even more precise indexing, filtering and = faceting=20 requirements, and solr/lucene provides that fine blade.

Wouldn't = it be=20 more constructive to pick some number, say 100M, and give that the "big = data"=20 moniker.  Then, perhaps for things are not that scalable, have some = separate area/label but still retain them.  Discarding all use = cases <=20 100M seems draconian.

 


I would have the same opinion if someone = wanted=20 unscalable solutions
for scoring w/ language models (e.g. not happy = with=20 smoothing for
unknown probabilities), or if someone claimed that = spatial=20 queries
should do slow things because they don't currently=20 support
interplanetary distances, and so on.


On Fri, Nov 9, = 2012 at 7:52=20 PM, Mark Bennett <mbennett@ideaeng.com> = wrote:
>=20 Hi Robert,
>
> I acknowledge your "-1" vote, and I'm = guessing that=20 your objection is maybe
> 70% "scalability", and only 30%=20 use-case?
>
> The older Levenstein stuff has been around = for a=20 long time, scalable or not,
> and already in real=20 systems.
>
> You seem to have a very "binary" on code = being "in"=20 or "out".  Is there any
> room in your world-view of code = for "gray=20 code", unsupported, incubator,
> what-have-you?  Maybe = analagous to=20 people who jailbreak their iPhones or
> = something?
>
>=20 You're an important part of the community, and working at Lucid, etc., = and
> clearly concerned about software quality.  When smart = folks=20 like you have
> such sharp opinions I do try to ponder them = against my=20 own circumstances.
>
> And on the quality of the old code, = was it=20 just the scalability, or were
> there other concerns such as = stability,=20 coding style, or possibly
> inconsistent = results?
>
> Isn't=20 the sandbox and admonished reference in Java docs = sufficient?
>
>=20 I'm harping on this because I'm really between a rock and hard place,=20 and
> also posted another question.
>
> Just trying = to=20 understand your very strong opinions, and I thank you for
> your = patience in this matter.  This issue is either going to fix or = break=20 my
> weekend / next-deliverble.
>
> Sincere = thanks,
>=20 Mark
>
>
> --
> Mark Bennett / New Idea = Engineering,=20 Inc. / mbennett@ideaeng.com
>=20 Direct: 408-733-0387 / Main: = 866-IDEA-ENG /=20 Cell: 408-829-6513
>
>
> On=20 Fri, Nov 9, 2012 at 4:37 PM, Robert Muir <rcmuir@gmail.com>=20 wrote:
>>
>> I'm -1 for having unscalable shit in = lucene's=20 core. This query should
>> have never been=20 added.
>>
>> I don't care if a few people complain = because=20 they aren't using
>> lowercasefilter or some other insanity. = Fix your=20 analysis chain. I
>> don't have any = sympathy.
>>
>>=20 On Fri, Nov 9, 2012 at 7:35 PM, Jack Krupansky <jack@basetechnology.com>>>=20 wrote:
>> > +1 for permitting a choice of fuzzy query=20 implementation.
>> >
>> > I agree that we want = a=20 super-fast fuzzy query for simple variations, but
>> >=20 I
>> > also agree that we should have the option to trade = off=20 speed for
>> > function.
>> >
>> > = But I=20 am also sympathetic to assuring that any core Lucene features = be
>>=20 > as
>> > performant as possible.
>> = >
>>=20 > Ultimately, if there was a single fuzzy query implementation that = did
>> > everything for everybody all of the time, that = would be=20 the way to go,
>> > but
>> > if choices need = to be=20 made to satisfy competing goals, we should support
>> > = going that=20 route.
>> >
>> > -- Jack Krupansky
>> = >
>> > From: Mark Bennett
>> > Sent: = Friday,=20 November 09, 2012 3:48 PM
>> > To: dev@lucene.apache.org
>&g= t; >=20 Subject: Re: FuzzyQuery vs SlowFuzsyQuery docs? -- was: Re: = [jira]
>>=20 > [Commented] (LUCENE-2667) Fix FuzzyQuery's defaults, so its=20 fast.
>> >
>> > Hi Robert,
>>=20 >
>> > On Thu, Sep 13, 2012 at 7:39 PM, Robert Muir = <rcmuir@gmail.com> = wrote:
>>=20 >>
>> >> ...
>> >> ... I'm = strongly=20 against having this
>> >> unscalable garbage in = lucene's=20 core.
>> >>
>> >> There is no use case = for ed=20 > 2, thats just crazy.
>> >
>> = >
>> > I=20 promise you there ARE use cases for edit distances > 2,=20 especially
>> > with
>> > longer words.  = Due to=20 NDA I can't go into details.
>> >
>> > Also = ed>2=20 can be useful when COMBINING that low-quality part of the
>> = >=20 search
>> > with other sub-queries, or additional business = rules.  Maybe instead of
>> > boiling an ocean this = lets you=20 just boil the sea.  ;-)
>> >
>> > I won't = comment=20 on the quality of the older Levenstein code, or the
>> >=20 likely
>> > very slow performance, nor where the code = should live,=20 etc.
>> >
>> > But your statement about "no = use case=20 for ed > 2" is simply not true.
>> > (whether you'd = agree with=20 any of them or not is certainly another
>> > = matter)
>>=20 >
>> > I understand your concerns about not having it = be the=20 default.  (or
>> > maybe
>> > having a = giant=20 warning message or something, whatever)
>> >
>> = >>=20 --
>> >> lucidworks.com
>> >>
>> = >>=20 = ---------------------------------------------------------------------
= >>=20 >> To unsubscribe, e-mail: dev-unsubscribe@lucene.= apache.org
>>=20 >> For additional commands, e-mail: dev-help@lucene.apache.org=
>>=20 >>
>> >
>>
>>=20 = ---------------------------------------------------------------------
= >>=20 To unsubscribe, e-mail: dev-unsubscribe@lucene.= apache.org
>>=20 For additional commands, e-mail: dev-help@lucene.apache.org=
>>
>

---------------------------------------------= ------------------------
To=20 unsubscribe, e-mail: dev-unsubscribe@lucene.= apache.org
For=20 additional commands, e-mail: dev-help@lucene.apache.org=

 

------=_NextPart_000_001E_01CDBF88.59264470--