Return-Path: X-Original-To: apmail-lucene-dev-archive@www.apache.org Delivered-To: apmail-lucene-dev-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id AF6BE29E8 for ; Sat, 7 May 2011 11:02:36 +0000 (UTC) Received: (qmail 75847 invoked by uid 500); 7 May 2011 11:02:34 -0000 Delivered-To: apmail-lucene-dev-archive@lucene.apache.org Received: (qmail 75774 invoked by uid 500); 7 May 2011 11:02:34 -0000 Mailing-List: contact dev-help@lucene.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: dev@lucene.apache.org Delivered-To: mailing list dev@lucene.apache.org Received: (qmail 75767 invoked by uid 99); 7 May 2011 11:02:34 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 07 May 2011 11:02:34 +0000 X-ASF-Spam-Status: No, hits=-0.0 required=5.0 tests=RCVD_IN_DNSWL_LOW,SPF_NEUTRAL X-Spam-Check-By: apache.org Received-SPF: neutral (nike.apache.org: local policy) Received: from [74.125.82.48] (HELO mail-ww0-f48.google.com) (74.125.82.48) by apache.org (qpsmtpd/0.29) with ESMTP; Sat, 07 May 2011 11:02:27 +0000 Received: by wwi18 with SMTP id 18so2923014wwi.5 for ; Sat, 07 May 2011 04:02:07 -0700 (PDT) MIME-Version: 1.0 Received: by 10.227.197.201 with SMTP id el9mr4934002wbb.22.1304766127214; Sat, 07 May 2011 04:02:07 -0700 (PDT) Received: by 10.227.11.67 with HTTP; Sat, 7 May 2011 04:02:07 -0700 (PDT) In-Reply-To: References: <25CE504A-0E2E-4CF3-A3B4-42EC88F10DF4@gmail.com> <5695D270-424F-4324-8214-37543F9BC6AF@apache.org> Date: Sat, 7 May 2011 07:02:07 -0400 Message-ID: Subject: Re: modularization discussion From: Michael McCandless To: dev@lucene.apache.org Content-Type: text/plain; charset=ISO-8859-1 Content-Transfer-Encoding: quoted-printable X-Virus-Checked: Checked by ClamAV on apache.org OK I opened: https://issues.apache.org/jira/browse/LUCENE-3079 Mike http://blog.mikemccandless.com On Sat, May 7, 2011 at 6:46 AM, Michael McCandless wrote: > I agree! =A0And I think you're saying the same thing as Grant. > > Ie, others are fully free to refactor stuff, as long as they don't > hurt Solr/Lucene (functionality, performance). > > But you are tempering that with a nice dose of reality (successfully > factoring out faceting will be insanely hard). > > I very much agree with that. > > And, I (and other refactor-itchers) very much want to hear the > specific technical skepticism/concerns on a given module: that > assessment is awesome and very useful. =A0In fact, I love your > enumeration of how faceting is so well integrated into Solr so much > that I'll go open an issue (to factor out faceting), and put your list > in! > > I think this will mean, in practice, that the refactoring should > itself proceed in baby steps. =A0Ie, birthing a new faceting module, > iterating on it, etc., and then at some point cutting Solr over to it, > are two events likely spread out substantially in time. > > Freedom to refactor/poach is the bread and butter of open source. > > Mike > > http://blog.mikemccandless.com > > On Fri, May 6, 2011 at 4:35 PM, Chris Hostetter > wrote: >> >> : To me, the third camp is just saying the proof is in the pudding. =A0I= f >> : you want to refactor, then go for it. =A0Just make sure everything sti= ll >> : works, which of course I know people will (but part of that means >> : actually running Solr, IMO). =A0Perhaps, more importantly don't get ma= d >> : that if I have only one day a week to work on Lucene/Solr that I spend >> : it putting a specific feature in a specific place. =A0Just because >> : something can/should be modularized, doesn't mean that a person workin= g >> : in that area must do it before they add whatever they were working on. >> : For instance, if and when function queries are a module, I will add to >> : them there and be happy to do so. =A0In the meantime, I will likely ad= d to >> : them in Solr if that is something I happen to be interested in at that >> : time b/c I can certainly add a new function in a day, but I can't >> : refactor the whole module _and_ add my new function in a day. >> >> +1 >> >> I want to get that printed on a t-shirt >> >> the corrolarry issue in my mind... >> >> I am happily in favor of code reuse and modularization in the abstract, >> and when it works in practice i'm plesantly delighted. >> >> But when people talk about modularization as a goal, and make a laundry >> list things in solr that people think should be refactored into modules >> (w/o showing specifics of what that module would look like) then i have = a >> hard time buying into some of these ideas panning out in a way that: >> =A0a) is a useful module to people in and of itself >> =A0b) doesn't hamstring the evolution/performance in solr. >> >> To look at "faceting" as a concrete example, there are big the reasons >> faceting works so well in Solr: Solr has total control over the >> index, knows exactly when the index has changed to rebuild caches, has a >> strict schema so it can make sense of field types and >> pick faceting algos accordingly, has multi-phase distributed search >> approach to get exact counts efficiently across multiple shards, etc... >> (and there are still a lot of additional enhancements and improvements >> that can be made to take even more advantage of knowledge solr has becau= se >> it "owns" the index that we no one has had time to tackle) >> >> I find it really hard to picture a way that this code could be refactore= d >> into a reusable module in such a way that it could have an API that woul= d >> be easily usable outside of Solr -- and when i do get a glimmer of an >> inkling of what that might look like, that vision scares me because of h= ow >> that API might then "hobble" Solr's ability to leverage it's total contr= ol >> of the underlying index to add additional performance/features. >> >> To be crystal clear: I recognize that this is *my* hangup -- I am not >> suggesting that "I am short sighted and have little imagination >> therefore this code should never be modularized." >> >> I'm trying to explain why i *personally* am hesitant and sceptical of ho= w >> well modularizations of features like like this might actually work in >> practice, and why i'm not eager to jump in and contribute on a goal whos= e >> end result is something that i can't fully picture (and when i can pictu= re >> it, i'm a little scared by what i see) >> >> That doesn't mean i'm opposed to it happening -- i would love to live in >> the land of candy where houses are made of ginger bread and sugar plums >> grow on trees, I'm just too skeptical that such a land exists (or is as >> great as legend describes) to go slogging along on an epic journey to tr= y >> and reach it -- i'm too old for that shit. >> >> I'm certainly not going to stop anyone else fro going on that quest -- b= ut >> i am entitled to voice my skepticism and concerns, just as adventursome >> folks are entitled to ignore me. >> >> >> -Hoss >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org >> For additional commands, e-mail: dev-help@lucene.apache.org >> >> > --------------------------------------------------------------------- To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org For additional commands, e-mail: dev-help@lucene.apache.org