lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Grant Ingersoll <>
Subject Re: Lucene & Solr a one way street?
Date Sun, 13 Mar 2011 15:47:11 GMT

On Mar 13, 2011, at 10:23 AM, Simon Willnauer wrote:

> Hey folks,
> I have recently tried to push some refactorings towards moving stuff
> from Solr to modules land to enable users of Lucene to benefit from
> the developments that have been made in Solr land during the past with
> very little success. Actually, it was a really disappointing
> experience whenever  I tried to kick off issues towards this
> direction. On LUCENE-2308 David asked a good question why FieldType is
> not ported to Lucene rather than a new development.
> I replied with:
> {quote}
> Moving stuff from Solr to Lucene involves lots of politics. It is way
> easier to let Solr adopt eventually than fight your way through the
> politics (this is my opinion though.)
> {quote}

I hadn't looked at 2308, but to me, if there are well written patches, then they should be
considered.  More modules make a lot of sense to me, as long as everyone is kept whole and
there are no performance losses.  Moving FTs to Lucene seems like a lot of work for little
benefit, but maybe that is just me.  Seems like most Lucene users like to roll their own in
that stuff or use Spring.

> Yet, while the answer to his question doesn't matter in this context
> but it raised some other question from Roberts side:
> {quote}
> Then why do we still have merged codebases?
> If this is the way things are, then we should un-merge the two projects.
> because as a lucene developer, i spend a lot of time trying to do my
> part to fix various things in Solr... if its a one-way-street then we
> need to un-merge.
> {quote}
> The discussions on LUCENE-2883 changed my personal reception on how
> things work here quite dramatically. I lost almost all enthusiasm to
> even try to push developments towards moving things out of Solr and
> into modules since literally every movement in this direction starts a
> lot of politics (at least this is my understanding drawn from the
> rather non-technical disagreements I have seen folks mentioning on
> this issue). I don't care where those politics come from but in my
> opinion we need to find an agreement how we deal with "stealing"
> functionality form Solr and make them available to lucene users. My
> personal opinion is that any refactoring, consolidation of APIs etc.
> should NOT be influenced by the fact that they have been Solr private
> and "might" influence further development on solr with regards to
> backwards compatibility etc.

I actually thought 2883 was a pretty good discussion.  The sum take away from it for me was
"go for it".  One person was hesitant about it.   I think sometimes you need to just put up
patches instead of having lengthy discussions.

> Moving features to modules should be first priority and please correct
> me if I am wrong this was one of the major reason why we merged the
> code base.

I don't think it is a first priority, but it is a benefit.  I also don't think it was the
majority reason for the merge.  I think the majority reason was that most of the Solr committers
were also Lucene committers and there was a fair amount of duplicated work and a desire to
be on the same version.  Modularization was/is also a benefit.

FWIW, I think the merge for the most part has been successful in most places.  We have better
tested code, faster code, etc.

> All users should benefit from the nice features which are
> currently hidden in the solr code base. FunctionQuery is a tiny one
> IMO and the frustration it caused on my side was immense. I don't even
> wanna try to suggest to make replication, faceting or even the cloud
> feature decoupled from Solr (I don't want to argue about the
> feasibility or the amount of work this would be here! Just lemme say
> one thing we can always branch and there is a large workforce out
> there that is willing to work on stuff like that).
> I can only agree with robert that if this is a one way street that the
> merge makes no sense anymore.

I guess the question people w/ Solr only hats on have (if there are such people), is which
way is that street going?  It seems like most people want to pull stuff out of Solr, but they
don't seem to want to put into it.  That's probably where some of the resistance comes from.
 If you want to modularize everything so that you can consume it outside of Solr, it usually
means you don't use Solr, which sometimes comes across that you don't care if the modularization
actually has a negative effect on Solr.  I'm all for modularization and enabling everyone,
but not at the cost of loss of performance in Solr.  As tightly coupled as Solr is, it's pretty
damn fast and resilient.  Show me that you keep that whole and I'll be +1 on everything.

You also have to keep in mind that some of these things, replication for instance, rely on
Solr things.  Are you really going to more the HTTP protocols to just Lucene?  What does that
even mean?  Lucene is a Java API.  It doesn't assume containers, etc.  Solr is the thing that
delivers Lucene in a container.

To me, lower level things like faceting and function query belong as modules, but higher level
container things belong in Solr.  Not sure where schema lands, but likely still in Solr. 
Most Lucene users seem resistant to XML based configuration either b/c they want to code it
themselves or already have a container to do that (Spring, etc.)

> Refactoring will bring a lot of benefits
> to both lucene AND solr. I also wanna quote Earwin here (I don't find
> the issue right now - this quote is from my memory): "we should try to
> decouple things rather than couple them even more tight". I moved out
> of all Solr issues since then and I am not willing to do any work on
> it until we find an agreement on this.

Well, I don't think that is really helpful, but you are of course free to do as you wish.
 Frankly, I think you should just put up patches and make commits.  You have full commit rights
just like everyone else.  A well written patch that is thoughtful is worth 1000x any discussion
on the theory of such a patch.

At any rate, I appreciate the well-thought out email and glad you raised the issue.

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message