lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Simon Willnauer <>
Subject Re: Lucene & Solr a one way street?
Date Sun, 13 Mar 2011 16:06:29 GMT
On Sun, Mar 13, 2011 at 4:47 PM, Grant Ingersoll <> wrote:
> On Mar 13, 2011, at 10:23 AM, Simon Willnauer wrote:
>> Hey folks,
>> I have recently tried to push some refactorings towards moving stuff
>> from Solr to modules land to enable users of Lucene to benefit from
>> the developments that have been made in Solr land during the past with
>> very little success. Actually, it was a really disappointing
>> experience whenever  I tried to kick off issues towards this
>> direction. On LUCENE-2308 David asked a good question why FieldType is
>> not ported to Lucene rather than a new development.
>> I replied with:
>> {quote}
>> Moving stuff from Solr to Lucene involves lots of politics. It is way
>> easier to let Solr adopt eventually than fight your way through the
>> politics (this is my opinion though.)
>> {quote}
> I hadn't looked at 2308, but to me, if there are well written patches, then they should
be considered.  More modules make a lot of sense to me, as long as everyone is kept whole
and there are no performance losses.  Moving FTs to Lucene seems like a lot of work for little
benefit, but maybe that is just me.  Seems like most Lucene users like to roll their own
in that stuff or use Spring.

I agree it should be a lucene dev and we can catch up in solr land if
needed. But this is not about this issue I just used it to introduce
the actual point I tried to make.
>> Yet, while the answer to his question doesn't matter in this context
>> but it raised some other question from Roberts side:
>> {quote}
>> Then why do we still have merged codebases?
>> If this is the way things are, then we should un-merge the two projects.
>> because as a lucene developer, i spend a lot of time trying to do my
>> part to fix various things in Solr... if its a one-way-street then we
>> need to un-merge.
>> {quote}
>> The discussions on LUCENE-2883 changed my personal reception on how
>> things work here quite dramatically. I lost almost all enthusiasm to
>> even try to push developments towards moving things out of Solr and
>> into modules since literally every movement in this direction starts a
>> lot of politics (at least this is my understanding drawn from the
>> rather non-technical disagreements I have seen folks mentioning on
>> this issue). I don't care where those politics come from but in my
>> opinion we need to find an agreement how we deal with "stealing"
>> functionality form Solr and make them available to lucene users. My
>> personal opinion is that any refactoring, consolidation of APIs etc.
>> should NOT be influenced by the fact that they have been Solr private
>> and "might" influence further development on solr with regards to
>> backwards compatibility etc.
> I actually thought 2883 was a pretty good discussion.  The sum take away from it for
me was "go for it".  One person was hesitant about it.   I think sometimes you need to just
put up patches instead of having lengthy discussions.

I kindly disagree, for me it was a very frustrating, "oh well then
fuck it.." discussion and I have to admit I moved out of it quite
quickly. This issue is just an example for lots of discussion like
that where improvements are discussed to death. I agree we should let
patches speak first but I got frustrated before I could upload one.
This is the actual problem. So now you could say this is my personal
problem which is certainly true to some extend but it should not be
that way IMO. Let a patch come and discuss then would be the approach
that I favor.
>> Moving features to modules should be first priority and please correct
>> me if I am wrong this was one of the major reason why we merged the
>> code base.
> I don't think it is a first priority, but it is a benefit.  I also don't think it was
the majority reason for the merge.  I think the majority reason was that most of the Solr
committers were also Lucene committers and there was a fair amount of duplicated work and
a desire to be on the same version.  Modularization was/is also a benefit.

I disagree here as well. but I don't want to pull out those 500 emails
thread again :)

> FWIW, I think the merge for the most part has been successful in most places.  We have
better tested code, faster code, etc.
>> All users should benefit from the nice features which are
>> currently hidden in the solr code base. FunctionQuery is a tiny one
>> IMO and the frustration it caused on my side was immense. I don't even
>> wanna try to suggest to make replication, faceting or even the cloud
>> feature decoupled from Solr (I don't want to argue about the
>> feasibility or the amount of work this would be here! Just lemme say
>> one thing we can always branch and there is a large workforce out
>> there that is willing to work on stuff like that).
>> I can only agree with robert that if this is a one way street that the
>> merge makes no sense anymore.
> I guess the question people w/ Solr only hats on have (if there are such people), is
which way is that street going?  It seems like most people want to pull stuff out of Solr,
but they don't seem to want to put into it.  That's probably where some of the resistance
comes from.  If you want to modularize everything so that you can consume it outside of Solr,
it usually means you don't use Solr, which sometimes comes across that you don't care if the
modularization actually has a negative effect on Solr.  I'm all for modularization and enabling
everyone, but not at the cost of loss of performance in Solr.  As tightly coupled as Solr
is, it's pretty damn fast and resilient.  Show me that you keep that whole and I'll be +1
on everything.

Grant, I think this is simply not true. I see way more effort in
putting stuff into solr which comes from lucene than the other way
around. Robert has done a tremendous amount of work to improve solr
and all of us try very hard to align solr with the development in
lucene. I think it the way from lucene to solr is less visible due to
the changes in lucene always contain fixes in solr land if needed. You
know all the refactoring like on ReaderContext etc where drastic
changes in solr and almost every dev has those changes. This simply
not true but I just realized how this reception could have build up.

About the performance - I think we can sort that out!
> You also have to keep in mind that some of these things, replication for instance, rely
on Solr things.  Are you really going to more the HTTP protocols to just Lucene?  What does
that even mean?  Lucene is a Java API.  It doesn't assume containers, etc.  Solr is the
thing that delivers Lucene in a container.

I don't know but its not much more than replicating segments really on
a very high level. not an issue here just an example.

> To me, lower level things like faceting and function query belong as modules, but higher
level container things belong in Solr.  Not sure where schema lands, but likely still in
Solr.  Most Lucene users seem resistant to XML based configuration either b/c they want to
code it themselves or already have a container to do that (Spring, etc.)

>> Refactoring will bring a lot of benefits
>> to both lucene AND solr. I also wanna quote Earwin here (I don't find
>> the issue right now - this quote is from my memory): "we should try to
>> decouple things rather than couple them even more tight". I moved out
>> of all Solr issues since then and I am not willing to do any work on
>> it until we find an agreement on this.
> Well, I don't think that is really helpful, but you are of course free to do as you wish.
 Frankly, I think you should just put up patches and make commits.  You have full commit
rights just like everyone else.  A well written patch that is thoughtful is worth 1000x any
discussion on the theory of such a patch.
> At any rate, I appreciate the well-thought out email and glad you raised the issue.


> -Grant
> ---------------------------------------------------------------------
> To unsubscribe, e-mail:
> For additional commands, e-mail:

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message