lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hoss Man (JIRA)" <>
Subject [jira] [Commented] (SOLR-2822) don't run update processors twice
Date Wed, 02 May 2012 01:12:51 GMT


Hoss Man commented on SOLR-2822:

Yonik suggested that this issue might be a good place for me to start learning more about
the internals of Solr Cloud.  First some background on what i started thinking about (archived
for posterity) but have since dismissed as a bad idea...

{panel:title=My Initial Bad Idea That I Think Sucks| borderStyle=dashed| borderColor=#ccc|
titleBGColor=#F7D6C1| bgColor=#F7E6D1} 

my first impression when yonik mentioned the problem was along the lines of the initial bug

bq. perhaps by using a processor chain dedicated to sub-updates first glance, this seemed like it would be an easy way to deal with the problem in a
way that would be very easy to understand when looking at your chains in the solrconfig.xml,
and wouldn't involve any sort of "magic" that might be hard for users to understand about
how distributed updates work...

* some chain(s) (probably the default) would end with DistributedUpdateProcessorFactory, which
would be configured with a "{{distrib.chain}}" init param value having some value {{$foo}}
* some other chain would be named {{$foo}}, and would end with RunUpdateProcessorFactory
* when DistributedUpdateProcessorFactory was triggered, it would forward the request to all
of the nodes (including itself) using the configured {{distrib.chain}}

But as i started looking at the code a bit more, i realized that was a trivially naive understanding
of what's needed here -- mainly because i completely forgot about that fact that there are
two types of forwarding:

 * some arbitrary node gets the initial request from a client, and forwards to the leader
 * the leader does version checking/adding on the command/docs, and forwards to *all* the

This is when i started thinking that maybe there should be _three_ named chains, and started
understanding Jan's comment above about choosing which logic should happen "on the leader

But the more i thought about the more the more the idea of specifying certain logic needed
to be run on the leader node made less sense.  I can think of lots of reasons for why you
would care about whether an UpdateProcessor ran on one node or N nodes (mainly the trade off
between an expensive computation you might only wnat to do once, vs processors that will add
a lot of fields to the request so you might want to defer until after distribution to minimize
data over the wire) but i couldn't come up with any compelling reason why, if you have a processor
you only want to execute _once_ you would care if it happens on the first node randomly selected
by the client, or if you really wanted to ensure it happened on the leader -- particularly
since leader election/promotion can happen to any node at any time, so you couldn't even use
the leader logic to try and take advantage of some local resources on a particular machine.

But even if we go back to the "two chain" model i suggested above, because of the leader logic,
the second chain would need to start with some UpdateProcessor that handled the "version"
& "forward to all nodes" work currently handled by DistributedUpdateProcessor when "isLeader"
is true.  Which means people editing their configs would need to not only remember to put
some sort of "SendDistributedUpdateProcessorFactory" at the end of the chains they expect
clients to use, but also some sort of "RecieveDistributedUpdateProcessorFactory" at the beginning
of the chain that "SendDistributedUpdateProcessorFactory" would be pointing at (or maybe both
are just instances of DistributedUpdateProcessorFactory and we make it smart enough to know
when it's the start vs end of the chain) ... which starts sounding really tedious and easy
to screw up -- not to mention it doesn't play very nicely with the idea of a hardcoded default
chain users get for free if they have none in their config; and it thoroughly complicates
the story of someone who starts with the example (no declared chains) in cloud mode and then
wants to customize a little, say by adding SignatureUpdateProcessor -- they wouldn't be able
to just configure a chain starting with SignatureUpdateProcessor, they'd need to add two,
and be careful about how they start and end.


i'm all for transparency and eliminating "magic" by having things spelled out in the configs,
but even for me this idea was starting to seem like a serious pain in the ass.

Which leads me to the idea i do think has merit...

{panel:title=My Current Idea That I Don't Think Sucks| borderStyle=dashed| borderColor=#ccc|
titleBGColor=#D6F7C1| bgColor=#E6F7D1} 

I think in an ideal world... 
 * Solr Cloud users should be able to just declare a straightforwrd chain of processors and
not *need* to worry about which nodes they run on (ie: any chain they used pre-cloud should
still work)
 * if a user *wants* to care what parts of the chain run once vs on every node, it should
be simple to indicate that decision by putting some "marker" processor (ie: DistributedUpdateProcessorFactory)
in the chain denoting when distribution should happen.
 * If you are using cloud mode, and don't specify where the distributed update logic happens,
then Solr should pick for you -- either "first" before any other processors, or "last" just
before RunUpdateProcessor ... I don't have an opinion which is better, i'm sure other people
who have been experimenting with Solr Cloud for a while can tell me.

(I'm usually the person opposed to doing things magically behind the scenes like this, but
"Solr Cloud" is becoming a central enough concept in Solr, with many components doing special
things if "zkEnabled", that I think moving forward it's "ok" to treat distributed updating
as the norm in cloud mode and optimize for it.)

The most straight forward way i can think of to do this would be:
 * if Solr is in "cloud mode" then when SolrCore is initializing the chains from solrconfig.xml,
any chain that doesn't include DistributedUpdateProcessorFactory should have it added automatically
(alternatively: maybe we only add it if RunUpdateProcessor is in the chain, so anyone using
chains w/o RunUpdateProcessor -- for some weird bizare purpose we can't imagine -- won't be
surprised by DistributedUpdateProcessorFactory getting added magically)
 * DistributedUpdateProcessor should add some param when forwarding to any other node indicating
that the request is being distributed
 * when an update request is received, if it has this special param in it, then any processor
in the chain prior to DistributedUpdateProcessorFactory is skipped

This idea is very similar to part of what Jan suggested in his comment above...

bq. The Distrib processor could set some state info on the request to the next node so that
chain processing could continue where it left off. E.g. &update.chain.nextproc=<name/id-of-next-proc>.
This would require introduction of named processor instances.

...but what i'm thinking of wouldn't require named processors and would be specific to distributed
updates (but wouldn't *precluded* named processors and more enhanced logic down the road if
someone wanted it).

I think this would be fairly feasible just by making some small modifications to DistributedUpdateProcessor
(to add the new special param when forwarding) and UpdateRequestProcessorChain (to inject
the DistributedUpdateProcessorFactory if cloud mode, and to skip up to the DistributedUpdateProcessorFactory
if the param is set).  I do however still think we should generalize somewhat:

 * DistributedUpdateProcessorFactory should be made to implement some marker interface with
no methods (ie: DistributedUpdateMarker)
 * UpdateRequestProcessorChain.init should scan for instances of DistributedUpdateMarker in
the chain (instead of looking explicitly for DistributedUpdateProcessorFactory) when deciding
whether to inject a new DistributedUpdateProcessorFactory into the chain
 * UpdateRequestProcessorChain.createProcessor should scan for instances of DistributedUpdateMarker
in the chain (instead of looking explicitly for DistributedUpdateProcessorFactory) when "skipping
ahead" if the special param is found in the request

...that way advanced users can implement their own distributed update processor implementing
that interface and register it explicitly in their chain if they are so inclined, or implement
a NoOp update processor implementing that interface if they want to bipass the magic completley.

As a possible optimization/simplification to what gets sent over the wire, the new param we
add that UpdateRequestProcessorChain would start looking for to "skip ahead" in the chain
could replace the existing "leader" boolean param DistributedUpdateProcessor currently uses
(aka: {{SEEN_LEADER}}) by having an enum style param (perhaps called "{{update.distrib}}"

 * {{none}} - default if unset, means no distribution has happend
 * {{toleader}} - means the request is being sent to the leader
 * {{fromleader}} - means the leader is sending the request to all nodes

UpdateRequestProcessorChain would only care if the value is not "{{none}}", in which case
it would skip ahead to the DistributedUpdateMarker in the chain.  DistributedUpdateProcessorFactory
would care if the value is "{{toleader}}" or "{{fromleader}}" in which case it's logic would
toggle in the same way it does currently for SEEN_LEADER.


So that's my idea.  As i mentioned, this is really the first time i've looked into any of
the "Solr Cloud" internals, and it's very possible i may still be missing something major.
So if you spot any holes in this idea, please let me know -- otherwise i'll try to take a
stab at a patch in the next few days.

> don't run update processors twice
> ---------------------------------
>                 Key: SOLR-2822
>                 URL:
>             Project: Solr
>          Issue Type: Sub-task
>          Components: SolrCloud, update
>            Reporter: Yonik Seeley
>             Fix For: 4.0
> An update will first go through processors until it gets to the point where it is forwarded
to the leader (or forwarded to replicas if already on the leader).
> We need a way to skip over the processors that were already run (perhaps by using a processor
chain dedicated to sub-updates?

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators:!default.jspa
For more information on JIRA, see:


To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message