lucene-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ted Dunning (Issue Comment Edited) (JIRA)" <j...@apache.org>
Subject [jira] [Issue Comment Edited] (SOLR-2765) Shard/Node states
Date Sun, 09 Oct 2011 03:55:29 GMT

    [ https://issues.apache.org/jira/browse/SOLR-2765?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13123607#comment-13123607
] 

Ted Dunning edited comment on SOLR-2765 at 10/9/11 3:55 AM:
------------------------------------------------------------

{quote}
the idea being if node after node comes up and triggers the watch, if there is already a state
update scheduled (5 seconds after the watch is triggered by default), a new update is not
asked for - eg if nodes are firing up by the hundreds, the state is still only updated once
every 5 seconds.
{quote}
Something like this is a great approach to metering the number of watch notifications that
need to be sent by the ZK server.  I would prefer to change the strategy slightly so that
an update is scheduled immediately if no update has happened in the last 5 seconds, but is
not scheduled more often than about every 5 seconds.  The update can then unconditionally
set the requisite watches.

This means that you get immediate response to the occasional change, but never require more
than n/5 updates per second where n is the number of clients.  You can even make the delay
be something like n/log_2 ( n )/20 for n >= 2 where n is the number of clients.  This gives
fast response for small clusters with notification rates that grow slowly for large clusters.
 At 1000 clients, you get 200 notifications per second worst case and at 8000 clients you
get 500 notifications per second.  In return for this very moderate transaction growth rate,
 it takes longer to register that a new cluster has appeared.  If many nodes register in rapid
succession, it takes 16 seconds for 8000 clients to all get the news, 5 seconds for 1000 clients
and just less than a second for 100 clients.  

My guess is that 100 clients is a very large number.

If counting clients is a pain, the constant value of 5 seconds delay is pretty good.

                
      was (Author: tdunning):
    {quote}
the idea being if node after node comes up and triggers the watch, if there is already a state
update scheduled (5 seconds after the watch is triggered by default), a new update is not
asked for - eg if nodes are firing up by the hundreds, the state is still only updated once
every 5 seconds.
{quote}
Something like this is a great approach to metering the number of watch notifications that
need to be sent by the ZK server.  I would prefer to change the strategy slightly so that
an update is scheduled immediately if no update has happened in the last 5 seconds, but is
not scheduled more often than about every 5 seconds.  The update can then unconditionally
set the requisite watches.

This means that you get immediate response to the occasional change, but never require more
than n/5 updates per second where n is the number of clients.  You can even make the delay
be something like n/log_2(n)/20 for n >= 2 where n is the number of clients.  This gives
fast response for small clusters with notification rates that grow slowly for large clusters.
 At 1000 clients, you get 200 notifications per second worst case and at 8000 clients you
get 500 notifications per second.  In return for this very moderate transaction growth rate,
 it takes longer to register that a new cluster has appeared.  If many nodes register in rapid
succession, it takes 16 seconds for 8000 clients to all get the news, 5 seconds for 1000 clients
and just less than a second for 100 clients.  

My guess is that 100 clients is a very large number.

If counting clients is a pain, the constant value of 5 seconds delay is pretty good.

                  
> Shard/Node states
> -----------------
>
>                 Key: SOLR-2765
>                 URL: https://issues.apache.org/jira/browse/SOLR-2765
>             Project: Solr
>          Issue Type: Sub-task
>          Components: SolrCloud, update
>            Reporter: Yonik Seeley
>             Fix For: 4.0
>
>         Attachments: incremental_update.patch, scheduled_executors.patch, shard-roles.patch
>
>
> Need state for shards that indicate they are recovering, active/enabled, or disabled.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

---------------------------------------------------------------------
To unsubscribe, e-mail: dev-unsubscribe@lucene.apache.org
For additional commands, e-mail: dev-help@lucene.apache.org


Mime
View raw message