incubator-blur-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Aaron McCurry <amccu...@gmail.com>
Subject Re: Shard takeover behavior
Date Tue, 18 Mar 2014 10:54:18 GMT
On Tue, Mar 18, 2014 at 1:30 AM, Ravikumar Govindarajan <
ravikumar.govindarajan@gmail.com> wrote:

> >
> > If I understand this one.  Favor the primary response until a certain
> > amount of time has passed then fall back to the secondary response
> assuming
> > it's available to return.
>
>
> Exactly. This is one such option. Another option is the first-past-the-post
>
> Buffer cache?  Are you referring to block cache?
>
>
> Yup. Was referring to the block-cache here. But like you said, we can just
> let it fall off the LRU
>
>  The interesting thing here is that Blur is fully committed to disk (HDFS)
>
> upon each mutate
>
> I think this is a new feature that I have missed in Blur. Will for sure
> check it out. This auto-solves the stale-read issue also
>
> The problem now is, I am doing quite low-level changes on top of blur. Some
> of them are..
>
> 1. Online Shard-Creation
> 2. Externalizing RowId->Shard mapping via BlurPartitioner
> 3. Splitting shards upon reaching configured size
> 4. Secondary read-only shard for availability...
>

I would love hear about more of the details of the implementations of
these.  :-)


>
> and many more such stuff needed for our app
>
> Hope to share and get feedback for these changes from Blur community once
> the system survives a couple of production-cycles.
>

That would be awesome.  Based on your other email, I would strongly
recommend you take a look at the 0.2.2 codebase.  It has MANY fixes,
performance improvements, and stability enhancements.  Let us know if you
have any questions.

Aaron


>
> --
> Ravi
>
>
> On Mon, Mar 17, 2014 at 7:17 PM, Aaron McCurry <amccurry@gmail.com> wrote:
>
> > On Sat, Mar 15, 2014 at 12:57 PM, Ravikumar Govindarajan <
> > ravikumar.govindarajan@gmail.com> wrote:
> >
> > > Aaron,
> > >
> > > I was thinking about another way of utilizing read-only shards
> > >
> > > Instead of logic/intelligence of finding a primary replica
> > struggling/down,
> > > can we opt for pushing a logic on client-side?
> > >
> > > We can take a few approaches as below
> > >
> > > 1. Query both primary/secondary shards in parallel and return which
> ever
> > > comes first
> >
> >
> > > 2. Query both primary/secondary shards in parallel. Wait for primary
> > > response as per configured delay. If not forthcoming, return
> secondary's
> > > response
> > >
> >
> > If I understand this one.  Favor the primary response until a certain
> > amount of time has passed then fall back to the secondary response
> assuming
> > it's available to return.
> >
> >
> > >
> > > These are useful only when client agrees for a "stale-read" scenario.
> > > "stale-read" in this case will be the last-commit of the index.
> > >
> > > What I am aiming at, is in the case of layout-conscious apps [layout
> does
> > > not change when VM update/crash/hang is restarted], we can always
> > fall-back
> > > on replica reads, resulting in greater availability but lesser
> > consistency
> > >
> > > A secondary-replica layout need to be present in ZK. Replica-shards
> > should
> > > be always served from a server other than primary. May be we can
> > switch-off
> > > buffer-cache for replica reads, as it is used only temporarily
> > >
> >
> > Buffer cache?  Are you referring to block cache?  Or a query cache?  Just
> > as a FYI, Blur's query cache is currently disabled.  As for the block
> > cache, maybe.  The block cache seems to help performance quite a bit and
> > usually is does so at little cost.  Also, we could flush the secondary
> > shard from the cache from time to time.  Or we could just let it fall out
> > of the LRU.
> >
> >
> > >
> > > 95% apps queue their indexing operations and can always retry after
> > primary
> > > comes back online.
> > >
> >
> > The interesting thing here is that Blur is fully committed to disk (HDFS)
> > upon each mutate.  So assuming that the secondary shard has refreshed,
> the
> > primary shard being down just means that you can't write to that shard.
> >  Reads should be in the same state.
> >
> >
> > >
> > > Please let me know your views on this
> > >
> >
> > I like all these ideas, the only thing I would add is that we we would
> need
> > to build these sort of options into Blur on a configured per-table basis.
> >  The querying both primary and secondary shards at the same time could
> > produce the most consistent respond times but at the cost of CPU
> resources
> > (obviously).
> >
> > Thanks for the thoughts and ideas!  I like it!
> >
> > Aaron
> >
> >
> > >
> > > --
> > > Ravi
> > >
> > >
> > > On Sat, Mar 8, 2014 at 8:56 PM, Aaron McCurry <amccurry@gmail.com>
> > wrote:
> > >
> > > > On Fri, Mar 7, 2014 at 5:42 AM, Ravikumar Govindarajan <
> > > > ravikumar.govindarajan@gmail.com> wrote:
> > > >
> > > > > >
> > > > > > Well it works that way for OOMs and for when the process drop
> hard
> > > > (Think
> > > > > > kill -9).  However when a shard server is shutdown it currently
> > ends
> > > > it's
> > > > > > session in ZooKeeper, thus triggering a layout change.
> > > > >
> > > > >
> > > > > Yes, may be we can have a config to determine whether it shud
> > > > end/maintain
> > > > > the session in ZK when doing a normal shutdown and then subsequent
> > > > restart.
> > > > > By this way, both MTTR-conscious and layout-conscious settings can
> be
> > > > > supported.
> > > > >
> > > >
> > > > That's a neat idea.  Once we have shards being served on multiple
> > servers
> > > > we should definitely take a look at this.  When we implement the
> > > > multi-shard serving I would guess that there will be 2 layout
> > strategies
> > > > (they might be implemented together).
> > > >
> > > > 1. Would be to get the N replicas online on different servers.
> > > > 2. Would the writing leader for the shard, assuming that it's needed.
> > > >
> > > >
> > > > >
> > > > > How do you think we can detect that a particular shard-server is
> > > > > struggling/shut-down and hence incoming search-requests need to go
> to
> > > > some
> > > > > other server?
> > > > >
> > > > > I am listing few paths off the top of my head
> > > > >
> > > > > 1. Process baby-sitters like supervisord, alerting controllers
> > > > > 2. Tracking first network-exception in controller and diverting to
> > > > > read-only
> > > > >     instance. Periodically may be re-try
> > > > > 3. Take a statistics based decision, based on previous response
> times
> > > > etc..
> > > > >
> > > >
> > > > Anding to this one and this may be obvious but measuring the response
> > > time
> > > > in comparison with other shards.  Meaning if the entire cluster is
> > > > experiencing an increase in load and all responses times are
> increasing
> > > we
> > > > wouldn't want to start killing off shard servers inadvertently.
> >  Looking
> > > > for outliers.
> > > >
> > > >
> > > > > 4. Build some kind of leasing mechanism in ZK etc...
> > > > >
> > > >
> > > > I think that all of these are good approaches.  Likely to determine
> > that
> > > a
> > > > node is misbehaving and should be killed/not used anymore we would
> want
> > > > multiple ways to measure that condition and then vote on the need
> kick
> > > out.
> > > >
> > > >
> > > > Aaron
> > > >
> > > > >
> > > > > --
> > > > > Ravi
> > > > >
> > > > >
> > > > > On Fri, Mar 7, 2014 at 8:01 AM, Aaron McCurry <amccurry@gmail.com>
> > > > wrote:
> > > > >
> > > > > > On Thu, Mar 6, 2014 at 6:30 AM, Ravikumar Govindarajan <
> > > > > > ravikumar.govindarajan@gmail.com> wrote:
> > > > > >
> > > > > > > I came to know about zk.session.timeout variable just now,
> while
> > > > > reading
> > > > > > > more about this problem.
> > > > > > >
> > > > > > > This will only trigger dead-node notification after the
> > configured
> > > > > > timeout
> > > > > > > exceeds. Setting it to 3-4 mins must be fine for OOMs and
> > > > > > rolling-restarts.
> > > > > > >
> > > > > >
> > > > > > Well it works that way for OOMs and for when the process drop
> hard
> > > > (Think
> > > > > > kill -9).  However when a shard server is shutdown it currently
> > ends
> > > > it's
> > > > > > session in ZooKeeper, thus triggering a layout change.
> > > > > >
> > > > > >
> > > > > > >
> > > > > > > Only extra stuff I am looking for, is to divert search
calls
> to a
> > > > > > read-only
> > > > > > > shard instance during this 3-4 mins time to avoid mini-outages
> > > > > > >
> > > > > >
> > > > > > Yes, and I think that the controllers will automatically spread
> the
> > > > > queries
> > > > > > across those servers that are online.  The BlurClient class
> already
> > > > > takes a
> > > > > > list of connection strings and treats all connections as equals.
> >  For
> > > > > > example, it's current use is to provide the client with all
the
> > > > > controllers
> > > > > > connection strings.  Internally if any one of the controllers
> goes
> > > down
> > > > > or
> > > > > > has a network issue another controller is automatically retried
> > > without
> > > > > the
> > > > > > user having to do anything.  There is back off, ping, and pooling
> > > logic
> > > > > in
> > > > > > the BlurClientManager that the BlurClient utilizes.
> > > > > >
> > > > > > Aaron
> > > > > >
> > > > > >
> > > > > > >
> > > > > > > --
> > > > > > > Ravi
> > > > > > >
> > > > > > >
> > > > > > >
> > > > > > > On Thu, Mar 6, 2014 at 3:34 PM, Ravikumar Govindarajan
<
> > > > > > > ravikumar.govindarajan@gmail.com> wrote:
> > > > > > >
> > > > > > > > What do you think of giving an extra leeway for shard-server
> > > > >  failover
> > > > > > > > cases?
> > > > > > > >
> > > > > > > > Ex: Whenever a shard-server process gets killed, the
> > > > controller-node
> > > > > > does
> > > > > > > > not immediately update-layout, but rather mark it
as a
> suspect.
> > > > > > > >
> > > > > > > > When we have a read-only back-up of shard, searches
can
> > continue
> > > > > > > > unhindered. Indexing during this time can be diverted
to a
> > queue,
> > > > > which
> > > > > > > > will store and retry-ops, when shard-server comes
online
> again.
> > > > > > > >
> > > > > > > > Over configured number of attempts/time, if the shard-server
> > does
> > > > not
> > > > > > > come
> > > > > > > > up, then one controller-server can authoritatively
mark it as
> > > down
> > > > > and
> > > > > > > > update the layout.
> > > > > > > >
> > > > > > > > --
> > > > > > > > Ravi
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message