hbase-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stack <st...@duboce.net>
Subject Re: [DISCUSS] No regions on Master node in 2.0
Date Mon, 25 Apr 2016 18:20:31 GMT
On Fri, Apr 8, 2016 at 1:42 AM, Elliott Clark <eclark@apache.org> wrote:

> # Without meta on master, we double assign and lose data.
> That is currently a fact that I have seen over and over on multiple loaded
> clusters. Some abstract clean up of deployment vs losing data is a
> no-brainer for me. Master assignment, region split, region merge are all
> risky, and all places that HBase can lose data. Meta being hosted on the
> master makes communication easier and less flakey. Running ITBLL on a loop
> that creates a new table every time, and without meta on master everything
> will fail pretty reliably in ~2 days. With meta on master things pass MUCH
> more.
The above is a problem of branch-1?

The discussion is what to do in 2.0 with the assumption that master state
would be done up on procedure v2 making most of the transitions now done
over zk and hbase:meta instead local to the master with only the final
state published to a remote meta (an RPC but if we can't make RPC work
reliably in our distributed system, thats a bigger problem).

> # Master hosting the system tables locates the system tables as close as
> possible to the machine that will be mutating the data.
> Data locality is something that we all work for. Short circuit local reads,
> Caching blocks in jvm, etc. Bringing data closer to the interested party
> has a long history of making things faster and better. Master is in charge
> of just about all mutations of all systems tables. It's in charge of
> changing meta, changing acls, creating new namespaces, etc. So put the
> memstore as close as possible to the system that's going to mutate meta.

Above is fine except for the bit where we need to be able to field reads.
Lets distribute the data to be read over the cluster rather than treat meta
reads with kid gloves hosted on a 'special' server; let these 'reads' be
like any other read the cluster takes (see next point)

> # If you want to make meta faster then moving it to other regionservers
> makes things worse.
> Meta can get pretty hot. Putting it with other regions that clients will be
> trying to access makes everything worse. It means that meta is competing
> with user requests. If meta gets served and other requests don't, causing
> more requests to meta; or requests to user regions get served and other
> clients get starved.
> At FB we've seen read throughput to meta doubled or more by swapping it to
> master. Writes to meta are also much faster since there's no rpc hop, no
> queueing, to fighting with reads. So far it has been the single biggest
> thing to make meta faster.
Is this just because meta had a dedicated server?


> On Thu, Apr 7, 2016 at 10:11 PM, Stack <stack@duboce.net> wrote:
> > I would like to start a discussion on whether Master should be carrying
> > regions or not. No hurry. I see this thread going on a while and what
> with
> > 2.0 being a ways out yet, there is no need to rush to a decision.
> >
> > First, some background.
> >
> > Currently in the master branch, HMaster hosts 'system tables': e.g.
> > hbase:meta. HMaster is doing more than just gardening the cluster,
> > bootstrapping and keeping all up and serving healthy as in branch-1; in
> > master branch, it is actually in the write path for the most critical
> > system regions.
> >
> > Master is this way because HMaster and HRegionServer servers have so much
> > in common, they should be just one binary, w/ HMaster as any other server
> > with the HMaster function a minor appendage runnable by any running
> > HRegionServer.
> >
> > I like this idea, but the unification work was just never finished. What
> is
> > in master branch is a compromise. HMaster is not a RegionServer but a
> > sort-of RegionServer doing part serving. So we have HMaster role, a new
> > part-RegionServer-carrying-special-regions role and then a full-on
> > HRegionServer role. We need to fix this messyness. We could revert to
> plain
> > branch-1 roles or carrying the
> > HMaster-function-is-something-any-RegionServer-could-execute through to
> > completion.
> >
> > More background from a time long-past with good comments by the likes of
> > our Francis Liu and Mighty Matteo Bertozzi are here [1], on unifying
> master
> > and meta-serving. Slightly related are old discussions on being able to
> > scale by splitting meta with good comments by our Elliott Clark [2].
> >
> > Also for consideration, the landscape has since changed. [1] was written
> > before we had ProcedureV2 available to us where we could record
> > intermediate transition states local to the Master rather than remote as
> > intermediate updates to an hbase:meta over rpc running on another node.
> >
> > Enough on the background.
> >
> > Let me provoke discussion by making the statement that we should undo
> > HMaster carrying any regions ever; that the HMaster function is work
> enough
> > for a single dedicated server and that it important enough that it cannot
> > take a background role on a serving RegionServer (I could go back from
> this
> > position if evidence HMaster role could be backgrounded). Notions of a
> > Master carrying system tables only are just not on given system tables
> will
> > be too big for a single server especially when hbase:meta is split (so we
> > can scale). This simple distinction of HMaster and RegionServer roles is
> > also what our users know and have gotten used to so needs to be a good
> > reason to change it (We can still pursue the single binary that can do
> > HMaster or HRegionServer role determined at runtime).
> >
> > Thanks,
> > St.Ack
> >
> > 1.
> >
> >
> https://docs.google.com/document/d/1xC-bCzAAKO59Xo3XN-Cl6p-5CM_4DMoR-WpnkmYZgpw/edit#heading=h.j5yqy7n04bkn
> > 2.
> >
> >
> https://docs.google.com/document/d/1eCuqf7i2dkWHL0PxcE1HE1nLRQ_tCyXI4JsOB6TAk60/edit#heading=h.80vcerzbkj93
> >

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message