hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "stack (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HBASE-14614) Procedure v2: Core Assignment Manager
Date Fri, 06 Jan 2017 23:20:58 GMT

     [ https://issues.apache.org/jira/browse/HBASE-14614?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

stack updated HBASE-14614:
--------------------------
    Release Note: 
Replaces the AssignmentManager with a new procedurev2-based AssignmentManager

h1. AMv2
Puts AssignmentManager up on top of the ProcedureV2 state machine with persistence engine.
Each assignment atom is now a Procedure implementation; e.g. an AssignProcedure and an UnassignProcedure.
Molecules of aggregated Procedures are used to do more involved assignment steps: e.g. the
move region procedure is made of an Unassign followed by an Assign subprocedure.

AMv2 is 1500 lines. Old AM was near 4000. Functionality has been moved out to Procedures.
In-memory states of regions and servers has been cleaned up stored in new RegionStates implementation.
RegionStateStore takes care of publishing final region state out to the hbase:meta table.

New RemoteProcedureDispatcher/RSProcedureDispatcher runs the Procedure-based assignments ‘remotely’.
Knows about ‘servers’. Does aggregation of assignments by time on a time/count basis so
can send procedures in batches rather than one per RPC. Procedure status comes back on the
back of the RegionServer heartbeat reporting online regions. The response is passed to the
AMv2 to ‘process’. It will check against the in-memory state. If there is a mismatch,
it fences out the RegionServer on the assumption that something went wrong on the RS side.Timeouts
trigger retries. The Procedure machine ensures only one operation at a time on any one region/table
using locking and smarts about what is serial and what can be run concurrently.

New accounting of RegionServer version will be used running rolling restarts.

‘States’ -- OPENING, CLOSING, etc. -- are now in-memory in-the-master only serialized
out to the ProcedureV2 WAL. They are no longer persisted to ZooKeeper.

h2. Assign Detail
The Assign starts by pushing the "assign" operation to the AssignmentManager and then will
go into a “waiting" state. The AM will batch the "assign" requests and ask the Balancer
where to put the region (the various policies will be respected: retain, round-robin, random).
Once the AM and the balancer have found a place for the region, the procedure will be resumed
and an "open region" request will be placed in the Remote Dispatcher queue, and the procedure
once again will go into a "waiting state".  The Remote Dispatcher will batch the various requests
for that server and they will be sent to the RS for execution. The RS will complete the open
operation by calling master.reportRegionStateTransition(). The AM will intercept the transition
report, and notify the procedure. The procedure will finish the assignment by publishing to
new state on hbase:meta or it will retry the assignment.

h3. Unassign Detail
 The Unassign starts by placing a "close region" request in the Remote Dispatcher queue, and
the procedure will then go into a "waiting state". The Remote Dispatcher will batch the various
requests for that server and they will be sent to the RS for execution. The RS will complete
the open operation by calling master.reportRegionStateTransition(). The AM will intercept
the transition report, and notify the procedure. The procedure will finish the unassign by
publishing its new state on meta or it will retry the unassign.

h1. New Configs
 * "hbase.procedure.remote.dispatcher.threadpool.size" defaults 128
 * "hbase.procedure.remote.dispatcher.delay.msec" default 150ms
 * "hbase.procedure.remote.dispatcher.max.queue.size" with default 32
 * "hbase.regionserver.rpc.startup.waittime" with default 60 seconds.
h1. TODO
As of this writing.

Put up a model diagram.

 * Handle region migration
 * Handle meta assignment first
 * Handle sys table assignment first (e.g. acl, namespace)
 * Handle table priorities
 * Do we report same AM metrics as we used too? We do it all in here now.


  was:
h1. AMv2
Puts AssignmentManager up on top of the ProcedureV2 state machine with persistence engine.
Each assignment atom is now a Procedure implementation; e.g. an AssignProcedure and an UnassignProcedure.
Molecules of aggregated Procedures are used to do more involved assignment steps: e.g. the
move region procedure is made of an Unassign followed by an Assign subprocedure.

AMv2 is 1500 lines. Old AM was near 4000. Functionality has been moved out to Procedures.
In-memory states of regions and servers has been cleaned up stored in new RegionStates implementation.
RegionStateStore takes care of publishing final region state out to the hbase:meta table.

New RemoteProcedureDispatcher/RSProcedureDispatcher runs the Procedure-based assignments ‘remotely’.
Knows about ‘servers’. Does aggregation of assignments by time on a time/count basis so
can send procedures in batches rather than one per RPC. Procedure status comes back on the
back of the RegionServer heartbeat reporting online regions. The response is passed to the
AMv2 to ‘process’. It will check against the in-memory state. If there is a mismatch,
it fences out the RegionServer on the assumption that something went wrong on the RS side.Timeouts
trigger retries. The Procedure machine ensures only one operation at a time on any one region/table
using locking and smarts about what is serial and what can be run concurrently.

New accounting of RegionServer version will be used running rolling restarts.

‘States’ -- OPENING, CLOSING, etc. -- are now in-memory in-the-master only serialized
out to the ProcedureV2 WAL. They are no longer persisted to ZooKeeper.

h2. Assign Detail
The Assign starts by pushing the "assign" operation to the AssignmentManager and then will
go into a “waiting" state. The AM will batch the "assign" requests and ask the Balancer
where to put the region (the various policies will be respected: retain, round-robin, random).
Once the AM and the balancer have found a place for the region, the procedure will be resumed
and an "open region" request will be placed in the Remote Dispatcher queue, and the procedure
once again will go into a "waiting state".  The Remote Dispatcher will batch the various requests
for that server and they will be sent to the RS for execution. The RS will complete the open
operation by calling master.reportRegionStateTransition(). The AM will intercept the transition
report, and notify the procedure. The procedure will finish the assignment by publishing to
new state on hbase:meta or it will retry the assignment.

h3. Unassign Detail
 The Unassign starts by placing a "close region" request in the Remote Dispatcher queue, and
the procedure will then go into a "waiting state". The Remote Dispatcher will batch the various
requests for that server and they will be sent to the RS for execution. The RS will complete
the open operation by calling master.reportRegionStateTransition(). The AM will intercept
the transition report, and notify the procedure. The procedure will finish the unassign by
publishing its new state on meta or it will retry the unassign.

h1. New Configs
 * "hbase.procedure.remote.dispatcher.threadpool.size" defaults 128
 * "hbase.procedure.remote.dispatcher.delay.msec" default 150ms
 * "hbase.procedure.remote.dispatcher.max.queue.size" with default 32
 * "hbase.regionserver.rpc.startup.waittime" with default 60 seconds.
h1. TODO
As of this writing.

Put up a model diagram.

 * Handle region migration
 * Handle meta assignment first
 * Handle sys table assignment first (e.g. acl, namespace)
 * Handle table priorities
 * Do we report same AM metrics as we used too? We do it all in here now.



> Procedure v2: Core Assignment Manager
> -------------------------------------
>
>                 Key: HBASE-14614
>                 URL: https://issues.apache.org/jira/browse/HBASE-14614
>             Project: HBase
>          Issue Type: Sub-task
>          Components: proc-v2
>    Affects Versions: 2.0.0
>            Reporter: Stephen Yuan Jiang
>            Assignee: Matteo Bertozzi
>             Fix For: 2.0.0
>
>         Attachments: HBASE-14614.master.001.patch, HBASE-14614.master.002.patch, HBASE-14614.master.003.patch,
HBASE-14614.master.004.patch, HBASE-14614.master.005.patch
>
>
> New AssignmentManager implemented using proc-v2.
>  - AssignProcedure handle assignment operation
>  - UnassignProcedure handle unassign operation
>  - MoveRegionProcedure handle move/balance operation
> Concurrent Assign operations are batched together and sent to the balancer
> Concurrent Assign and Unassign operation ready to be sent to the RS are batched together
> This patch is an intermediate state where we add the new AM as AssignmentManager2() to
the master, to be reached by tests. but the new AM will not be integrated with the rest of
the system. Only new am unit-tests will exercise the new assigment manager. The integration
with the master code is part of HBASE-14616



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message