hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "stack (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HBASE-16169) Make RegionSizeCalculator scalable
Date Tue, 15 Nov 2016 05:12:58 GMT

     [ https://issues.apache.org/jira/browse/HBASE-16169?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

stack updated HBASE-16169:
--------------------------
    Release Note: 
Added couple of API's to Admin.java:

Returns region load map of all regions hosted on a region server
Map<byte[], RegionLoad> getRegionLoad(ServerName sn) throws IOException;

Returns region load map of all regions of a table hosted on a region server
Map<byte[], RegionLoad> getRegionLoad(ServerName sn, TableName tableName) throws IOException

Added an API to region server:

public GetRegionLoadResponse getRegionLoad(RpcController controller,
    GetRegionLoadRequest request) throws ServiceException;

Primary intention is to use this API for RegionSizeCalculator and not rely on Master for ClusterStatus.
On large clusters, ClusterStatus() can take a long time. IfMaster is down/busy, then some
of the jobs timeout/fail. Other possible uses:
1. If there is a lighter version of GetClusterStatus API (i.e without the ServerLoad for each
RS), then custom maintenance tools can be better. In current world ClusterStatus is heavy.
With the new APIs, each API's payload is smaller and distributed. So custom tools can call
getRegionLoad() when needed, it will be more accurate. This helps with large clusters. For
tools that don't need RegionLoad, the lighter version of API is fine enough.
2. Another use case is a tool like RSTop - since we can see selective metrics at RegionLevel
(possibly even deltas between each RPC to the server).

  was:
Added couple of API's to Admin.java:

Returns region load map of all regions hosted on a region server
Map<byte[], RegionLoad> getRegionLoad(ServerName sn) throws IOException;

Returns region load map of all regions of a table hosted on a region server
Map<byte[], RegionLoad> getRegionLoad(ServerName sn, TableName tableName) throws IOException

Added an API to region server:

public GetRegionLoadResponse getRegionLoad(RpcController controller,
    GetRegionLoadRequest request) throws ServiceException;



> Make RegionSizeCalculator scalable
> ----------------------------------
>
>                 Key: HBASE-16169
>                 URL: https://issues.apache.org/jira/browse/HBASE-16169
>             Project: HBase
>          Issue Type: Sub-task
>          Components: mapreduce, scaling
>            Reporter: Thiruvel Thirumoolan
>            Assignee: Thiruvel Thirumoolan
>             Fix For: 2.0.0, 1.4.0
>
>         Attachments: HBASE-16169.master.000.patch, HBASE-16169.master.001.patch, HBASE-16169.master.002.patch,
HBASE-16169.master.003.patch, HBASE-16169.master.004.patch, HBASE-16169.master.005.patch,
HBASE-16169.master.006.patch
>
>
> RegionSizeCalculator is needed for better split generation of MR jobs. This requires
RegionLoad which can be obtained via ClusterStatus, i.e. accessing Master. We don't want master
to be in this path.
> The proposal is to add an API to the RegionServer that gets RegionLoad of all regions
hosted on it or those of a table if specified. RegionSizeCalculator can use the latter.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message