hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Yu Li (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-10932) Improve RowCounter to allow mapper number set/control
Date Wed, 09 Apr 2014 06:07:20 GMT

    [ https://issues.apache.org/jira/browse/HBASE-10932?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13963852#comment-13963852

Yu Li commented on HBASE-10932:

Hi [~jdcryans] and [~ndimiduk],

Are you suggesting using the RowCounter in the old o.a.hadoop.hbase.mapred package? It seems
to me this class is deprecated and it's using old mapred APIs. What's more, while issue the
rowcounter command using hbase/hadoop script, it will launch RowCounter in o.a.hadoop.hbase.mapreduce
package by default.

For the getSplits method in the new TableInputFormatBase, from the method comments, it's designed
to make splits number matching number of regions, so I don't think this is a bug but something
to improve for the *in-use* RowCounter:
   * Calculates the splits that will serve as input for the map tasks. The
   * number of splits matches the number of regions in a table.
   * @param context  The current job context.
   * @return The list of input splits.
   * @throws IOException When creating the list of splits fails.
   * @see org.apache.hadoop.mapreduce.InputFormat#getSplits(
   *   org.apache.hadoop.mapreduce.JobContext)
And this is the exact reason we introduce a new RowCounterInputFormat class to override the
getSplits method rather than modifying the existing one.

As to the new parameter, yes user could pass -Dmapred.map.tasks, but I think it better to
add an explicit parameter so user could see how it works from usage message. IMHO, assuming
hbase users have background of MR might not be a good idea.

> Improve RowCounter to allow mapper number set/control
> -----------------------------------------------------
>                 Key: HBASE-10932
>                 URL: https://issues.apache.org/jira/browse/HBASE-10932
>             Project: HBase
>          Issue Type: Improvement
>          Components: mapreduce
>            Reporter: Yu Li
>            Assignee: Yu Li
>            Priority: Minor
>         Attachments: HBASE-10932_v1.patch
> The typical use case of RowCounter is to do some kind of data integrity checking, like
after exporting some data from RDBMS to HBase, or from one HBase cluster to another, making
sure the row(record) number matches. Such check commonly won't require much on response time.
> Meanwhile, based on current impl, RowCounter will launch one mapper per region, and each
mapper will send one scan request. Assuming the table is kind of big like having tens of regions,
and the cpu core number of the whole MR cluster is also enough, the parallel scan requests
sent by mapper would be a real burden for the HBase cluster.
> So in this JIRA, we're proposing to make rowcounter support an additional option "--maps"
to specify mapper number, and make each mapper able to scan more than one region of the target

This message was sent by Atlassian JIRA

View raw message