hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Reid Chan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-19226) Limit the reduce tasks number of incremental load
Date Sun, 12 Nov 2017 01:59:00 GMT

    [ https://issues.apache.org/jira/browse/HBASE-19226?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16248760#comment-16248760
] 

Reid Chan commented on HBASE-19226:
-----------------------------------

Would you mind explaining the following codes?
{code}
for (ImmutableBytesWritable startKey : sorted) {
        if (offset == bucket[bucketIndex]) {
          writer.append(startKey, NullWritable.get());
          bucketIndex++;
          offset = 0;
        }
        offset++;
}
{code}

> Limit the reduce tasks number of incremental load
> -------------------------------------------------
>
>                 Key: HBASE-19226
>                 URL: https://issues.apache.org/jira/browse/HBASE-19226
>             Project: HBase
>          Issue Type: Improvement
>            Reporter: Yun Zhao
>            Assignee: Yun Zhao
>            Priority: Minor
>         Attachments: HBASE-19226.master.001.patch, HBASE-19226.master.002.patch
>
>
> When using MapReduce job to perform an incremental load into a table,the number of reduce
tasks is the current number of regions. If there are too many regions, will lead to network+disk
I/O is too large, affecting the real-time request.
> Need to use a configuration to set a number or ratio?
> Limit running reduce tasks since [https://issues.apache.org/jira/browse/MAPREDUCE-5583],
the old version can only be set reduce number.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

Mime
View raw message