Mailing-List: contact issues-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Date: Wed, 16 Oct 2013 18:51:46 +0000 (UTC)
From: "Nick Dimiduk (JIRA)" <jira@apache.org>
To: issues@hbase.apache.org
Message-ID: <JIRA.12630488.1359767426239.75210.1381949506978@arcas>
In-Reply-To: <JIRA.12630488.1359767426239@arcas>
References: <JIRA.12630488.1359767426239@arcas>
Subject: [jira] [Commented] (HBASE-7743) Replace *SortReducers with Hadoop
 Secondary Sort
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/HBASE-7743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13797120#comment-13797120 ] 

Nick Dimiduk commented on HBASE-7743:
-------------------------------------

bq. Having reducers group all cells per row will not handle very large rows very well.

That's precisely the reason I stumbled into this in the first place :)

> Replace *SortReducers with Hadoop Secondary Sort
> ------------------------------------------------
>
>                 Key: HBASE-7743
>                 URL: https://issues.apache.org/jira/browse/HBASE-7743
>             Project: HBase
>          Issue Type: Sub-task
>          Components: mapreduce, Performance
>            Reporter: Nick Dimiduk
>
> The mapreduce package provides two Reducer implementations, KeyValueSortReducer and PutSortReducer, which are used by Import, ImportTsv, and WALPlayer in conjunction with the HFileOutputFormat. Both of these implementations make use of a TreeSet to sort values matching a key. This reducer will OOM when rows are large.
> A better solution would be to implement secondary sort of the values. That way hadoop sorts the records, spilling to disk when necessary.


--
This message was sent by Atlassian JIRA
(v6.1#6144)