Mailing-List: contact dev-help@hbase.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@hbase.apache.org
Date: Sat, 2 Feb 2013 01:11:11 +0000 (UTC)
From: "Nick Dimiduk (JIRA)" <jira@apache.org>
To: dev@hbase.apache.org
Message-ID: <JIRA.12630488.1359767426239.233433.1359767471723@arcas>
In-Reply-To: <JIRA.12630488.1359767426239@arcas>
References: <JIRA.12630488.1359767426239@arcas>
Subject: [jira] [Created] (HBASE-7743) KeyValueSortReducer and
 PutSortReducers buffer entire value-groups in memory
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit

Nick Dimiduk created HBASE-7743:
-----------------------------------

             Summary: KeyValueSortReducer and PutSortReducers buffer entire value-groups in memory
                 Key: HBASE-7743
                 URL: https://issues.apache.org/jira/browse/HBASE-7743
             Project: HBase
          Issue Type: Improvement
          Components: mapreduce
            Reporter: Nick Dimiduk


The mapreduce package provides two Reducer implementations, KeyValueSortReducer and PutSortReducer, which are used by Import, ImportTsv, and WALPlayer in conjunction with the HFileOutputFormat. Both of these implementations make use of a TreeSet to sort values matching a key. This reducer will OOM when rows are large.

A better solution would be to implement secondary sort of the values. That way hadoop sorts the records, spilling to disk when necessary.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira