hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jerry He (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-7743) Replace *SortReducers with Hadoop Secondary Sort
Date Sun, 06 Mar 2016 01:09:40 GMT

    [ https://issues.apache.org/jira/browse/HBASE-7743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15181935#comment-15181935
] 

Jerry He commented on HBASE-7743:
---------------------------------

Linked to HBASE-13897 as well.

> Replace *SortReducers with Hadoop Secondary Sort
> ------------------------------------------------
>
>                 Key: HBASE-7743
>                 URL: https://issues.apache.org/jira/browse/HBASE-7743
>             Project: HBase
>          Issue Type: Sub-task
>          Components: mapreduce, Performance
>            Reporter: Nick Dimiduk
>             Fix For: 2.0.0
>
>
> The mapreduce package provides two Reducer implementations, KeyValueSortReducer and PutSortReducer,
which are used by Import, ImportTsv, and WALPlayer in conjunction with the HFileOutputFormat.
Both of these implementations make use of a TreeSet to sort values matching a key. This reducer
will OOM when rows are large.
> A better solution would be to implement secondary sort of the values. That way hadoop
sorts the records, spilling to disk when necessary.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message