hbase-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Heng Chen (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HBASE-7743) Replace *SortReducers with Hadoop Secondary Sort
Date Fri, 27 Nov 2015 09:47:11 GMT

    [ https://issues.apache.org/jira/browse/HBASE-7743?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15029690#comment-15029690
] 

Heng Chen commented on HBASE-7743:
----------------------------------

Any progress?  
IMO we can remove TreeSet at least to save memory used.  
As for very large rows,  we can not avoid OOM because we should group all cells per row. 

And it is not encouraged when we design Table.  

> Replace *SortReducers with Hadoop Secondary Sort
> ------------------------------------------------
>
>                 Key: HBASE-7743
>                 URL: https://issues.apache.org/jira/browse/HBASE-7743
>             Project: HBase
>          Issue Type: Sub-task
>          Components: mapreduce, Performance
>            Reporter: Nick Dimiduk
>             Fix For: 2.0.0
>
>
> The mapreduce package provides two Reducer implementations, KeyValueSortReducer and PutSortReducer,
which are used by Import, ImportTsv, and WALPlayer in conjunction with the HFileOutputFormat.
Both of these implementations make use of a TreeSet to sort values matching a key. This reducer
will OOM when rows are large.
> A better solution would be to implement secondary sort of the values. That way hadoop
sorts the records, spilling to disk when necessary.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message