hbase-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From te...@apache.org
Subject hbase git commit: HBASE-15171 Avoid counting duplicate kv and generating lots of small hfiles in PutSortReducer (Yu Li)
Date Wed, 27 Jan 2016 17:49:56 GMT
Repository: hbase
Updated Branches:
  refs/heads/branch-1 6d56ecfcf -> 630ad95c9


HBASE-15171 Avoid counting duplicate kv and generating lots of small hfiles in PutSortReducer
(Yu Li)


Project: http://git-wip-us.apache.org/repos/asf/hbase/repo
Commit: http://git-wip-us.apache.org/repos/asf/hbase/commit/630ad95c
Tree: http://git-wip-us.apache.org/repos/asf/hbase/tree/630ad95c
Diff: http://git-wip-us.apache.org/repos/asf/hbase/diff/630ad95c

Branch: refs/heads/branch-1
Commit: 630ad95c923f642d006274b9b1a14397a6713412
Parents: 6d56ecf
Author: tedyu <yuzhihong@gmail.com>
Authored: Wed Jan 27 09:49:45 2016 -0800
Committer: tedyu <yuzhihong@gmail.com>
Committed: Wed Jan 27 09:49:45 2016 -0800

----------------------------------------------------------------------
 .../java/org/apache/hadoop/hbase/mapreduce/PutSortReducer.java   | 4 +++-
 1 file changed, 3 insertions(+), 1 deletion(-)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/hbase/blob/630ad95c/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/PutSortReducer.java
----------------------------------------------------------------------
diff --git a/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/PutSortReducer.java
b/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/PutSortReducer.java
index 1706aec..5533bad 100644
--- a/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/PutSortReducer.java
+++ b/hbase-server/src/main/java/org/apache/hadoop/hbase/mapreduce/PutSortReducer.java
@@ -67,9 +67,11 @@ public class PutSortReducer extends
           for (Cell cell: cells) {
             KeyValue kv = KeyValueUtil.ensureKeyValueTypeForMR(cell);
             map.add(kv);
-            curSize += kv.heapSize();
           }
         }
+        for(KeyValue kv: map){
+          curSize +=kv.heapSize();
+        }
       }
       context.setStatus("Read " + map.size() + " entries of " + map.getClass()
           + "(" + StringUtils.humanReadableInt(curSize) + ")");


Mime
View raw message