parquet-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From b...@apache.org
Subject incubator-parquet-mr git commit: PARQUET-164: Add warning when scaling row group sizes.
Date Tue, 10 Feb 2015 07:07:41 GMT
Repository: incubator-parquet-mr
Updated Branches:
  refs/heads/master 807915b4c -> f48bca051


PARQUET-164: Add warning when scaling row group sizes.

Author: Ryan Blue <blue@apache.org>

Closes #119 from rdblue/PARQUET-164-add-memory-manager-warning and squashes the following
commits:

241144f [Ryan Blue] PARQUET-164: Add warning when scaling row group sizes.


Project: http://git-wip-us.apache.org/repos/asf/incubator-parquet-mr/repo
Commit: http://git-wip-us.apache.org/repos/asf/incubator-parquet-mr/commit/f48bca05
Tree: http://git-wip-us.apache.org/repos/asf/incubator-parquet-mr/tree/f48bca05
Diff: http://git-wip-us.apache.org/repos/asf/incubator-parquet-mr/diff/f48bca05

Branch: refs/heads/master
Commit: f48bca0510703b0673709b10a806a9d54894a999
Parents: 807915b
Author: Ryan Blue <blue@apache.org>
Authored: Mon Feb 9 23:07:35 2015 -0800
Committer: Ryan Blue <blue@apache.org>
Committed: Mon Feb 9 23:07:35 2015 -0800

----------------------------------------------------------------------
 parquet-hadoop/src/main/java/parquet/hadoop/MemoryManager.java | 4 ++++
 1 file changed, 4 insertions(+)
----------------------------------------------------------------------


http://git-wip-us.apache.org/repos/asf/incubator-parquet-mr/blob/f48bca05/parquet-hadoop/src/main/java/parquet/hadoop/MemoryManager.java
----------------------------------------------------------------------
diff --git a/parquet-hadoop/src/main/java/parquet/hadoop/MemoryManager.java b/parquet-hadoop/src/main/java/parquet/hadoop/MemoryManager.java
index fd399e0..7bb0665 100644
--- a/parquet-hadoop/src/main/java/parquet/hadoop/MemoryManager.java
+++ b/parquet-hadoop/src/main/java/parquet/hadoop/MemoryManager.java
@@ -108,6 +108,10 @@ public class MemoryManager {
       scale = 1.0;
     } else {
       scale = (double) totalMemoryPool / totalAllocations;
+      LOG.warn(String.format(
+          "Total allocation exceeds %.2f%% (%,d bytes) of heap memory\n" +
+          "Scaling row group sizes to %.2f%% for %d writers",
+          100*memoryPoolRatio, totalMemoryPool, 100*scale, writerList.size()));
     }
 
     int maxColCount = 0;


Mime
View raw message