Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id CDDE2200D04 for ; Mon, 11 Sep 2017 09:37:06 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id CC55C1609C4; Mon, 11 Sep 2017 07:37:06 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id 1EEEF1609C3 for ; Mon, 11 Sep 2017 09:37:05 +0200 (CEST) Received: (qmail 83917 invoked by uid 500); 11 Sep 2017 07:37:04 -0000 Mailing-List: contact commits-help@hive.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: hive-dev@hive.apache.org Delivered-To: mailing list commits@hive.apache.org Received: (qmail 83906 invoked by uid 99); 11 Sep 2017 07:37:04 -0000 Received: from git1-us-west.apache.org (HELO git1-us-west.apache.org) (140.211.11.23) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 11 Sep 2017 07:37:04 +0000 Received: by git1-us-west.apache.org (ASF Mail Server at git1-us-west.apache.org, from userid 33) id 51648F32A9; Mon, 11 Sep 2017 07:37:03 +0000 (UTC) Content-Type: text/plain; charset="us-ascii" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit From: kgyrtkirk@apache.org To: commits@hive.apache.org Message-Id: X-Mailer: ASF-Git Admin Mailer Subject: hive git commit: HIVE-17344: LocalCache element memory usage is not calculated properly. (Janos Gub via Zoltan Haindrich, reviewed by Sergey Shelukhin) Date: Mon, 11 Sep 2017 07:37:03 +0000 (UTC) archived-at: Mon, 11 Sep 2017 07:37:07 -0000 Repository: hive Updated Branches: refs/heads/master 8deb77940 -> 076bd7716 HIVE-17344: LocalCache element memory usage is not calculated properly. (Janos Gub via Zoltan Haindrich, reviewed by Sergey Shelukhin) Signed-off-by: Zoltan Haindrich Project: http://git-wip-us.apache.org/repos/asf/hive/repo Commit: http://git-wip-us.apache.org/repos/asf/hive/commit/076bd771 Tree: http://git-wip-us.apache.org/repos/asf/hive/tree/076bd771 Diff: http://git-wip-us.apache.org/repos/asf/hive/diff/076bd771 Branch: refs/heads/master Commit: 076bd7716e705132521b267c748ed781b6ec12cb Parents: 8deb779 Author: Janos Gub Authored: Mon Sep 11 09:30:49 2017 +0200 Committer: Zoltan Haindrich Committed: Mon Sep 11 09:32:52 2017 +0200 ---------------------------------------------------------------------- ql/src/java/org/apache/hadoop/hive/ql/io/orc/LocalCache.java | 8 ++++++-- 1 file changed, 6 insertions(+), 2 deletions(-) ---------------------------------------------------------------------- http://git-wip-us.apache.org/repos/asf/hive/blob/076bd771/ql/src/java/org/apache/hadoop/hive/ql/io/orc/LocalCache.java ---------------------------------------------------------------------- diff --git a/ql/src/java/org/apache/hadoop/hive/ql/io/orc/LocalCache.java b/ql/src/java/org/apache/hadoop/hive/ql/io/orc/LocalCache.java index b375aea..e28eb34 100644 --- a/ql/src/java/org/apache/hadoop/hive/ql/io/orc/LocalCache.java +++ b/ql/src/java/org/apache/hadoop/hive/ql/io/orc/LocalCache.java @@ -48,7 +48,7 @@ class LocalCache implements OrcInputFormat.FooterCache { public long fileLength, fileModTime; public int getMemoryUsage() { - return bb.remaining() + 100; // 100 is for 2 longs, BB and java overheads (semi-arbitrary). + return bb.capacity() + 100; // 100 is for 2 longs, BB and java overheads (semi-arbitrary). } } @@ -78,8 +78,12 @@ class LocalCache implements OrcInputFormat.FooterCache { } public void put(Path path, OrcTail tail) { + ByteBuffer bb = tail.getSerializedTail(); + if (bb.capacity() != bb.remaining()) { + throw new RuntimeException("Bytebuffer allocated for path: " + path + " has remaining: " + bb.remaining() + " != capacity: " + bb.capacity()); + } cache.put(path, new TailAndFileData(tail.getFileTail().getFileLength(), - tail.getFileModificationTime(), tail.getSerializedTail().duplicate())); + tail.getFileModificationTime(), bb.duplicate())); } @Override