Mailing-List: contact dev-help@hive.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@hive.apache.org
Content-Type: multipart/alternative;
 boundary="===============0605793623481637585=="
MIME-Version: 1.0
Subject: Re: Review Request 57353: Intern Properties objects referenced from
 PartitionDesc to reduce memory pressure.
From: Misha Dmitriev <misha@cloudera.com>
To: Sahil Takiar <takiar.sahil@gmail.com>, Xuefu Zhang <xuefuz@yahoo-inc.com>,
 Alan Gates <gates@hortonworks.com>, Rui Li <rui.li@intel.com>,
 Chaozhong Yang <yangchaozhong.2009@gmail.com>, j.prasanth.j@gmail.com,
 Vihang Karajgaonkar <vihang@cloudera.com>,
 Sergio Pena <sergio.pena@cloudera.com>
Cc: hive <dev@hive.apache.org>, Misha Dmitriev <misha@cloudera.com>
Date: Fri, 21 Apr 2017 20:15:55 -0000
Message-ID: <20170421201555.23980.31091@reviews-vm2.apache.org>
Auto-Submitted: auto-generated
Sender: Misha Dmitriev <noreply@reviews.apache.org>
References: <20170307012208.40991.75510@reviews-vm2.apache.org>
In-Reply-To: <20170307012208.40991.75510@reviews-vm2.apache.org>
Reply-To: Misha Dmitriev <misha@cloudera.com>
archived-at: Fri, 21 Apr 2017 20:16:00 -0000

--===============0605793623481637585==
MIME-Version: 1.0
Content-Type: text/plain; charset="utf-8"
Content-Transfer-Encoding: 7bit


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/57353/#review172706
-----------------------------------------------------------


common/src/java/org/apache/hadoop/hive/common/CopyOnFirstWriteProperties.java
Lines 314 (patched)
<https://reviews.apache.org/r/57353/#comment245763>

    Oh, you are right, I didn't realize that google.common.collect.MapMaker that this thing uses internally always returns a ConcurrentMap, which is equivalent to ConcurrentHashMap in terms of thread safety. So, no need for extra synchronization here. Fixed.


- Misha Dmitriev


On March 7, 2017, 1:22 a.m., Misha Dmitriev wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/57353/
> -----------------------------------------------------------
> 
> (Updated March 7, 2017, 1:22 a.m.)
> 
> 
> Review request for hive, Chaozhong Yang, Alan Gates, Rui Li, Prasanth_J, Sergio Pena, Sahil Takiar, Vihang Karajgaonkar, and Xuefu Zhang.
> 
> 
> Bugs: HIVE-16079
>     https://issues.apache.org/jira/browse/HIVE-16079
> 
> 
> Repository: hive-git
> 
> 
> Description
> -------
> 
> When multiple concurrent Hive queries run, a separate copy of
> org.apache.hadoop.hive.ql.metadata.Partition and
> ql.plan.PartitionDesc is created for each table partition
> per each query instance. So when in my benchmark explained in
> HIVE-16079 we have 2000 partitions and 50 concurrent queries running
> over them, we end up, in the worst case, with 2000*50=100,000 instances
> of Partition and PartitionDesc in memory. These objects themselves
> collectively take just ~2% of memory. However, other data structures
> that each of them reference, take a lot more. In particular, Properties
> objects take more than 20% of memory. When we have 50 concurrent
> read-only queries, there are 50 identical copies of Properties per
> each partition. That's a huge waste of memory.
> 
> This change introduces a new class that extends Properties, called
> CopyOnFirstWriteProperties. It utilizes a unique interned copy of
> Properties whenever possible. However, when one of the methods that
> modify properties is called, a copy is created. When this class is
> used, memory consumption by Properties falls from 20% to 5..6%.
> 
> 
> Diffs
> -----
> 
>   common/src/java/org/apache/hadoop/hive/common/CopyOnFirstWriteProperties.java PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/SerializationUtilities.java 247d5890ea8131404b9543d22876ca4c052578e0 
>   ql/src/java/org/apache/hadoop/hive/ql/plan/PartitionDesc.java d05c1c68fdb7296c0346d73967071da1ebe7bb72 
> 
> 
> Diff: https://reviews.apache.org/r/57353/diff/1/
> 
> 
> Testing
> -------
> 
> 
> Thanks,
> 
> Misha Dmitriev
> 
>


--===============0605793623481637585==--