hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Xuefu Zhang" <xzh...@cloudera.com>
Subject Re: Review Request 34455: HIVE-10550 Dynamic RDD caching optimization for HoS.[Spark Branch]
Date Thu, 28 May 2015 04:16:06 GMT


> On May 27, 2015, 10:13 p.m., Xuefu Zhang wrote:
> > common/src/java/org/apache/hadoop/hive/conf/HiveConf.java, line 2062
> > <https://reviews.apache.org/r/34455/diff/3/?file=972428#file972428line2062>
> >
> >     Sorry for pointing this out late. I'm not certain if it's a good idea to expose
these two configurations. Also this introduces a change of  behavior. For now, can we get
rid of them and change the persistency level back to MEM+DISK?
> >     
> >     We can come back to revisit this later on. At this moment, I don't feel confident
to make the call.
> 
> chengxiang li wrote:
>     persistent to MEM + DISK may hurt the performance in certain cases, i think at least
we should have a switch to open/close this optimization,
> 
> Xuefu Zhang wrote:
>     Agreed. However, before we find out more about in what cases this helps or hurts,
I think it's better we keep the existing behavior. This doesn't prevent us from adding a flag
later on.
> 
> chengxiang li wrote:
>     Ok, i would remove these configurations from patch in temp, we can discuss later
when we got more knowledge about it.

Please feel free to create a followup JIRA to do more research. We can try different data
sizes and persistancy levels to see the result. At that time, we can decide if it makes sense
to introduce configurations. Thanks.


- Xuefu


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/34455/#review85451
-----------------------------------------------------------


On May 28, 2015, 3:30 a.m., chengxiang li wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/34455/
> -----------------------------------------------------------
> 
> (Updated May 28, 2015, 3:30 a.m.)
> 
> 
> Review request for hive, Chao Sun, Jimmy Xiang, and Xuefu Zhang.
> 
> 
> Bugs: HIVE-10550
>     https://issues.apache.org/jira/browse/HIVE-10550
> 
> 
> Repository: hive-git
> 
> 
> Description
> -------
> 
> see jira description
> 
> 
> Diffs
> -----
> 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/CacheTran.java PRE-CREATION 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/MapTran.java 2170243 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/ReduceTran.java e60dfac 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkPlan.java ee5c78a 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkPlanGenerator.java 3f240f5 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkUtilities.java e6c845c 
> 
> Diff: https://reviews.apache.org/r/34455/diff/
> 
> 
> Testing
> -------
> 
> 
> Thanks,
> 
> chengxiang li
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message