hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jimmy Xiang" <jxi...@cloudera.com>
Subject Re: Review Request 28064: HIVE-8844 Choose a persisent policy for RDD caching [Spark Branch]
Date Sat, 15 Nov 2014 03:17:17 GMT


> On Nov. 15, 2014, 2:34 a.m., Szehon Ho wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/spark/ShuffleTran.java, line 39
> > <https://reviews.apache.org/r/28064/diff/1/?file=764642#file764642line39>
> >
> >     OK, does spark handle that if we pass NONE in by doing no-op?   If that's the
case, then maybe cleaner for our code in that case.  I'm a bit confused what NONE means. 
> >     
> >     If we dont want to call NONE due to side-effects, can we just change the HadoopRDD
call to:
> >     
> >     storageHandler.equals(StorageHandler.NONE) ? hadoopRdd : ...
> >     
> >     Then the logic is centralized to there.
> 
> Jimmy Xiang wrote:
>     Sure. Will fix it as suggested. Thanks.
> 
> Chao Sun wrote:
>     persist() also register the RDD for GC clean up, but there seem to have no extra
cost besides that. Either way is fine to me.

Checked the spark code, it should do nothing for none, but it indeed does something. Probably
a Spark issue for now.


- Jimmy


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/28064/#review61616
-----------------------------------------------------------


On Nov. 15, 2014, 3:14 a.m., Jimmy Xiang wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/28064/
> -----------------------------------------------------------
> 
> (Updated Nov. 15, 2014, 3:14 a.m.)
> 
> 
> Review request for hive and Xuefu Zhang.
> 
> 
> Bugs: HIVE-8844
>     https://issues.apache.org/jira/browse/HIVE-8844
> 
> 
> Repository: hive-git
> 
> 
> Description
> -------
> 
> Changed spark cache policy to be configurable with default memory+disk.
> 
> 
> Diffs
> -----
> 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/MapInput.java 79baea7 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/ShuffleTran.java 8565ba0 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkPlanGenerator.java 11f4236 
> 
> Diff: https://reviews.apache.org/r/28064/diff/
> 
> 
> Testing
> -------
> 
> 
> Thanks,
> 
> Jimmy Xiang
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message