hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jimmy Xiang" <jxi...@cloudera.com>
Subject Re: Review Request 29063: HIVE-8843 Release RDD cache when Hive query is done [Spark Branch]
Date Wed, 17 Dec 2014 03:54:21 GMT


> On Dec. 17, 2014, 3:06 a.m., Xuefu Zhang wrote:
> >

Fixed and attached v3 to the jira. Could not update the diff on this RB because RB has problem
to find the file in the repo suddenly.


> On Dec. 17, 2014, 3:06 a.m., Xuefu Zhang wrote:
> > ql/src/java/org/apache/hadoop/hive/ql/exec/spark/MapInput.java, line 65
> > <https://reviews.apache.org/r/29063/diff/2/?file=793364#file793364line65>
> >
> >     To be safe, let's do this instead:
> >     result = hadoopRDD.mapToPair(new CopyFunction());
> >     sparkPlan.addCachedRDDId(result.id());
> >     result = result.persist(StorageLevel.MEMORY_AND_DISK());
> >     
> >     Maybe rdd.persist() returns the same reference, but that's the implementation.
This is just to be on the safe side.

Sure.


- Jimmy


-----------------------------------------------------------
This is an automatically generated e-mail. To reply, visit:
https://reviews.apache.org/r/29063/#review65280
-----------------------------------------------------------


On Dec. 16, 2014, 11:29 p.m., Jimmy Xiang wrote:
> 
> -----------------------------------------------------------
> This is an automatically generated e-mail. To reply, visit:
> https://reviews.apache.org/r/29063/
> -----------------------------------------------------------
> 
> (Updated Dec. 16, 2014, 11:29 p.m.)
> 
> 
> Review request for hive, chengxiang li and Xuefu Zhang.
> 
> 
> Bugs: HIVE-8843
>     https://issues.apache.org/jira/browse/HIVE-8843
> 
> 
> Repository: hive-git
> 
> 
> Description
> -------
> 
> Release cached RDD after a job is done, if any.
> 
> 
> Diffs
> -----
> 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/LocalHiveSparkClient.java 3102ab2

>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/MapInput.java f748ee0 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/RemoteHiveSparkClient.java e1946d5

>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/ShuffleTran.java 94c6c98 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkPlan.java 53dd4a0 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/SparkPlanGenerator.java 438efab 
>   ql/src/java/org/apache/hadoop/hive/ql/exec/spark/status/impl/LocalSparkJobStatus.java
b8cc6e6 
>   spark-client/src/main/java/org/apache/hive/spark/client/JobContext.java 8565bd8 
>   spark-client/src/main/java/org/apache/hive/spark/client/JobContextImpl.java 137539b

>   spark-client/src/main/java/org/apache/hive/spark/client/MonitorCallback.java 3693f8f

>   spark-client/src/main/java/org/apache/hive/spark/client/RemoteDriver.java 8c07ee1 
>   spark-client/src/test/java/org/apache/hive/spark/client/TestSparkClient.java 7f5ecd0

> 
> Diff: https://reviews.apache.org/r/29063/diff/
> 
> 
> Testing
> -------
> 
> 
> Thanks,
> 
> Jimmy Xiang
> 
>


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message