spark-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ted Yu <yuzhih...@gmail.com>
Subject Re: Does calling sqlContext.cacheTable("oldTableName") remove the cached contents of the oldTable
Date Fri, 18 Dec 2015 17:47:46 GMT
This method in CacheManager:
  private[sql] def lookupCachedData(plan: LogicalPlan): Option[CachedData]
= readLock {
    cachedData.find(cd => plan.sameResult(cd.plan))

Ied me to the following in
sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/plans/logical/LogicalPlan.scala
:

  def sameResult(plan: LogicalPlan): Boolean = {

There is detailed comment above this method which should give some idea.

Cheers

On Fri, Dec 18, 2015 at 9:21 AM, Sahil Sareen <sareensah@gmail.com> wrote:

> Thanks Ted!
>
> Yes, The schema might be different or the same.
> What would be the answer for each situation?
>
> On Fri, Dec 18, 2015 at 6:02 PM, Ted Yu <yuzhihong@gmail.com> wrote:
>
>> CacheManager#cacheQuery() is called where:
>>   * Caches the data produced by the logical representation of the given
>> [[Queryable]].
>> ...
>>     val planToCache = query.queryExecution.analyzed
>>     if (lookupCachedData(planToCache).nonEmpty) {
>>
>> Is the schema for dfNew different from that of dfOld ?
>>
>> Cheers
>>
>> On Fri, Dec 18, 2015 at 3:33 AM, Sahil Sareen <sareensah@gmail.com>
>> wrote:
>>
>>> Spark 1.5.2
>>>
>>> dfOld.registerTempTable("oldTableName")
>>> sqlContext.cacheTable("oldTableName")
>>> // ....
>>> // do something
>>> // ....
>>> dfNew.registerTempTable("oldTableName")
>>> sqlContext.cacheTable("oldTableName")
>>>
>>>
>>> Now when I use the "oldTableName" table I do get the latest contents
>>> from dfNew but do the contents of dfOld get removed from the memory?
>>>
>>> Or is the right usage to do this:
>>> dfOld.registerTempTable("oldTableName")
>>> sqlContext.cacheTable("oldTableName")
>>> // ....
>>> // do something
>>> // ....
>>> dfNew.registerTempTable("oldTableName")
>>> sqlContext.unCacheTable("oldTableName") <========== unCache the old
>>> contents first
>>> sqlContext.cacheTable("oldTableName")
>>>
>>> -Sahil
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: user-unsubscribe@spark.apache.org
>>> For additional commands, e-mail: user-help@spark.apache.org
>>>
>>>
>>
>

Mime
View raw message