spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Yunling Cai (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (SPARK-25091) Spark Thrift Server: UNCACHE TABLE and CLEAR CACHE does not clean up executor memory
Date Sat, 11 Aug 2018 19:14:00 GMT

     [ https://issues.apache.org/jira/browse/SPARK-25091?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Yunling Cai updated SPARK-25091:
--------------------------------
    Description: 
UNCACHE TABLE and CLEAR CACHE does not clean up executor memory.

Through Spark UI, although in Storage, we see the cached table removed. In Executor, the executors
continue to hold the RDD and the memory is not cleared. This results in huge waste in executor
memory usage. As we call CACHE TABLE, we run into issues where the cached tables are spilled
to disk instead of reclaiming the memory storage. 

Steps to reproduce:

CACHE TABLE test.test_cache;

UNCACHE TABLE test.test_cache;

== Storage shows table is not cached; Executor shows the executor storage memory does not
change == 

CACHE TABLE test.test_cache;

CLEAR CACHE;

== Storage shows table is not cached; Executor shows the executor storage memory does not
change == 

  was:
UNCACHE TABLE and CLEAR CACHE does not clean up executor memory.

Through Spark UI, although in Storage, we see the cached table removed. In Executor, the executors
continue to hold the RDD and the memory is not cleared. This results in huge waste in executor
memory usage. As we call CACHE TABLE, we run into issues where the cached tables are spilled
to disk instead of reclaiming the memory storage. 


> Spark Thrift Server: UNCACHE TABLE and CLEAR CACHE does not clean up executor memory
> ------------------------------------------------------------------------------------
>
>                 Key: SPARK-25091
>                 URL: https://issues.apache.org/jira/browse/SPARK-25091
>             Project: Spark
>          Issue Type: Bug
>          Components: Spark Core
>    Affects Versions: 2.3.1
>            Reporter: Yunling Cai
>            Priority: Major
>
> UNCACHE TABLE and CLEAR CACHE does not clean up executor memory.
> Through Spark UI, although in Storage, we see the cached table removed. In Executor,
the executors continue to hold the RDD and the memory is not cleared. This results in huge waste
in executor memory usage. As we call CACHE TABLE, we run into issues where the cached tables
are spilled to disk instead of reclaiming the memory storage. 
> Steps to reproduce:
> CACHE TABLE test.test_cache;
> UNCACHE TABLE test.test_cache;
> == Storage shows table is not cached; Executor shows the executor storage memory does
not change == 
> CACHE TABLE test.test_cache;
> CLEAR CACHE;
> == Storage shows table is not cached; Executor shows the executor storage memory does
not change == 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message