spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "xinzhang (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (SPARK-23022) Spark Thrift Server always cache resource issues
Date Wed, 10 Jan 2018 02:19:00 GMT

     [ https://issues.apache.org/jira/browse/SPARK-23022?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

xinzhang updated SPARK-23022:
-----------------------------
    Description: 
Hi. I use the Thrift Server for SparkSQL . I queried muiltle queries.I deply the Spark  on
Yarn. 
When I finish my query.Thrift Server always cache the Yarn Resources. Any suggests will be
helpful.

------------------------------------------------------------------------------------
Here is the img .
------------------------------------------------------------------------------------
!https://user-images.githubusercontent.com/8244097/34752652-8d224416-f5ee-11e7-89d3-5868c128378d.png!
!https://user-images.githubusercontent.com/8244097/34752397-215bea08-f5ed-11e7-89f4-13ef9ab78904.png!
!https://user-images.githubusercontent.com/8244097/34752403-2756d224-f5ed-11e7-97d3-9c7d21c48f3a.png!
!https://user-images.githubusercontent.com/8244097/34752409-2ba1d3ce-f5ed-11e7-85d6-9e46ae8a3e2b.png!
------------------------------------------------------------------------------------
Here is the Spark Conf .
------------------------------------------------------------------------------------

{code:java}
spark.dynamicAllocation.minExecutors 1
spark.dynamicAllocation.maxExecutors 500
spark.dynamicAllocation.enabled true
spark.shuffle.service.enabled true
spark.executor.instances 2
spark.executor.memory   6g
#serializer
spark.serializer org.apache.spark.serializer.KryoSerializer
spark.kryoserializer.buffer.max 512m
#spark.kryo.unsafe true
spark.kryo.referenceTracking false
spark.rdd.compress true
spark.memory.offHeap.enabled true
spark.memory.offHeap.size 1g

spark.yarn.archive hdfs://ns/data1/hadooptmp/spark2.2.1/jars
spark.yarn.am.memory 2g
spark.driver.memory 4g
spark.driver.maxResultSize 2g

#SPARK SQL 
spark.sql.shuffle.partitions 500
spark.sql.statistics.fallBackToHdfs true
spark.sql.orc.filterPushdown true
spark.sql.autoBroadcastJoinThreshold 104857600
spark.sql.adaptive.enabled true

spark.history.fs.logDirectory  hdfs://ns/data4/hadooptmp/spark-history
spark.eventLog.enabled  true
spark.eventLog.compress  true
spark.eventLog.dir  hdfs://ns/data4/hadooptmp/spark-history
spark.yarn.historyServer.address  172.31.10.119:18080
spark.io.compression.codec snappy
spark.executor.logs.rolling.enableCompression true
spark.dynamicAllocation.executorIdleTimeout 10s
spark.network.timeout 600s
spark.sql.parquet.writeLegacyFormat true
{code}



  was:

Hi. I use the Thrift Server for SparkSQL . I query muiltle query.I deply the Spark  on Yarn.

When I finish my query.Thrift Server always cache the Yarn Resources. Any suggests will be
helpful.

------------------------------------------------------------------------------------
Here is the img .
------------------------------------------------------------------------------------
!https://user-images.githubusercontent.com/8244097/34752652-8d224416-f5ee-11e7-89d3-5868c128378d.png!
!https://user-images.githubusercontent.com/8244097/34752397-215bea08-f5ed-11e7-89f4-13ef9ab78904.png!
!https://user-images.githubusercontent.com/8244097/34752403-2756d224-f5ed-11e7-97d3-9c7d21c48f3a.png!
!https://user-images.githubusercontent.com/8244097/34752409-2ba1d3ce-f5ed-11e7-85d6-9e46ae8a3e2b.png!
------------------------------------------------------------------------------------
Here is the Spark Conf .
------------------------------------------------------------------------------------

{code:java}
spark.dynamicAllocation.minExecutors 1
spark.dynamicAllocation.maxExecutors 500
spark.dynamicAllocation.enabled true
spark.shuffle.service.enabled true
spark.executor.instances 2
spark.executor.memory   6g
#serializer
spark.serializer org.apache.spark.serializer.KryoSerializer
spark.kryoserializer.buffer.max 512m
#spark.kryo.unsafe true
spark.kryo.referenceTracking false
spark.rdd.compress true
spark.memory.offHeap.enabled true
spark.memory.offHeap.size 1g

spark.yarn.archive hdfs://ns/data1/hadooptmp/spark2.2.1/jars
spark.yarn.am.memory 2g
spark.driver.memory 4g
spark.driver.maxResultSize 2g

#SPARK SQL 
spark.sql.shuffle.partitions 500
spark.sql.statistics.fallBackToHdfs true
spark.sql.orc.filterPushdown true
spark.sql.autoBroadcastJoinThreshold 104857600
spark.sql.adaptive.enabled true

spark.history.fs.logDirectory  hdfs://ns/data4/hadooptmp/spark-history
spark.eventLog.enabled  true
spark.eventLog.compress  true
spark.eventLog.dir  hdfs://ns/data4/hadooptmp/spark-history
spark.yarn.historyServer.address  172.31.10.119:18080
spark.io.compression.codec snappy
spark.executor.logs.rolling.enableCompression true
spark.dynamicAllocation.executorIdleTimeout 10s
spark.network.timeout 600s
spark.sql.parquet.writeLegacyFormat true
{code}




> Spark Thrift Server always cache resource issues
> ------------------------------------------------
>
>                 Key: SPARK-23022
>                 URL: https://issues.apache.org/jira/browse/SPARK-23022
>             Project: Spark
>          Issue Type: Bug
>          Components: Deploy
>    Affects Versions: 2.2.1
>         Environment: CentOS6.x 
> Spark2.x
> JDK1.8
>            Reporter: xinzhang
>
> Hi. I use the Thrift Server for SparkSQL . I queried muiltle queries.I deply the Spark
 on Yarn. 
> When I finish my query.Thrift Server always cache the Yarn Resources. Any suggests will
be helpful.
> ------------------------------------------------------------------------------------
> Here is the img .
> ------------------------------------------------------------------------------------
> !https://user-images.githubusercontent.com/8244097/34752652-8d224416-f5ee-11e7-89d3-5868c128378d.png!
> !https://user-images.githubusercontent.com/8244097/34752397-215bea08-f5ed-11e7-89f4-13ef9ab78904.png!
> !https://user-images.githubusercontent.com/8244097/34752403-2756d224-f5ed-11e7-97d3-9c7d21c48f3a.png!
> !https://user-images.githubusercontent.com/8244097/34752409-2ba1d3ce-f5ed-11e7-85d6-9e46ae8a3e2b.png!
> ------------------------------------------------------------------------------------
> Here is the Spark Conf .
> ------------------------------------------------------------------------------------
> {code:java}
> spark.dynamicAllocation.minExecutors 1
> spark.dynamicAllocation.maxExecutors 500
> spark.dynamicAllocation.enabled true
> spark.shuffle.service.enabled true
> spark.executor.instances 2
> spark.executor.memory   6g
> #serializer
> spark.serializer org.apache.spark.serializer.KryoSerializer
> spark.kryoserializer.buffer.max 512m
> #spark.kryo.unsafe true
> spark.kryo.referenceTracking false
> spark.rdd.compress true
> spark.memory.offHeap.enabled true
> spark.memory.offHeap.size 1g
> spark.yarn.archive hdfs://ns/data1/hadooptmp/spark2.2.1/jars
> spark.yarn.am.memory 2g
> spark.driver.memory 4g
> spark.driver.maxResultSize 2g
> #SPARK SQL 
> spark.sql.shuffle.partitions 500
> spark.sql.statistics.fallBackToHdfs true
> spark.sql.orc.filterPushdown true
> spark.sql.autoBroadcastJoinThreshold 104857600
> spark.sql.adaptive.enabled true
> spark.history.fs.logDirectory  hdfs://ns/data4/hadooptmp/spark-history
> spark.eventLog.enabled  true
> spark.eventLog.compress  true
> spark.eventLog.dir  hdfs://ns/data4/hadooptmp/spark-history
> spark.yarn.historyServer.address  172.31.10.119:18080
> spark.io.compression.codec snappy
> spark.executor.logs.rolling.enableCompression true
> spark.dynamicAllocation.executorIdleTimeout 10s
> spark.network.timeout 600s
> spark.sql.parquet.writeLegacyFormat true
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message