hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From ?? ? <qiuff...@hotmail.com>
Subject hive on spark job not start enough executors
Date Fri, 09 Sep 2016 08:30:06 GMT
Hi there,


I encountered a problem that makes hive on spark with a very low performance.

I'm using spark 1.6.2 and hive 2.1.0, I specified


    spark.shuffle.service.enabled    true
    spark.dynamicAllocation.enabled  true

in my spark-default.conf file (the file is in both spark and hive conf folder) to make spark
job to get executors dynamically.
The configuration works correctly when I run spark jobs, but when I use hive on spark, it
only started a few executors although there are more enough cores and memories to start more
executors.
For example, for the same SQL query, if I run on sparkSQL, it can start more than 20 executors,
but with hive on spark, only 3.

How can I improve the performance on hive on spark? Any suggestions please.

Thanks,
Minghao Feng


Mime
View raw message