hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "JoneZhang (JIRA)" <j...@apache.org>
Subject [jira] [Created] (HIVE-12649) Hive on Spark will resubmitted application when not enough resouces to launch yarn application master
Date Fri, 11 Dec 2015 03:56:11 GMT
JoneZhang created HIVE-12649:
--------------------------------

             Summary: Hive on Spark will resubmitted application when not enough resouces
to launch yarn application master
                 Key: HIVE-12649
                 URL: https://issues.apache.org/jira/browse/HIVE-12649
             Project: Hive
          Issue Type: Bug
    Affects Versions: 1.2.1, 1.1.1
            Reporter: JoneZhang
            Assignee: Xuefu Zhang


Hive on spark will estimate reducer number when the query is not set reduce number,which cause
a application submit.The application will pending if the yarn queue's resources is insufficient.
So there are more than one pending applications probably because 
there are more than one estimate call.The failure is soft, so it doesn't prevent subsequent
processings. We can make that a hard failure

That code is found in 
728237     at org.apache.hadoop.hive.ql.exec.spark.SparkUtilities.getSparkSession(SparkUtilities.java:112)
 728238     at org.apache.hadoop.hive.ql.optimizer.spark.SetSparkReducerParallelism.process(SetSparkReducerParallelism.java:115)




--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message