hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Brock Noland (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-9040) Spark Memory can be formatted string
Date Mon, 08 Dec 2014 22:33:13 GMT

    [ https://issues.apache.org/jira/browse/HIVE-9040?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14238586#comment-14238586
] 

Brock Noland commented on HIVE-9040:
------------------------------------

Thank you! +1 pending tests

> Spark Memory can be formatted string
> ------------------------------------
>
>                 Key: HIVE-9040
>                 URL: https://issues.apache.org/jira/browse/HIVE-9040
>             Project: Hive
>          Issue Type: Sub-task
>          Components: Spark
>            Reporter: Brock Noland
>            Assignee: Jimmy Xiang
>             Fix For: spark-branch
>
>         Attachments: HIVE-9040.1-spark.patch, HIVE-9040.2-spark.patch, HIVE-9040.3-spark.patch
>
>
> Here: https://github.com/apache/hive/blob/spark/ql/src/java/org/apache/hadoop/hive/ql/exec/spark/session/SparkSessionImpl.java#L72
> we call {{getInt}} on {{spark.executor.memory}} but this is a formatted string, example
here: http://spark.apache.org/docs/1.0.1/configuration.html as such, I get:
> {noformat}
> 2014-12-08 03:04:48,114 WARN  [HiveServer2-Handler-Pool: Thread-34]: spark.SetSparkReducerParallelism
(SetSparkReducerParallelism.java:process(141)) - Failed to create spark client.
> java.lang.NumberFormatException: For input string: "23000m"
>         at java.lang.NumberFormatException.forInputString(NumberFormatException.java:65)
>         at java.lang.Integer.parseInt(Integer.java:492)
>         at java.lang.Integer.parseInt(Integer.java:527)
>         at scala.collection.immutable.StringLike$class.toInt(StringLike.scala:229)
>         at scala.collection.immutable.StringOps.toInt(StringOps.scala:31)
>         at org.apache.spark.SparkConf$$anonfun$getInt$2.apply(SparkConf.scala:184)
>         at org.apache.spark.SparkConf$$anonfun$getInt$2.apply(SparkConf.scala:184)
>         at scala.Option.map(Option.scala:145)
>         at org.apache.spark.SparkConf.getInt(SparkConf.scala:184)
>         at org.apache.hadoop.hive.ql.exec.spark.session.SparkSessionImpl.getMemoryAndCores(SparkSessionImpl.java:72)
> {noformat}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message