hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Bibin A Chundatt (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (YARN-5842) spark job getting failed with memory not avail
Date Sun, 06 Nov 2016 03:11:58 GMT

    [ https://issues.apache.org/jira/browse/YARN-5842?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15641020#comment-15641020
] 

Bibin A Chundatt commented on YARN-5842:
----------------------------------------

In nodemanager {{yarn.nodemanager.pmem-check-enabled}} is true and {{yarn.nodemanager.vmem-pmem-ratio}}
ratio by default is {{2.1}} . vm reaches {{4G*2.1}} would be the limit in your case. Check
the container fail is due to increase in virtual memory 

> spark job getting failed with memory not avail
> ----------------------------------------------
>
>                 Key: YARN-5842
>                 URL: https://issues.apache.org/jira/browse/YARN-5842
>             Project: Hadoop YARN
>          Issue Type: Task
>          Components: applications
>         Environment: spark running in emr 4.3 with hadoop 2.7 and spark 1.6.0
>            Reporter: Mohamed Kajamoideen
>
> > config <- spark_config()
> > config$`sparklyr.shell.driver-memory` <- "4G"
> > config$`sparklyr.shell.executor-memory` <- "4G"
> > sc <- spark_connect(master = "yarn-client", config = config) 
> Error: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage
27.0 failed 4 times, most recent failure: Lost task 0.3 in stage 27.0 (TID 1941, ip- .ec2.internal):
org.apache.spark.SparkException: Values to assemble cannot be null.
> 	at org.apache.spark.ml.feature.VectorAssembler$$anonfun$assemble$1.apply(VectorAssembler.scala:154)
> 	at org.apache.spark.ml.feature.VectorAssembler$$anonfun$assemble$1.apply(VectorAssembler.scala:137)
> 	at scala.collection.IndexedSeqOptimized$class.foreach(IndexedSeqOptimized.scala:33)
> 	at scala.collection.mutable.WrappedArray.foreach(WrappedArray.scala:34)
> 	at org.apache.spark.ml.feature.VectorAssembler$.assemble(VectorAssembler.scala:137)
> 	at org.apache.spark.ml.feature.VectorAssembler$$anonfun$3.apply(VectorAssembler.scala:95)
> 	at org.apache.spark.ml.feature.VectorAssembler$$anonfun$3.apply(VectorAssembler.scala:94)
> 	at org.apache.spark.sql.catalyst.expressions.GeneratedClass$SpecificUnsafeProjection.apply(Unknown
Sou



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org


Mime
View raw message