spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ravindra (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (SPARK-26606) parameters passed in extraJavaOptions are not being picked up
Date Fri, 11 Jan 2019 19:22:00 GMT

    [ https://issues.apache.org/jira/browse/SPARK-26606?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16740683#comment-16740683
] 

Ravindra edited comment on SPARK-26606 at 1/11/19 7:21 PM:
-----------------------------------------------------------

So, if you look at the below image, you can see the spark launch command for my spark job.
So, you can see the launch command which has the expected values. But the individual parts
like  "-Dapp.env=prod"  "-Dapp.country=US" "-Dapp.banner=WMT" are missing. I am suspecting
this could be the reason why my source code is throwing an exception that the jvm params are
null.   !Screen Shot 2019-01-11 at 1.12.33 PM.png!


was (Author: rrb441):
So, if you look at the below image, you can see the spark launch command for my spark job.
So, you can see the launch command which has the expected values. But the individual parts
like  "-Dapp.env=prod"  "-Dapp.country=US" "-Dapp.banner=WMT" are missing. I am suspecting
this could be the reason why my source code throwing an exception that the jvm params are
null.   !Screen Shot 2019-01-11 at 1.12.33 PM.png!

> parameters passed in extraJavaOptions are not being picked up 
> --------------------------------------------------------------
>
>                 Key: SPARK-26606
>                 URL: https://issues.apache.org/jira/browse/SPARK-26606
>             Project: Spark
>          Issue Type: Bug
>          Components: Spark Submit
>    Affects Versions: 2.3.1
>            Reporter: Ravindra
>            Priority: Major
>              Labels: java, spark
>         Attachments: Screen Shot 2019-01-09 at 4.31.01 PM.png, Screen Shot 2019-01-11
at 1.12.33 PM.png
>
>
> driver.extraJavaOptions and executor.extraJavaOptions are not being picked up . Even
though I see the parameters are being passed fine in the spark launch command I do not see
these parameters are being picked up for some unknown reason. My source code throws an error
stating the java params are empty
>  
> This is my spark submit command: 
>     output=`spark-submit \
>  --class com.demo.myApp.App \
>  --conf 'spark.executor.extraJavaOptions=-Dapp.env=dev -Dapp.country=US -Dapp.banner=ABC
-Doracle.net.tns_admin=/work/artifacts/oracle/current -Djava.security.egd=[file:/dev/./urandom|file:///dev/urandom]' \
>  --conf 'spark.driver.extraJavaOptions=-Dapp.env=dev -Dapp.country=US -Dapp.banner=ABC
-Doracle.net.tns_admin=/work/artifacts/oracle/current -Djava.security.egd=[file:/dev/./urandom|file:///dev/urandom]'
\
>  --executor-memory "$EXECUTOR_MEMORY" \
>  --executor-cores "$EXECUTOR_CORES" \
>  --total-executor-cores "$TOTAL_CORES" \
>  --driver-memory "$DRIVER_MEMORY" \
>  --deploy-mode cluster \
>  /home/spark/asm//current/myapp-*.jar 2>&1 &`



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org


Mime
View raw message