spark-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Mahesh (JIRA)" <>
Subject [jira] [Comment Edited] (SPARK-21399) Difficult to debug code in spark-class
Date Fri, 14 Jul 2017 06:26:00 GMT


Mahesh edited comment on SPARK-21399 at 7/14/17 6:25 AM:

Closing the issue based on response from Marcelo.

I did the get the problem eventually, the jdk I use returns 1 even upon successful launch,
which will cause spark-class to terminate the run. 

Thanks for everyone's time.

was (Author: maheshs):
Closing the issue based on response from Marcelo.

> Difficult to debug code in spark-class
> --------------------------------------
>                 Key: SPARK-21399
>                 URL:
>             Project: Spark
>          Issue Type: Improvement
>          Components: Spark Shell, Spark Submit
>    Affects Versions: 2.1.1
>         Environment: Running spark on a version of hottub.
>            Reporter: Mahesh
>            Priority: Minor
> The code below is difficult to debug, and its not clear what is happening, it appears
to use some kind of shell magic, if possible it should be replaced with an easier to understand
code. Specifically there needs to be a log of the exact java command that is finally invoked.

> Using input redirection means that adding comments to the build_command will break the
code ,I am not sure but the below code for build_command is not equivalent to one in the code,
although semantically it should be.
> build_command() {
>   "$RUNNER" -Xmx128m -cp "$LAUNCH_CLASSPATH" org.apache.spark.launcher.Main "$@"
>   printf "%d\0" $?
>   #printf "Something\0"
> }
> *Original code below*
> *build_command() {
>   "$RUNNER" -Xmx128m -cp "$LAUNCH_CLASSPATH" org.apache.spark.launcher.Main "$@"
>   printf "%d\0" $?
> }
> CMD=()
> while IFS= read -d '' -r ARG; do
>   CMD+=("$ARG")
> done < <(build_command "$@")*

This message was sent by Atlassian JIRA

To unsubscribe, e-mail:
For additional commands, e-mail:

View raw message