reef-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Markus Weimer (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (REEF-1787) Unmanaged REEF on Spark packages global.jar resources incorrectly
Date Wed, 26 Apr 2017 18:01:04 GMT

    [ https://issues.apache.org/jira/browse/REEF-1787?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15985273#comment-15985273
] 

Markus Weimer commented on REEF-1787:
-------------------------------------

Interesting. This looks as if we erroneously follow the symlinks and use the original path
in the JAR we create on the driver. There should be an option to use the path to the symlink
instead in the ZIP file library we use.

> Unmanaged REEF on Spark packages global.jar resources incorrectly
> -----------------------------------------------------------------
>
>                 Key: REEF-1787
>                 URL: https://issues.apache.org/jira/browse/REEF-1787
>             Project: REEF
>          Issue Type: Bug
>          Components: REEF Driver, REEF Runtime YARN
>         Environment: YARN 2.7.3+
>            Reporter: Sergiy Matusevych
>            Assignee: Sergiy Matusevych
>              Labels: jar, resource, spark, yarn
>   Original Estimate: 168h
>  Remaining Estimate: 168h
>
> When REEF application is being launched in Unmanaged AM mode from Spark driver on YARN,
it packages the global resources incorrectly. As a result, REEF Evaluators cannot find {{REEFLauncher}}
class and fail.
> All global resources required by REEF application are being packaged into the {{global.jar}}
file on HDFS; however, when packaged from Spark, the resources include full path to the files,
e.g.
> {code}
> $ jar tvf global.jar
>  15082 Wed Apr 19 19:03:50 PDT 2017 mnt/data/0/local/nm/usercache/hadoop/filecache/18/reef-driver-on-spark_2.11-1.01.jar
> 16963704 Wed Apr 19 19:03:50 PDT 2017 mnt/data/0/local/nm/usercache/hadoop/filecache/15/reef-examples-0.16.0-SNAPSHOT-shaded.jar
> {code}
> whereas the same {{global.jar}} produced by REEF client does _not_ include such paths.
> We have to investigate whether absence or presence of such paths in the {{global.jar}}
causes the Evaluator start failure.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message