reef-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sergiy Matusevych (JIRA)" <>
Subject [jira] [Commented] (REEF-1787) Unmanaged REEF on Spark packages global.jar resources incorrectly
Date Wed, 26 Apr 2017 18:09:04 GMT


Sergiy Matusevych commented on REEF-1787:

Yes. On Spark Driver, the resource is addressed by symlink to another symlink; REEF successfully
packages the required files, but it fails to trim the files' paths inside the {{global.jar}}.
I am fixing it now.

> Unmanaged REEF on Spark packages global.jar resources incorrectly
> -----------------------------------------------------------------
>                 Key: REEF-1787
>                 URL:
>             Project: REEF
>          Issue Type: Bug
>          Components: REEF Driver, REEF Runtime YARN
>         Environment: YARN 2.7.3+
>            Reporter: Sergiy Matusevych
>            Assignee: Sergiy Matusevych
>              Labels: jar, resource, spark, yarn
>   Original Estimate: 168h
>  Remaining Estimate: 168h
> When REEF application is being launched in Unmanaged AM mode from Spark driver on YARN,
it packages the global resources incorrectly. As a result, REEF Evaluators cannot find {{REEFLauncher}}
class and fail.
> All global resources required by REEF application are being packaged into the {{global.jar}}
file on HDFS; however, when packaged from Spark, the resources include full path to the files,
> {code}
> $ jar tvf global.jar
>  15082 Wed Apr 19 19:03:50 PDT 2017 mnt/data/0/local/nm/usercache/hadoop/filecache/18/reef-driver-on-spark_2.11-1.01.jar
> 16963704 Wed Apr 19 19:03:50 PDT 2017 mnt/data/0/local/nm/usercache/hadoop/filecache/15/reef-examples-0.16.0-SNAPSHOT-shaded.jar
> {code}
> whereas the same {{global.jar}} produced by REEF client does _not_ include such paths.
> We have to investigate whether absence or presence of such paths in the {{global.jar}}
causes the Evaluator start failure.

This message was sent by Atlassian JIRA

View raw message