reef-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Sergiy Matusevych (JIRA)" <j...@apache.org>
Subject [jira] [Created] (REEF-1787) Unmanaged REEF on Spark packages global.jar resources incorrectly
Date Wed, 26 Apr 2017 17:52:04 GMT
Sergiy Matusevych created REEF-1787:
---------------------------------------

             Summary: Unmanaged REEF on Spark packages global.jar resources incorrectly
                 Key: REEF-1787
                 URL: https://issues.apache.org/jira/browse/REEF-1787
             Project: REEF
          Issue Type: Bug
          Components: REEF Driver, REEF Runtime YARN
         Environment: YARN 2.7.3+
            Reporter: Sergiy Matusevych
            Assignee: Sergiy Matusevych


When REEF application is being launched in Unmanaged AM mode from Spark driver on YARN, it
packages the global resources incorrectly. As a result, REEF Evaluators cannot find {{REEFLauncher}}
class and fail.

All global resources required by REEF application are being packaged into the {{global.jar}}
file on HDFS; however, when packaged from Spark, the resources include full path to the files,
e.g.
{code}
$ jar tvf global.jar
 15082 Wed Apr 19 19:03:50 PDT 2017 mnt/data/0/local/nm/usercache/hadoop/filecache/18/reef-driver-on-spark_2.11-1.01.jar
16963704 Wed Apr 19 19:03:50 PDT 2017 mnt/data/0/local/nm/usercache/hadoop/filecache/15/reef-examples-0.16.0-SNAPSHOT-shaded.jar
{code}
whereas the same {{global.jar}} produced by REEF client does _not_ include such paths.

We have to investigate whether absence or presence of such paths in the {{global.jar}} causes
the Evaluator start failure.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

Mime
View raw message