reef-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Reed Umbrasas <ryum...@microsoft.com.INVALID>
Subject RE: How does REEFLauncher get copied over to the cluster?
Date Fri, 19 Jan 2018 04:45:16 GMT
Thank you! I spent two days running the examples from IntelliJ and wondering why the zip didn’t
have any of the REEF classes. It turns out things work as expected when invoking my program
from the command line via the "uber" jar.

Reed

-----Original Message-----
From: Sergiy Matusevych [mailto:sergiy.matusevych@gmail.com] 
Sent: Thursday, January 18, 2018 7:39 PM
To: dev@reef.apache.org
Subject: Re: How does REEFLauncher get copied over to the cluster?

Hi Reed,

You are right that REEF jars have to be part of the job configuration on the client. In our
HelloREEF example, we specify only one jar - the one that contains the user application classes.
We usually assume that if the jar contains HelloDriver, all other necessary files are also
there (including REEF files). That works *only* if our application build process produces
an uber-jar that contains all dependencies. usually, it's a file named ./target/*-shaded.jar

If you don't have an uber-jar, you have to add all required dependencies by hand to the app
configuration - using the same

.set(DriverConfiguration.GLOBAL_LIBRARIES, "...")

call.

Hope that helps!

Cheers,
Sergiy.




On Thu, Jan 18, 2018 at 11:59 AM, Reed Umbrasas < ryumbra@microsoft.com.invalid> wrote:

> Hi,
>
> Can someone please point me to where REEF framework classes (i.e.
> REEFLauncher) get copied over to the remote cluster?
>
> I found that the job submission handler does package up a JAR file (
> https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithu
> b.com%2Fapache%2Freef%2Fblob%2Fmaster%2Flang%2Fjava%2F&data=02%7C01%7C
> ryumbra%40microsoft.com%7C35d6d93cf5d342b787da08d55eee36f9%7C72f988bf8
> 6f141af91ab2d7cd011db47%7C1%7C0%7C636519299576100801&sdata=vq4vlrZdcRj
> c7FwZIUOCJSjDNNOx0TzEnqZq5nrvZRI%3D&reserved=0
> reef-runtime-hdinsight/src/main/java/org/apache/reef/
> runtime/hdinsight/client/HDInsightJobSubmissionHandler.java#L99) and 
> copy it over to the cluster; however, it only seems to contain the 
> user classes (i.e. HelloDriver). So how do the classes provided by 
> REEF make it into the remote cluster from the client?
>
> Thanks,
> Reed
>
Mime
View raw message