reef-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Julia Wang (QIUHE)" <Qiuhe.W...@microsoft.com.INVALID>
Subject RE: Questions on REEF .NET Driver execution
Date Fri, 09 Mar 2018 00:09:50 GMT
Yes, the command line we submitted to YARN job is 

reef\Org.Apache.REEF.Bridge.exe with class path, and 
-Dproc_reef org.apache.reef.bridge.client.YarnBootstrapREEFLauncher yarn-job-parameters.json
reef/local/yarn-app-parameters.json

YarnBootstrapREEFLauncher is the entry class in Java for .Net launcher. Eventually it calls
REEFLauncher in Java. 

What is your scenario? Do you use REEF client? How do you submit Job to YARN? 

-- Julia

-----Original Message-----
From: Chenxi Zhao [mailto:Chenxi.Zhao@microsoft.com.INVALID] 
Sent: Thursday, March 8, 2018 2:43 PM
To: dev@reef.apache.org
Subject: RE: Questions on REEF .NET Driver execution

Appreciate everyone's help! I finally got through the problem.

The reason is that in order to let java bridge class to able to call JNI to Org.Apache.REEF.Bridge,
it has to go through "Org.Apache.REEF.Bridge.exe". So the command line would be like "Org.Apache.REEF.Bridge.exe
java -Xmx512m -XX:PermSize=128m -XX:MaxPermSize=128m -classpath 'reef/local/*;reef/global/*;'
-Dproc_reef org.apache.reef.bridge.client.AzureBatchBootstrapREEFLauncher reef/job-submission-params.json"
(Here uses my ongoing AzureBatch implementation).

I found a secondary issue is that, to implement a bridge launcher for a runtime in lang/java/reef-bridge-client,
the class name has to be a substring of "REEFLauncher". It is enforced in https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgithub.com%2Fapache%2Freef%2Fblob%2Fmaster%2Flang%2Fcs%2FOrg.Apache.REEF.Bridge%2FDriverLauncher.cpp%23L92&data=04%7C01%7CQiuhe.Wang%40microsoft.com%7C8c3d950caa4b4923613108d58545f509%7C72f988bf86f141af91ab2d7cd011db47%7C1%7C0%7C636561457870462102%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDAiLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwifQ%3D%3D%7C-2&sdata=4uYz2TGqBkMu6NXxOoIZbtxi7sMdVnFrdV%2BulArzTxg%3D&reserved=0,
which is very unfriendly code. I think this may be worth to open a ticket. Shall we?

Thanks,
Chenxi




-----Original Message-----
From: Julia Wang (QIUHE) <Qiuhe.Wang@microsoft.com.INVALID>
Sent: Thursday, March 8, 2018 1:20 PM
To: dev@reef.apache.org
Subject: RE: Questions on REEF .NET Driver execution

.Net hello REEF runs on environment that needs both .Net and Java installed. If runs on YARN,
it also needs Hadoop in the environment set up. Not sure what additional settings are required
for AzureBatch. 

For dlls and jar files used by REEF driver, as long as you put all REEF dlls in the current
client folder, REEF.Client wraps all needed jar files and extract them on the fly. Additional
application files can be set through JobRequestBuilder and also put in the client working
directory, REEF will pick them up. 

--Julia

-----Original Message-----
From: Scott Inglis [mailto:msinglft@gmail.com]
Sent: Thursday, March 8, 2018 12:05 PM
To: dev@reef.apache.org
Subject: Re: Questions on REEF .NET Driver execution

Chenxi and I also talked offline -- and I wanted to reflect those discussions to the dev alias.

So with his changes, he is using the HelloREEF example to test the Azure batch runtime. If
he changes HelloREEF back to local runtime, it works. So its only when running with AzureBatch
that this happens. Because this works with the local runtime, I think we can rule out any
missing dll dependencies (as this change does add Azure SDK dlls to the project from what
I understand).

So the other possibility is the current working folder and how the paths are handled -- maybe
these are different for the runtime? Like local does something different than what AzureBatch
does with setting up the environment.

Chenxi was going to look into this area.


On Thu, Mar 8, 2018 at 11:34 AM, Markus Weimer <markus@weimo.de> wrote:

> On Wed, Mar 7, 2018 at 11:08 PM, Chenxi Zhao < 
> Chenxi.Zhao@microsoft.com.invalid> wrote:
>
> > I notice that in .NET, Client is submitting to the java bridge classes.
> > E.g. YarnBootstrapREEFLauncher and LocalClient. In these launchers, 
> > they are binding Driver configuration like ON_DRIVER_STARTED to .NET 
> > side
> driver
> > code through JNI call
> > https://na01.safelinks.protection.outlook.com/?url=https%3A%2F%2Fgit
> > hub.com%2Fapache%2F&data=04%7C01%7CQiuhe.Wang%40microsoft.com%7Cf916
> > 44d4841c428bd51908d5852fdebd%7C72f988bf86f141af91ab2d7cd011db47%7C1%
> > 7C0%7C636561363004417602%7CUnknown%7CTWFpbGZsb3d8eyJWIjoiMC4wLjAwMDA
> > iLCJQIjoiV2luMzIiLCJBTiI6Ik1haWwifQ%3D%3D%7C-2&sdata=srwjDmNK4%2FxMt
> > fFPQejocbLhhVnu2KeFqkr%2BujssrX0%3D&reserved=0
> > reef/blob/master/lang/java/reef-bridge-java/src/main/
> java/org/apache/reef/
> > javabridge/NativeInterop.java#L38).
> >
> > Could some one help me understand how java bridge finds the dll that 
> > contains those natives calls in local/Yarn runtime?
> >
>
> IIRC, we are asking YARN to launch `Driver.exe` which in turn depends 
> on that DLL. That exe launches both the JVM and the CLR in-process and 
> wires it all up. Doug would know better :)
>
> Markus
>
Mime
View raw message