reef-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sergiy Matusevych <sergiy.matusev...@gmail.com>
Subject REEF runs from Spark!
Date Thu, 27 Apr 2017 05:28:00 GMT
Hi fellow reefers,

After almost a year of refactorings and bugfixes, I was able to run REEF
application from Spark today. Woo-hoo! All I tried so far was Hello REEF
application running from a Spark driver on YARN, but other Java/REEF/YARN
applications should also run from Spark just fine. I'll keep testing it
tomorrow.

There is a PR pending: https://github.com/apache/reef/pull/1299

We still have a lot of work to do - we need to try (and write unit tests!)
for various failure scenarios, test on different versions of YARN, fix
outstanding related bugs, such as
https://issues.apache.org/jira/browse/REEF-1782 , and, most importantly,
make sure we can launch .NET applications from Spark.

There is a lot of fun work to do - please check out the JIRA at
https://issues.apache.org/jira/browse/REEF and send us your pull requests!

Cheers,
Sergiy.

P.S. I have a sample Spark+REEF app in a private repo. We will open source
it soon, although there is not much to open source - it's basically a
REEF-on-REEF example (namely, a try block from
org.apache.reef.examples.reefonreef.ReefOnReefDriver), transplanted almost
verbatim into the Spark driver.

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message