reef-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sergiy Matusevych <>
Subject REEF runs from Spark!
Date Thu, 27 Apr 2017 05:28:00 GMT
Hi fellow reefers,

After almost a year of refactorings and bugfixes, I was able to run REEF
application from Spark today. Woo-hoo! All I tried so far was Hello REEF
application running from a Spark driver on YARN, but other Java/REEF/YARN
applications should also run from Spark just fine. I'll keep testing it

There is a PR pending:

We still have a lot of work to do - we need to try (and write unit tests!)
for various failure scenarios, test on different versions of YARN, fix
outstanding related bugs, such as , and, most importantly,
make sure we can launch .NET applications from Spark.

There is a lot of fun work to do - please check out the JIRA at and send us your pull requests!


P.S. I have a sample Spark+REEF app in a private repo. We will open source
it soon, although there is not much to open source - it's basically a
REEF-on-REEF example (namely, a try block from
org.apache.reef.examples.reefonreef.ReefOnReefDriver), transplanted almost
verbatim into the Spark driver.

  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message