reef-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Byung-Gon Chun <>
Subject Re: Write-up about REEF as a library
Date Wed, 15 Feb 2017 22:59:21 GMT
Thanks for the nice summary!
Looking forward to listening to the lessons learned from REEF on Spark.

2017. 2. 16. 오전 7:48 Markus Weimer <> 작성:

> As I have some knowledge of both these efforts, maybe I can provide a useful summary:
> Gon's students were working on Spark-on-REEF. The idea being to replace the bottom parts
of the Spark system with REEF: Resource allocation, communications protocols and the likes.
One instant benefit of that is that such a version of Spark can run anywhere REEF is available.
> Sergiy is now working on the inverse architecture: REEF on Spark. The idea here is that
REEF Jobs can be part of a longer Spark pipeline. The REEF job then gets Spark's Executors
as Evaluators and access to the data stored on them. The benefit of this approach is that
we add the flexibility of REEF to the Spark surface without giving up on the attractive API
veneer Spark can offer over both.
> Markus

View raw message