reef-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Markus Weimer <>
Subject Re: Write-up about REEF as a library
Date Wed, 15 Feb 2017 22:48:38 GMT
As I have some knowledge of both these efforts, maybe I can provide a 
useful summary:

Gon's students were working on Spark-on-REEF. The idea being to replace 
the bottom parts of the Spark system with REEF: Resource allocation, 
communications protocols and the likes. One instant benefit of that is 
that such a version of Spark can run anywhere REEF is available.

Sergiy is now working on the inverse architecture: REEF on Spark. The 
idea here is that REEF Jobs can be part of a longer Spark pipeline. The 
REEF job then gets Spark's Executors as Evaluators and access to the 
data stored on them. The benefit of this approach is that we add the 
flexibility of REEF to the Spark surface without giving up on the 
attractive API veneer Spark can offer over both.


View raw message