Mailing-List: contact dev-help@reef.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@reef.apache.org
Subject: Re: Write-up about REEF as a library
To: dev@reef.apache.org
References: <CADq0cj7d0Uf5-ytz-Xz7dknuFVB07o8ogoiZTs9Vt1LyaXSFxA@mail.gmail.com>
 <047d7e95-7a34-5656-89e1-d2ab797d7044@weimo.de>
 <CADq0cj4hA7OY_vhX-c_G7Xz20i1C1gKCV_sN3CXCmK5hg4Gg2w@mail.gmail.com>
 <CAJ3duG7REuEwO-u6jOv4aV5Gqn6p7ZE6KvSqD3PwytnBCgV=aA@mail.gmail.com>
 <CADq0cj6fHdrrRUQ3HVNSL5pt6WT=+hVb9BBnHAxORGeWVVbrzA@mail.gmail.com>
 <CAJ3duG5A4mQNQGpfm0cyRfM=_Urpf9Mbq6ROYSTHvSr2tH44sQ@mail.gmail.com>
From: Markus Weimer <markus@weimo.de>
Message-ID: <9c8fe2c7-b536-3eda-a750-8292fc6d7b49@weimo.de>
Date: Wed, 15 Feb 2017 14:48:38 -0800
User-Agent: Mozilla/5.0 (Windows NT 10.0; WOW64; rv:45.0) Gecko/20100101
 Thunderbird/45.7.1
MIME-Version: 1.0
In-Reply-To: <CAJ3duG5A4mQNQGpfm0cyRfM=_Urpf9Mbq6ROYSTHvSr2tH44sQ@mail.gmail.com>
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 7bit
archived-at: Wed, 15 Feb 2017 22:48:45 -0000

As I have some knowledge of both these efforts, maybe I can provide a 
useful summary:

Gon's students were working on Spark-on-REEF. The idea being to replace 
the bottom parts of the Spark system with REEF: Resource allocation, 
communications protocols and the likes. One instant benefit of that is 
that such a version of Spark can run anywhere REEF is available.

Sergiy is now working on the inverse architecture: REEF on Spark. The 
idea here is that REEF Jobs can be part of a longer Spark pipeline. The 
REEF job then gets Spark's Executors as Evaluators and access to the 
data stored on them. The benefit of this approach is that we add the 
flexibility of REEF to the Spark surface without giving up on the 
attractive API veneer Spark can offer over both.

Markus