Mailing-List: contact hama-user-help@incubator.apache.org; run by ezmlm
Precedence: bulk
Reply-To: hama-user@incubator.apache.org
Received-SPF: pass (nike.apache.org: domain of mspreitz@us.ibm.com designates
 32.97.182.138 as permitted sender)
In-Reply-To: 
 <CAJ-=ys=4xiDkGX5xN+b-08TW--cb0CELcyvVfR97Cbi1-fXEGw@mail.gmail.com>
References: 
 <CADYHM8zOJv0UgK1T+0-ehO5V15CXupcGyOVSDMzBx4Rx5nnBbg@mail.gmail.com>
	<4EE1B704.2000600@apache.org>
	<CADYHM8z5BwghJzua_wsZFx4TuD8nsOfN=Rau8LCUz8s1d783-Q@mail.gmail.com>
 <CAJ-=ys=4xiDkGX5xN+b-08TW--cb0CELcyvVfR97Cbi1-fXEGw@mail.gmail.com>
To: hama-user@incubator.apache.org
MIME-Version: 1.0
Subject: Re: Comparing BSP and MR
From: Mike Spreitzer <mspreitz@us.ibm.com>
Message-ID: 
 <OFC3C0309E.88FA3E61-ON85257961.0067BD10-85257961.00689514@us.ibm.com>
Date: Fri, 9 Dec 2011 14:02:19 -0500
Content-Type: multipart/alternative;
 boundary="=_alternative 006893DC85257961_="

--=_alternative 006893DC85257961_=
Content-Type: text/plain; charset="US-ASCII"

It looks to me like the original questions, and the ensuing discussion, 
are a mix of two concerns.  (1) BSP has some abstractions that have been 
realized in many concrete systems, and these abstractions can be compared 
to MapReduce independently of the details of particular implementations. 
(2) is asking for details of the implementations in Giraph and Hama.  I 
think it is very valuable to separate these.  Even for concrete systems 
like Hama and Giraph, the APIs are sufficiently abstract to allow quite a 
variety of implementations.

Regarding the comparison of abstractions, the first thing to note is that 
MapReduce omits two important concepts from BSP, namely iteration and 
per-component state.  To make a very meaningful comparison, you have to 
compare BSP with an iterated use of MapReduce.  Imagine running the same 
MapReduce job over and over again, with the output of one iteration being 
the input to the next.  In this case you can pair up each reduce task 
(except those of the final iteration) of one iteration with a map task of 
the following iteration --- the one that reads the reduce task's output. 
It is that pair that corresponds to a "compute" invocation in BSP.  Note 
that in iterated MapReduce there are two synchronization barriers per 
iteration: one between map and reduce, and one between iterations.  BSP 
has just one barrier per iteration.  Iterated MapReduce externalizes the 
intermediate data that flows from one iteration's reduce tasks to the next 
iteration's map tasks; in the corresponding BSP computation this 
intermediate data is just held inside the compute invocations.

Regards,
Mike
--=_alternative 006893DC85257961_=--