hadoop-mapreduce-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ralph Castain <...@open-mpi.org>
Subject Re: MapReduce and MPI
Date Thu, 01 Dec 2011 22:15:37 GMT

On Dec 1, 2011, at 2:47 PM, <Milind.Bhandarkar@emc.com> wrote:

> Ralph,
> 
> At the MPI Forum meeting at SC11, Jeff mentioned that C++ bindings are
> going to be dropped from the standard,

Yes - reason being mostly that (a) very few applications use them, and (b) they have proven
to be more trouble than they are worth. We are constantly finding bugs due to conflicts between
MPI specifications and C++ compilers, and (quite frankly) the lack of experienced C++ programmers
in the MPI developer community is a serious problem. So keeping those bindings alive is difficult.

> and that no other language bindings
> were proposed. Do you think there is enough time for Java bindings to make
> it into the 3.0 standard ?

I don't know about the 3.0 standard -  could happen, if I can do it fast enough and the Forum
accepts it for that release, or may have to follow in 3.1. The obstacle we have to overcome
re the Forum is that Java got a bad name in the early years of the binding attempts due to
performance issues and lack of attention to details. The performance "problem" largely stemmed
from the issue of binding processes to at least NUMA regions - the C implementations were
far faster - and the poor performance of Java in general during that time. The latter has
largely been resolved over the years, and the former is solvable with some work.

The detail issue reflected the problem of trying to create a single, non-sectarian set of
Java bindings that fit all MPI implementations. This meant that you could really only cover
90% of MPI functionality - beyond that, you have to integrate tightly to the implementation.
The academics who did the original work didn't want to do so, and thus left functions out,
resulting in the MPI community "looking down" on the result.

All put together, the MPI community wound up not thinking much of the Java world. As I said,
things have changed, and I believe a high-quality implementation of Java bindings can gain
acceptance. Once we have it for one MPI, we can (due to the OMPI license) offer it up to the
other implementations with a fair degree of confidence they will adopt it.

As for the MPI Forum, what we need is a "champion" to propose adoption of the bindings once
implemented. If people want them (i.e., the user community is larger than C++, which has a
total of 3 identified applications), we can show the implementation is of quality, and we
have developers willing to support it, then we can get them adopted.

I've scoped the job and it looks doable with reasonable effort. One other person on the list
(Deepak Sharma) has offered to help, and Jeff has offered to provide advice as he wrote the
original OMPI bindings. Getting it thru the OMPI devel approval represents a miniature MPI
Forum process, but I think we can do it given Jeff and my roles there.

HTH
Ralph

> 
> - Milind
> 
> On 12/1/11 3:31 AM, "Ralph Castain" <rhc@open-mpi.org> wrote:
> 
>> Hi folks
>> 
>> I'm a lead developer on the Open MPI project, and recently joined the
>> Hadoop community to help with Hamster. A couple of people have asked me
>> about using MPI more generally inside MapReduce, and it does indeed seem
>> a good candidate to use that method of communication.
>> 
>> It seems to me, though, that a pre-requisite for moving that direction is
>> a good set of Java MPI bindings. I've checked with the MPI community, and
>> while there have been a couple of attempts at this, nothing really
>> "adopted" by the community has been done.
>> 
>> I have some spare time while I search for a new job, so I thought I might
>> tackle this over the next few weeks. However, I thought I would assess
>> the actual level of interest before investing the time - it isn't a
>> trivial thing to do, but doable if there is interest.
>> 
>> Thanks
>> Ralph
>> 
>> 
> 


Mime
View raw message