hadoop-mapreduce-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ralph Castain <...@open-mpi.org>
Subject Re: MapReduce and MPI
Date Wed, 21 Dec 2011 06:22:07 GMT
Just a quick update on this notion. Several of us in the OMPI community got together and successfully
integrated Java bindings into the OMPI code base, and we have enough support that we can probably
get this approved within that organization. I've written a wrapper compiler and added support
within mpirun to make it relatively easy to use, so what remains is documentation (hope to
have an initial cut at that done on Wed) and extending coverage to all MPI functions (we have
send/recv and a number of other basic things done, but still need collectives and MPI-2 dynamics).
The latter will be a work-in-progress (there are a LOT of MPI functions), with the more common
functions covered over the next few weeks.

We also need test codes, of course, and could use help with generating those plus actual testing.
 Volunteers are welcome. There are several Fortran and C test suites out there that are rather
extensive - having some subset of those in Java would be a major step forward. I can point
you to the branch where this work is being done (it is public, with controlled write privileges)
and provide example tests on request.

As for the 3.0 standard, that is indeed out-of-reach. The MPI Forum requires 9 months lead
time for approval of any new proposal, and the 3.0 approval meeting is in Jan. However, this
is a continuous process with revisions being released on a quarterly basis. So it isn't a
"hit the date or die" issue - it is strictly a question of persevering long enough to gain
acceptance, and the pace of the process will largely be driven by the level of user interest.

HTH
Ralph

On Dec 1, 2011, at 3:15 PM, Ralph Castain wrote:

> 
> On Dec 1, 2011, at 2:47 PM, <Milind.Bhandarkar@emc.com> wrote:
> 
>> Ralph,
>> 
>> At the MPI Forum meeting at SC11, Jeff mentioned that C++ bindings are
>> going to be dropped from the standard,
> 
> Yes - reason being mostly that (a) very few applications use them, and (b) they have
proven to be more trouble than they are worth. We are constantly finding bugs due to conflicts
between MPI specifications and C++ compilers, and (quite frankly) the lack of experienced
C++ programmers in the MPI developer community is a serious problem. So keeping those bindings
alive is difficult.
> 
>> and that no other language bindings
>> were proposed. Do you think there is enough time for Java bindings to make
>> it into the 3.0 standard ?
> 
> I don't know about the 3.0 standard -  could happen, if I can do it fast enough and the
Forum accepts it for that release, or may have to follow in 3.1. The obstacle we have to overcome
re the Forum is that Java got a bad name in the early years of the binding attempts due to
performance issues and lack of attention to details. The performance "problem" largely stemmed
from the issue of binding processes to at least NUMA regions - the C implementations were
far faster - and the poor performance of Java in general during that time. The latter has
largely been resolved over the years, and the former is solvable with some work.
> 
> The detail issue reflected the problem of trying to create a single, non-sectarian set
of Java bindings that fit all MPI implementations. This meant that you could really only cover
90% of MPI functionality - beyond that, you have to integrate tightly to the implementation.
The academics who did the original work didn't want to do so, and thus left functions out,
resulting in the MPI community "looking down" on the result.
> 
> All put together, the MPI community wound up not thinking much of the Java world. As
I said, things have changed, and I believe a high-quality implementation of Java bindings
can gain acceptance. Once we have it for one MPI, we can (due to the OMPI license) offer it
up to the other implementations with a fair degree of confidence they will adopt it.
> 
> As for the MPI Forum, what we need is a "champion" to propose adoption of the bindings
once implemented. If people want them (i.e., the user community is larger than C++, which
has a total of 3 identified applications), we can show the implementation is of quality, and
we have developers willing to support it, then we can get them adopted.
> 
> I've scoped the job and it looks doable with reasonable effort. One other person on the
list (Deepak Sharma) has offered to help, and Jeff has offered to provide advice as he wrote
the original OMPI bindings. Getting it thru the OMPI devel approval represents a miniature
MPI Forum process, but I think we can do it given Jeff and my roles there.
> 
> HTH
> Ralph
> 
>> 
>> - Milind
>> 
>> On 12/1/11 3:31 AM, "Ralph Castain" <rhc@open-mpi.org> wrote:
>> 
>>> Hi folks
>>> 
>>> I'm a lead developer on the Open MPI project, and recently joined the
>>> Hadoop community to help with Hamster. A couple of people have asked me
>>> about using MPI more generally inside MapReduce, and it does indeed seem
>>> a good candidate to use that method of communication.
>>> 
>>> It seems to me, though, that a pre-requisite for moving that direction is
>>> a good set of Java MPI bindings. I've checked with the MPI community, and
>>> while there have been a couple of attempts at this, nothing really
>>> "adopted" by the community has been done.
>>> 
>>> I have some spare time while I search for a new job, so I thought I might
>>> tackle this over the next few weeks. However, I thought I would assess
>>> the actual level of interest before investing the time - it isn't a
>>> trivial thing to do, but doable if there is interest.
>>> 
>>> Thanks
>>> Ralph
>>> 
>>> 
>> 
> 


Mime
View raw message