hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Milind Bhandarkar (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-2911) Hamster: Hadoop And Mpi on the same cluSTER
Date Thu, 17 May 2012 16:00:14 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-2911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13277916#comment-13277916
] 

Milind Bhandarkar commented on MAPREDUCE-2911:
----------------------------------------------

I am excited to report that, thanks to great efforts by Ralph Castain and Wangda Tan, Hamster
(i.e. OpenMPI on Yarn) now works flawlessly, and is scheduled to be merged to OpenMPI trunk
soon. This effort was equivalent to building a second floor on a mobile home while it was
hurtling down the freeway at 65 MPH :-) Thanks to both Ralph & Wangda.

According to Ralph:

"Lots of cleanup and documentation to do, and performance sucks per HPC
standards. But at least it works!"

To my knowledge, this is the first application framework implemented in C that uses the multi-lingual
protobuf APIs for Yarn. (For secure environments, a small java-based shim is needed.)

Also, it is encouraging that no changes were needed in Yarn to make resource allocation work
for MPI. (MPI as a standard came along in 1994, 18 years before Yarn was designed.)

Currently, using MPI-IO functionality in MPI requires a shared posix  file-system mounted
on every node. However, this will change in future. For some distributed file systems (*cough*),
which offer posix interface, MPI-IO works today.

Once it is decided whether BigTop can include Non-ASF packages, we plan to work with BigTop
community to integrate OpenMPI (new BSD-licensed) in the big data stack.

I am closing this issue as fixed.
                
> Hamster: Hadoop And Mpi on the same cluSTER
> -------------------------------------------
>
>                 Key: MAPREDUCE-2911
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2911
>             Project: Hadoop Map/Reduce
>          Issue Type: New Feature
>          Components: mrv2
>    Affects Versions: 0.23.0
>         Environment: All Unix-Environments
>            Reporter: Milind Bhandarkar
>            Assignee: Ralph H Castain
>             Fix For: 0.24.0
>
>   Original Estimate: 336h
>  Remaining Estimate: 336h
>
> MPI is commonly used for many machine-learning applications. OpenMPI (http://www.open-mpi.org/)
is a popular BSD-licensed version of MPI. In the past, running MPI application on a Hadoop
cluster was achieved using Hadoop Streaming (http://videolectures.net/nipsworkshops2010_ye_gbd/),
but it was kludgy. After the resource-manager separation from JobTracker in Hadoop, we have
all the tools needed to make MPI a first-class citizen on a Hadoop cluster. I am currently
working on the patch to make MPI an application-master. Initial version of this patch will
be available soon (hopefully before September 10.) This jira will track the development of
Hamster: The application master for MPI.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message