hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Luke Lu (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-2911) Hamster: Hadoop And Mpi on the same cluSTER
Date Wed, 31 Aug 2011 20:10:11 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-2911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13094859#comment-13094859
] 

Luke Lu commented on MAPREDUCE-2911:
------------------------------------

{quote}
the problem with mpiexec is it's license:

"Mpiexec is free software and is licensed for use under the GNU General Public License, version
2."

OpenMPI, on the other hand, is BSD-licensed, and implements the MPI-2 standard.
{quote}

You're confusing mpiexec the software from OSC with mpiexec the standard specified in MPI
standard (MPI-2.2 section 8). OpenMPI includes an mpiexec executable, as many other implementations
do.

bq. what they do in the user-code is equivalent to what users can do in map-reduce code, so
I do not see an issue here.

They're not equivalent, as normal mapreduce tasks's communication cannot be attack by other
users. The Open MPI implementation is somewhat equivalent to a user implementing extra insecure
protocols in map/reduce tasks in addition to the standard secure hadoop mapreduce protocols.

> Hamster: Hadoop And Mpi on the same cluSTER
> -------------------------------------------
>
>                 Key: MAPREDUCE-2911
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2911
>             Project: Hadoop Map/Reduce
>          Issue Type: New Feature
>          Components: mrv2
>    Affects Versions: 0.23.0
>         Environment: All Unix-Environments
>            Reporter: Milind Bhandarkar
>            Assignee: Milind Bhandarkar
>             Fix For: 0.23.0
>
>   Original Estimate: 336h
>  Remaining Estimate: 336h
>
> MPI is commonly used for many machine-learning applications. OpenMPI (http://www.open-mpi.org/)
is a popular BSD-licensed version of MPI. In the past, running MPI application on a Hadoop
cluster was achieved using Hadoop Streaming (http://videolectures.net/nipsworkshops2010_ye_gbd/),
but it was kludgy. After the resource-manager separation from JobTracker in Hadoop, we have
all the tools needed to make MPI a first-class citizen on a Hadoop cluster. I am currently
working on the patch to make MPI an application-master. Initial version of this patch will
be available soon (hopefully before September 10.) This jira will track the development of
Hamster: The application master for MPI.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message