hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ralph Castain (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-2911) Hamster: Hadoop And Mpi on the same cluSTER
Date Fri, 25 Nov 2011 16:41:40 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-2911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13157232#comment-13157232

Ralph Castain commented on MAPREDUCE-2911:

Ah - my bad. I didn't realize I was looking at the comments in reverse order :-)

After reading the comments in the correct order, I now better understand the thread and see
that Milind is following what I had suggested. As to the discussion of secure communications,
this is a continuing issue in the MPI community. The problem is that securing at the message
level creates considerable overhead and severely impacts MPI performance.

What the community has chosen to do is secure at the user level, and then check socket connections
to ensure we are talking to someone from within our own application. Thus, we launch based
on ssh-like authentication requirements. During MPI_Init, we wireup socket connections. As
each connection is made, we exchange an initial "ident" message that checks to ensure that
the process on the other end is a member of our application. If it isn't, we drop the connection.

If you want to add further security during the socket formation phase, nobody will object
- though we might put it on a configuration basis so others aren't impacted as it will slow
down launch times on very large clusters.


> Hamster: Hadoop And Mpi on the same cluSTER
> -------------------------------------------
>                 Key: MAPREDUCE-2911
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2911
>             Project: Hadoop Map/Reduce
>          Issue Type: New Feature
>          Components: mrv2
>    Affects Versions: 0.23.0
>         Environment: All Unix-Environments
>            Reporter: Milind Bhandarkar
>            Assignee: Milind Bhandarkar
>             Fix For: 0.24.0
>   Original Estimate: 336h
>  Remaining Estimate: 336h
> MPI is commonly used for many machine-learning applications. OpenMPI (http://www.open-mpi.org/)
is a popular BSD-licensed version of MPI. In the past, running MPI application on a Hadoop
cluster was achieved using Hadoop Streaming (http://videolectures.net/nipsworkshops2010_ye_gbd/),
but it was kludgy. After the resource-manager separation from JobTracker in Hadoop, we have
all the tools needed to make MPI a first-class citizen on a Hadoop cluster. I am currently
working on the patch to make MPI an application-master. Initial version of this patch will
be available soon (hopefully before September 10.) This jira will track the development of
Hamster: The application master for MPI.

This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira


View raw message