Mailing-List: contact mapreduce-issues-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: mapreduce-issues@hadoop.apache.org
Date: Fri, 25 Nov 2011 16:41:40 +0000 (UTC)
From: "Ralph Castain (Commented) (JIRA)" <jira@apache.org>
To: mapreduce-issues@hadoop.apache.org
Message-ID: 
 <615375295.12305.1322239300046.JavaMail.tomcat@hel.zones.apache.org>
In-Reply-To: 
 <885977775.6304.1314687877760.JavaMail.tomcat@hel.zones.apache.org>
Subject: [jira] [Commented] (MAPREDUCE-2911) Hamster: Hadoop And Mpi on the
 same cluSTER
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit


    [ https://issues.apache.org/jira/browse/MAPREDUCE-2911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13157232#comment-13157232 ] 

Ralph Castain commented on MAPREDUCE-2911:
------------------------------------------

Ah - my bad. I didn't realize I was looking at the comments in reverse order :-)

After reading the comments in the correct order, I now better understand the thread and see that Milind is following what I had suggested. As to the discussion of secure communications, this is a continuing issue in the MPI community. The problem is that securing at the message level creates considerable overhead and severely impacts MPI performance.

What the community has chosen to do is secure at the user level, and then check socket connections to ensure we are talking to someone from within our own application. Thus, we launch based on ssh-like authentication requirements. During MPI_Init, we wireup socket connections. As each connection is made, we exchange an initial "ident" message that checks to ensure that the process on the other end is a member of our application. If it isn't, we drop the connection.

If you want to add further security during the socket formation phase, nobody will object - though we might put it on a configuration basis so others aren't impacted as it will slow down launch times on very large clusters.

HTH
Ralph

                
> Hamster: Hadoop And Mpi on the same cluSTER
> -------------------------------------------
>
>                 Key: MAPREDUCE-2911
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2911
>             Project: Hadoop Map/Reduce
>          Issue Type: New Feature
>          Components: mrv2
>    Affects Versions: 0.23.0
>         Environment: All Unix-Environments
>            Reporter: Milind Bhandarkar
>            Assignee: Milind Bhandarkar
>             Fix For: 0.24.0
>
>   Original Estimate: 336h
>  Remaining Estimate: 336h
>
> MPI is commonly used for many machine-learning applications. OpenMPI (http://www.open-mpi.org/) is a popular BSD-licensed version of MPI. In the past, running MPI application on a Hadoop cluster was achieved using Hadoop Streaming (http://videolectures.net/nipsworkshops2010_ye_gbd/), but it was kludgy. After the resource-manager separation from JobTracker in Hadoop, we have all the tools needed to make MPI a first-class citizen on a Hadoop cluster. I am currently working on the patch to make MPI an application-master. Initial version of this patch will be available soon (hopefully before September 10.) This jira will track the development of Hamster: The application master for MPI.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira