hadoop-mapreduce-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vinod Kumar Vavilapalli (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (MAPREDUCE-2911) Hamster: Hadoop And Mpi on the same cluSTER
Date Tue, 30 Aug 2011 15:51:38 GMT

    [ https://issues.apache.org/jira/browse/MAPREDUCE-2911?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13093838#comment-13093838
] 

Vinod Kumar Vavilapalli commented on MAPREDUCE-2911:
----------------------------------------------------

This is a great addition! +1 for the simple design.

I guess the client spins till all the containers are allocated (Oh it's HOD all over again
:) )

Like MR, the client could continuously ping the AM via RPC to know about the status of container-allocations.
Presence/absence of HDFS file sure is a simple beginning.

bq. Also, if any of the container (running orte), exits abnormally, entire virtual MPI cluster
is terminated. (This limitation will be removed in the next version.)
How does the mpi client get to know about this? It'd be great if mpi-run automatically detects
this.

Security can be postponed for the first-cut.

For now, +1 to hadoop-openmpi module under hadoop-mapreduce-project. Once we have this, we
will have a platform(yarn) and two frameworks(MR and MPI) using it. Then we can move out yarn
out of mapreduce-project, mapreduce and MPI can move into a frameworks aggregation module
(the app-store, gawd, I can't believe I just said that :) )

> Hamster: Hadoop And Mpi on the same cluSTER
> -------------------------------------------
>
>                 Key: MAPREDUCE-2911
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2911
>             Project: Hadoop Map/Reduce
>          Issue Type: New Feature
>          Components: mrv2
>    Affects Versions: 0.23.0
>         Environment: All Unix-Environments
>            Reporter: Milind Bhandarkar
>            Assignee: Milind Bhandarkar
>             Fix For: 0.23.0
>
>   Original Estimate: 336h
>  Remaining Estimate: 336h
>
> MPI is commonly used for many machine-learning applications. OpenMPI (http://www.open-mpi.org/)
is a popular BSD-licensed version of MPI. In the past, running MPI application on a Hadoop
cluster was achieved using Hadoop Streaming (http://videolectures.net/nipsworkshops2010_ye_gbd/),
but it was kludgy. After the resource-manager separation from JobTracker in Hadoop, we have
all the tools needed to make MPI a first-class citizen on a Hadoop cluster. I am currently
working on the patch to make MPI an application-master. Initial version of this patch will
be available soon (hopefully before September 10.) This jira will track the development of
Hamster: The application master for MPI.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message