hadoop-common-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (JIRA)" <j...@apache.org>
Subject [jira] Commented: (HADOOP-6483) Provide Hadoop as a Service based on standards
Date Fri, 08 Jan 2010 11:58:55 GMT

    [ https://issues.apache.org/jira/browse/HADOOP-6483?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12797989#action_12797989

Steve Loughran commented on HADOOP-6483:

This could be good, though the term "Standards" is a vague one, that could include things
like WS-RF, WS-Notification and WS-BaseFaults if you aren't careful, things best left alone.
I write as someone who has implemented all of these and worked on three different SOAP stacks.

# Clearly, I need to know more. Are you just planning POSTed JSDL or something else?
# This should be a contrib/ in mapreduce or something on its own
# see also my slides on "Mombasa", the long-haul way to meet elephants [http://www.slideshare.net/steve_l/long-haul-hadoop]
. It notes there that job submission is independent of the work; JSDL can be used for this;
it's batch-ness does match the way the JobTracker works

> Provide Hadoop as a Service based on standards
> ----------------------------------------------
>                 Key: HADOOP-6483
>                 URL: https://issues.apache.org/jira/browse/HADOOP-6483
>             Project: Hadoop Common
>          Issue Type: New Feature
>            Reporter: Yang Zhou
> Hadoop as a Service provides a standards-based web services interface that layers on
top of Hadoop on Demand and allows Hadoop jobs to be submitted via popular schedulers, such
as Sun Grid Engine (SGE), Platform LSF, Microsoft HPC Server 2008 etc., to local or remote
Hadoop clusters.  This allows multiple Hadoop clusters within an organization to be efficiently
shared and provides flexibility, allowing remote Hadoop clusters, offered as Cloud services,
to be used for experimentation and burst capacity. HaaS hides complexity, allowing users to
submit many types of compute or data intensive work via a single scheduler without actually
knowing where it will be done. Additionally providing a standards-based front-end to Hadoop
means that users would be able to easily choose HaaS providers without being locked in, i.e.
via proprietary interfaces such as Amazon's map/reduce service.  
> Our HaaS implementation uses the OGF High Performance Computing Basic Profile standard
to define interoperable job submission descriptions and management interfaces to Hadoop. It
uses Hadoop on Demand to provision capacity. Our HaaS implementation also supports files stage
in/out with protocols like FTP, SCP and GridFTP.
> Our HaaS implementation also provides a suit of RESTful interface which  compliant with

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message