hadoop-common-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Apache Wiki <wikidi...@apache.org>
Subject [Hadoop Wiki] Update of "BristolHadoopWorkshop" by SteveLoughran
Date Thu, 13 Aug 2009 13:54:38 GMT
Dear Wiki user,

You have subscribed to a wiki page or wiki category on "Hadoop Wiki" for change notification.

The following page has been changed by SteveLoughran:
http://wiki.apache.org/hadoop/BristolHadoopWorkshop

The comment on the change is:
write up long-haul hadoop

------------------------------------------------------------------------------
  
  When you bring up a cluster, even if every service has been asked to see if it is healthy,
they still have the problem of talking to everything. The best check: push work through the
system. Wait for things to fail, try and guess the problem. Having work to push through that
is designed to stress the system's interconnected -and whose failure can be diagnosed with
ease- would be nice.
  
- That is, for all those people asking for a !HappyHadoop JPS page, it isn't enough. A cluster
may cope with some of the workers going down, but it is not actually functional unless every
node that is up can talk to every other node that is up, that nothing is coming up listening
on IPv6, that the TaskTracker hasn't decided to only run on localhost, etc. etc.
+ That is, for all those people asking for a !HappyHadoop JSP page, it isn't enough. A cluster
may cope with some of the workers going down, but it is not actually functional unless every
node that is up can talk to every other node that is up, that nothing is coming up listening
on IPv6, that the TaskTracker hasn't decided to only run on localhost, etc. etc.
  
+ == Long-Haul Hadoop ==
+ 
+ This talk discussed the notion of a long-haul interface to Hadoop. 
+ 
+ This is a recurrent theme in various bug reports -anywhere where people
+ want to submit jobs from a distance and keep an eye on them. Often this
+ need surfaces in a request for some kind of remote API to the Job
+ Tracker.
+ 
+ This talk discussed the fact that remote Job Tracker API is not
+ sufficient. Being able to submit one job is nice, but if you are
+ chaining things together, you need to submit a sequence of operations,
+ and the logic to chain it together. You may also want things like
+ notifications of progress/failure. What you have is a workflow.
+ 
+ Workflows could be handled at the low level with something that can run
+ a sequence of MR jobs, and any Java classes which implement the Tool
+ interface. The Tool would be run in the datacentre, in some
+ medium-availability host, so you could switch your laptop off and know
+ that the program was still running. 
+ 
+ There is work underway at Yahoo! with Oozie, a workflow system for
+ Hadoop; Cascading and Pig Latin are also languages to describe
+ sequences of operations, so again, you need to run them. In the talk,
+ SmartFrog was used as a constraint-language for ordering work, which
+ shows that the language set is even broader. But there is a key point
+ here: the model of a workflow engine is the same, regardless of the
+ language. You create workflows, you schedule them with parameters and
+ options, you await results.
+ 
+ The long-haul model for workflow can be the same for multiple back ends.
+ 
+ The talk looked at the options for long-haul-ness, of which there are
+ two
+ 
+ WS-* : the big, comfortable, safe long-haul option, the Airbus A380.
+ You, the passenger, get looked after by the cabin crew. 
+ 
+ The floatplane. Agile, can get around fast, but you read the location of
+ the life vest instructions very carefully, make a note of the exit in
+ the roof and hope that you aren't the one who has to get on the float to
+ dock the plane with the boat. It isn't quite as comfy as the big plane,
+ but it is easier to get up and around with it.
+ 
+ Two RESTful world views were discussed
+ 
+  * A pure REST: PUT/DELETE model of workflow objects, in which even their queue state is
manipulated using the full REST model. This is clean, ideal for clients such as Restlet, and
HTML5 browsers. 
+  * An HTTP Post model, in which work is POSTed to a queue server, URLs returned; operations
to the queued workflows via POST or PUT, GET for state updates.
+ 
+ Steve gave a partial demonstration of Mombasa, his prototype
+ "long-haul route to the elephants". This consists of:
+ 
+  * A RESTy interface built from JAX-RS, hosted as the Jersey runtime under Jetty, deployed
in-datacentre by SmartFrog
+    
+  * A Portlet GUI to the same set of operations, this time running in-datacentre in a portlet
server. (Which may be liferay-on-tomcat, but
+  does not need to be). It is implicitly implementing the HTTP Post  model.
+  
+ Currently the portlet is not using the long-haul API itself, though
+ there is no reason why it should not, in which case it will not only
+ drive the API, it will test it. 
+ 
+ Other Portlets will apparently provide cluster management by talking to
+ the relevant "cloud" APIs: Add/decommission nodes, view logs, etc, and
+ simple HDFS file access.
+ 
+ Long-haul filesystem access is another issue. Ideally, WebDAV would be
+ good, as there are so many clients and it is a pure REST API. 
+ But parts of the WebDAV spec are odd (same FS semantics as Win98/FAT),
+ and you can be sure of interop grief. Amazon S3 is simpler, as long as
+ you avoid their daft authentication mechanism. 
+ 

Mime
View raw message