Return-Path: 
 <core-dev-return-63136-apmail-hadoop-core-dev-archive=hadoop.apache.org@hadoop.apache.org>
Delivered-To: apmail-hadoop-core-dev-archive@www.apache.org
Received: (qmail 91688 invoked from network); 27 Feb 2009 11:56:40 -0000
Received: from hermes.apache.org (HELO mail.apache.org) (140.211.11.2)
  by minotaur.apache.org with SMTP; 27 Feb 2009 11:56:40 -0000
Received: (qmail 53703 invoked by uid 500); 27 Feb 2009 11:56:38 -0000
Delivered-To: apmail-hadoop-core-dev-archive@hadoop.apache.org
Received: (qmail 53273 invoked by uid 500); 27 Feb 2009 11:56:36 -0000
Mailing-List: contact core-dev-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:core-dev-help@hadoop.apache.org>
List-Unsubscribe: <mailto:core-dev-unsubscribe@hadoop.apache.org>
List-Post: <mailto:core-dev@hadoop.apache.org>
List-Id: <core-dev.hadoop.apache.org>
Reply-To: core-dev@hadoop.apache.org
Delivered-To: mailing list core-dev@hadoop.apache.org
Received: (qmail 53262 invoked by uid 99); 27 Feb 2009 11:56:36 -0000
Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 Feb 2009 03:56:36 -0800
X-ASF-Spam-Status: No, hits=-2000.0 required=10.0
	tests=ALL_TRUSTED
X-Spam-Check-By: apache.org
Received: from [140.211.11.140] (HELO brutus.apache.org) (140.211.11.140)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 27 Feb 2009 11:56:34 +0000
Received: from brutus (localhost [127.0.0.1])
	by brutus.apache.org (Postfix) with ESMTP id 2C4E9234C4C0
	for <core-dev@hadoop.apache.org>; Fri, 27 Feb 2009 03:56:13 -0800 (PST)
Message-ID: <541411768.1235735773180.JavaMail.jira@brutus>
Date: Fri, 27 Feb 2009 03:56:13 -0800 (PST)
From: "Steve Loughran (JIRA)" <jira@apache.org>
To: core-dev@hadoop.apache.org
Subject: [jira] Commented: (HADOOP-5303) Hadoop Workflow System (HWS)
In-Reply-To: <1076780325.1235274181793.JavaMail.jira@brutus>
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-Virus-Checked: Checked by ClamAV on apache.org


    [ https://issues.apache.org/jira/browse/HADOOP-5303?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12677333#action_12677333 ] 

Steve Loughran commented on HADOOP-5303:
----------------------------------------


I'd be against embedding stuff and playing xmlns nesting tricks the way you've suggested.

{code}
<action name="myPigjob">
  <pig xmlns="hws:pig:1">
  ...
  </pig>
</action>
{code}

# doesn't take long before you end up in XSD-Hell 
# stops tools being able to understand the inputs and outputs of custom actions. Which is useful if you are trying to decide where to run things, how to schedule different workflows over space and time.

{code}
<action name="myPigjob" type="pig">
  <task name="pig" >
   <inputs>
     <dataset url="s3:/traffic/oystercard/london/2009" />
     <dataset url="s3:/traffic/anpr/london/2009/" />
     <file path="/users/steve/pig/car-vs-tube.pig" />
   </inputs>
   <option name="limit" value="40000" />
   <option name="pigfile" value="/users/steve/pig/car-vs-tube.pig" />
   <option name="startdate" value="2009-01-01" />
   <option name="enddate" value="2009-02-28" />
  <outputs>
    <dataset url="hdfs://traffic/anpr/london" />
  </outputs>
  </task>
</action>
{code}


Its uglier, but having file inputs and outputs explicit could make high level workload scheduling much easier, especially once you start deciding which rack to create VMs on 

> Hadoop Workflow System (HWS)
> ----------------------------
>
>                 Key: HADOOP-5303
>                 URL: https://issues.apache.org/jira/browse/HADOOP-5303
>             Project: Hadoop Core
>          Issue Type: New Feature
>            Reporter: Alejandro Abdelnur
>            Assignee: Alejandro Abdelnur
>         Attachments: hws-preso-v1_0_2009FEB22.pdf, hws-v1_0_2009FEB22.pdf
>
>
> This is a proposal for a system specialized in running Hadoop/Pig jobs in a control dependency DAG (Direct Acyclic Graph), a Hadoop workflow application.
> Attached there is a complete specification and a high level overview presentation.
> ----
> *Highlights* 
> A Workflow application is DAG that coordinates the following types of actions: Hadoop, Pig, Ssh, Http, Email and sub-workflows. 
> Flow control operations within the workflow applications can be done using decision, fork and join nodes. Cycles in workflows are not supported.
> Actions and decisions can be parameterized with job properties, actions output (i.e. Hadoop counters, Ssh key/value pairs output) and file information (file exists, file size, etc). Formal parameters are expressed in the workflow definition as {{${VAR}}} variables.
> A Workflow application is a ZIP file that contains the workflow definition (an XML file), all the necessary files to run all the actions: JAR files for Map/Reduce jobs, shells for streaming Map/Reduce jobs, native libraries, Pig scripts, and other resource files.
> Before running a workflow job, the corresponding workflow application must be deployed in HWS.
> Deploying workflow application and running workflow jobs can be done via command line tools, a WS API and a Java API.
> Monitoring the system and workflow jobs can be done via a web console, command line tools, a WS API and a Java API.
> When submitting a workflow job, a set of properties resolving all the formal parameters in the workflow definitions must be provided. This set of properties is a Hadoop configuration.
> Possible states for a workflow jobs are: {{CREATED}}, {{RUNNING}}, {{SUSPENDED}}, {{SUCCEEDED}}, {{KILLED}} and {{FAILED}}.
> In the case of a action failure in a workflow job, depending on the type of failure, HWS will attempt automatic retries, it will request a manual retry or it will fail the workflow job.
> HWS can make HTTP callback notifications on action start/end/failure events and workflow end/failure events.
> In the case of workflow job failure, the workflow job can be resubmitted skipping previously completed actions. Before doing a resubmission the workflow application could be updated with a patch to fix a problem in the workflow application code.
> ----

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.