hadoop-yarn-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Loughran (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (YARN-1564) add some basic workflow YARN services
Date Tue, 03 Jun 2014 16:03:02 GMT

     [ https://issues.apache.org/jira/browse/YARN-1564?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Steve Loughran updated YARN-1564:

    Attachment: YARN-1564-001.patch

Here is a package containing the set of workflow yarn services we've been using, and tests.
They are described in the javadocs and package-info.java; the key ones are:

# {{WorkflowCompositeService}}: stops when a child stops; propagates up failures.
# {{WorkflowSequenceService}}: starts its children in sequence, stopping when all have been
run, or after the first failure.
# {{WorkflowForkedProcessService}} fork a potentially long-lived process; matching the lifecycle
of the YARN service to the process. This logs output, caching a few recent lines, rather than
buffers it -for that long-lived execution.
# {{WorkflowCallbackService}} executes any {{Callable}} (in java 8, a lambda) when started,
stopping the service when it finishes, & propagating exceptions. Lets you run anything
in a workflow, or just add new YARN services without subclassing. 

I've been using these for about 9-12 months, building my YARN apps from them, so I consider
them fairly mature in terms of functionality.

Note also that the {{ServiceParent extends Service }} interface offering public access to
child services *is* effectively implemented by the YARN {{CompositeService}} -all we need
to do is explicitly implement the interface and turn the two methods public.

  void addService(Service service);
  List<Service> getServices();

> add some basic workflow YARN services
> -------------------------------------
>                 Key: YARN-1564
>                 URL: https://issues.apache.org/jira/browse/YARN-1564
>             Project: Hadoop YARN
>          Issue Type: New Feature
>          Components: api
>    Affects Versions: 2.4.0
>            Reporter: Steve Loughran
>            Assignee: Steve Loughran
>            Priority: Minor
>         Attachments: YARN-1564-001.patch
>   Original Estimate: 24h
>  Remaining Estimate: 24h
> I've been using some alternative composite services to help build workflows of process
execution in a YARN AM.
> They and their tests could be moved in YARN for the use by others -this would make it
easier to build aggregate services in an AM

This message was sent by Atlassian JIRA

View raw message