pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Abhishek Agarwal (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (PIG-4680) Enable pig job graphs to resume from last successful state
Date Fri, 09 Oct 2015 13:26:26 GMT

     [ https://issues.apache.org/jira/browse/PIG-4680?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Abhishek Agarwal updated PIG-4680:
    Attachment: PIG-4680.patch

Here is a first cut implementation. I am in the process of adding tests. 

> Enable pig job graphs to resume from last successful state
> ----------------------------------------------------------
>                 Key: PIG-4680
>                 URL: https://issues.apache.org/jira/browse/PIG-4680
>             Project: Pig
>          Issue Type: Improvement
>          Components: impl
>            Reporter: Abhishek Agarwal
>            Assignee: Abhishek Agarwal
>         Attachments: PIG-4680.patch
> Pig scripts can have multiple ETL jobs in the DAG which may take hours to finish. In
case of transient errors, the job fails. When the job is rerun, all the nodes in Job graph
will rerun. Some of these nodes may have already run successfully. Redundant runs lead to
wastage of cluster capacity and pipeline delays. 
> In case of failure, we can persist the graph state. In next run, only the failed nodes
and their successors will rerun. This is of course subject to preconditions such as 
>  - Pig script has not changed
>  - Input locations have not changed
>  - Output data from previous run is intact
>  - Configuration has not changed

This message was sent by Atlassian JIRA

View raw message