hadoop-pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Pradeep Kamath (JIRA)" <j...@apache.org>
Subject [jira] Updated: (PIG-1265) Change LoadMetadata and StoreMetadata to use Job instead of Configuraiton and add a cleanupOnFailure method to StoreFuncInterface
Date Sat, 27 Feb 2010 00:10:05 GMT

     [ https://issues.apache.org/jira/browse/PIG-1265?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Pradeep Kamath updated PIG-1265:
--------------------------------

    Assignee: Pradeep Kamath  (was: Pradeep Kamath)
      Status: Patch Available  (was: Open)

> Change LoadMetadata and StoreMetadata to use Job instead of Configuraiton and add a cleanupOnFailure
method to StoreFuncInterface
> ---------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: PIG-1265
>                 URL: https://issues.apache.org/jira/browse/PIG-1265
>             Project: Pig
>          Issue Type: Bug
>    Affects Versions: 0.7.0
>            Reporter: Pradeep Kamath
>            Assignee: Pradeep Kamath
>             Fix For: 0.7.0
>
>         Attachments: PIG-1265.patch
>
>
> Speaking to the hadoop team folks, the direction in hadoop is to use Job instead of Configuration
- for example InputFormat/OutputFormat implementations use Job to store input/output location.
So pig should also do the same in LoadMetadata and StoreMetadata to be closer to hadoop.
> Currently when a job fails, pig assumes the output locations (corresponding to the stores
in the job) are hdfs locations and attempts to delete them. Since output locations could be
non hdfs locations, this cleanup should be delegated to the StoreFuncInterface implementation
- hence a new method - cleanupOnFailure() should be introduced in StoreFuncInterface and a
default implementation should be provided in the StoreFunc abstract class which checks if
the location exists on hdfs and deletes it if so.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message