hadoop-pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hadoop QA (JIRA)" <j...@apache.org>
Subject [jira] Commented: (PIG-1265) Change LoadMetadata and StoreMetadata to use Job instead of Configuraiton and add a cleanupOnFailure method to StoreFuncInterface
Date Tue, 02 Mar 2010 01:08:05 GMT

    [ https://issues.apache.org/jira/browse/PIG-1265?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12839946#action_12839946
] 

Hadoop QA commented on PIG-1265:
--------------------------------

-1 overall.  Here are the results of testing the latest attachment 
  http://issues.apache.org/jira/secure/attachment/12437515/PIG-1265-2.patch
  against trunk revision 916793.

    +1 @author.  The patch does not contain any @author tags.

    +1 tests included.  The patch appears to include 17 new or modified tests.

    +1 javadoc.  The javadoc tool did not generate any warning messages.

    +1 javac.  The applied patch does not increase the total number of javac compiler warnings.

    +1 findbugs.  The patch does not introduce any new Findbugs warnings.

    -1 release audit.  The applied patch generated 520 release audit warnings (more than the
trunk's current 519 warnings).

    +1 core tests.  The patch passed core unit tests.

    +1 contrib tests.  The patch passed contrib unit tests.

Test results: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/217/testReport/
Release audit warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/217/artifact/trunk/patchprocess/releaseAuditDiffWarnings.txt
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/217/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Console output: http://hudson.zones.apache.org/hudson/job/Pig-Patch-h7.grid.sp2.yahoo.net/217/console

This message is automatically generated.

> Change LoadMetadata and StoreMetadata to use Job instead of Configuraiton and add a cleanupOnFailure
method to StoreFuncInterface
> ---------------------------------------------------------------------------------------------------------------------------------
>
>                 Key: PIG-1265
>                 URL: https://issues.apache.org/jira/browse/PIG-1265
>             Project: Pig
>          Issue Type: Bug
>    Affects Versions: 0.7.0
>            Reporter: Pradeep Kamath
>            Assignee: Pradeep Kamath
>             Fix For: 0.7.0
>
>         Attachments: PIG-1265-2.patch, PIG-1265.patch
>
>
> Speaking to the hadoop team folks, the direction in hadoop is to use Job instead of Configuration
- for example InputFormat/OutputFormat implementations use Job to store input/output location.
So pig should also do the same in LoadMetadata and StoreMetadata to be closer to hadoop.
> Currently when a job fails, pig assumes the output locations (corresponding to the stores
in the job) are hdfs locations and attempts to delete them. Since output locations could be
non hdfs locations, this cleanup should be delegated to the StoreFuncInterface implementation
- hence a new method - cleanupOnFailure() should be introduced in StoreFuncInterface and a
default implementation should be provided in the StoreFunc abstract class which checks if
the location exists on hdfs and deletes it if so.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.


Mime
View raw message