hadoop-pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Alan Gates (JIRA)" <j...@apache.org>
Subject [jira] Updated: (PIG-145) LocalFile ignores active container and HDataStorage can't copy to other DataStrorage
Date Sat, 29 Mar 2008 19:19:24 GMT

     [ https://issues.apache.org/jira/browse/PIG-145?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Alan Gates updated PIG-145:

    Status: Open  (was: Patch Available)

Your changes broke the semantics of copyToLocal in grunt.  Previously, if a pig job stored
the results to a file (ie store a to 'myfile';) then myfile is created as a directory.  In
grunt, a call to copyToLocal on that directory would then copy the directory and it's files
out of the dfs.  Now, that call in grunt results in:

grunt> copyToLocal /user/pig/tests/results/gates.1206814491/Checkin_1.out bla
2008-03-29 12:17:15,115 [main] ERROR org.apache.pig.tools.grunt.GruntParser - java.lang.ClassCastException:
org.apache.pig.backend.hadoop.datastorage.HFile        at org.apache.pig.backend.hadoop.datastorage.HDirectory.copy(HDirectory.java:80)
       at org.apache.pig.backend.hadoop.datastorage.HPath.copy(HPath.java:111)        at org.apache.pig.tools.grunt.GruntParser.processCopyToLocal(GruntParser.java:421)
       at org.apache.pig.tools.pigscript.parser.PigScriptParser.parse(PigScriptParser.java:155)
       at org.apache.pig.tools.grunt.GruntParser.parseContOnError(GruntParser.java:73)   
    at org.apache.pig.tools.grunt.Grunt.run(Grunt.java:54)        at org.apache.pig.Main.main(Main.java:253)

> LocalFile ignores active container and HDataStorage can't copy to other DataStrorage
> ------------------------------------------------------------------------------------
>                 Key: PIG-145
>                 URL: https://issues.apache.org/jira/browse/PIG-145
>             Project: Pig
>          Issue Type: Bug
>          Components: impl
>            Reporter: Charlie Groves
>         Attachments: PIG-145-DataStorage_Bugs.patch
> As part of starting to rewrite the DataStorage APIs, I wrote some unit tests for the
existing DataStorage implementations to make sure I wasn't breaking anything.  In testing
the open code, I found that LocalFile doesn't respect the active container you set you LocalDataStorage,
so if you open a relative file, it's relative to wherever you're running the code.  Similarly,
while testing the copy operations, I found that HFile doesn't allow copying to anything other
than other HFiles and that HDirectory's copy operation was never used because it had the wrong
> The attached patch fixes these issues and adds tests for much of the DataStorage API
for both of the existing backends.  There are no tests for the sopen code as I'm planning
on changing that significantly in rewriting these.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message