pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Cheolsoo Park (JIRA)" <j...@apache.org>
Subject [jira] [Resolved] (PIG-3400) FS commands do not work with S3 paths
Date Sun, 07 Sep 2014 03:36:28 GMT

     [ https://issues.apache.org/jira/browse/PIG-3400?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Cheolsoo Park resolved PIG-3400.
--------------------------------
    Resolution: Won't Fix

> FS commands do not work with S3 paths
> -------------------------------------
>
>                 Key: PIG-3400
>                 URL: https://issues.apache.org/jira/browse/PIG-3400
>             Project: Pig
>          Issue Type: Bug
>          Components: grunt
>            Reporter: Cheolsoo Park
>            Assignee: Cheolsoo Park
>         Attachments: PIG-3400-2.patch, PIG-3400-3.patch, PIG-3400.patch
>
>
> To reproduce issues, run the following commands w/ S3 paths:
> {code}
> pig -e 'ls s3://<path>'
> pig -e 'mkdir s3://<path>'
> pig -e 'cp s3://<path1> s3://<path2>'
> pig -e 'mv s3://<path1> s3://<path2>'
> pig -e 'copyToLocal s3://<path>/<file> .'
> pig -e 'copyFromLocal <file> s3://<path>'
> {code}
> As of now, none of these commands works in Apache Pig, whereas they do in EMR Pig. The
problem is that in GruntParser, DataStorage is constructed using the default file system provided
by configuration, and s3 paths are not recognized.
> Instead, we should construct DataStorage based on the given URL. For example,
> {code}
> DataStorage dfs = new HDataStorage(new Path(path).toUri(), mConf);
> {code}



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message