pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Akihiro Matsukawa (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (PIG-3552) UriUtil used by reducer estimator should support viewfs
Date Thu, 31 Oct 2013 21:51:27 GMT

     [ https://issues.apache.org/jira/browse/PIG-3552?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Akihiro Matsukawa updated PIG-3552:
-----------------------------------

    Attachment:     (was: UriUtil_enable_viewfs.patch)

> UriUtil used by reducer estimator should support viewfs
> -------------------------------------------------------
>
>                 Key: PIG-3552
>                 URL: https://issues.apache.org/jira/browse/PIG-3552
>             Project: Pig
>          Issue Type: Bug
>          Components: impl
>         Environment: Pig 0.11+
>            Reporter: Akihiro Matsukawa
>            Assignee: Akihiro Matsukawa
>         Attachments: UriUtil_enable_viewfs.patch, UriUtil_tests.patch
>
>
> On Hadoop2 running viewfs throws error 
> {code}
> 2013-10-30 02:12:46,916 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
- Using reducer estimator: org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.InputSizeReducerEstimator
> 2013-10-30 02:12:46,917 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.InputSizeReducerEstimator
- BytesPerReducer=1610612736 maxReducers=999 totalInputFileSize=-1
> 2013-10-30 02:12:46,917 [main] INFO  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.JobControlCompiler
- Could not estimate number of reducers and no requested or default parallelism set. Defaulting
to 1 reducer.
> {code}
> This is caused by the fact that UriUtil.isHDFSFileOrLocalOrS3N only supports files that
start with "/", "hdfs:", "file:", "s3n:" but not "viewfs:".
> Trivial fix, patch attached. 
> Testing: ant test-commit passes.



--
This message was sent by Atlassian JIRA
(v6.1#6144)

Mime
View raw message