pig-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Daniel Dai (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (PIG-4814) AvroStorage does not take namenode HA as part of schema file url
Date Fri, 19 Feb 2016 22:29:18 GMT

     [ https://issues.apache.org/jira/browse/PIG-4814?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Daniel Dai updated PIG-4814:
----------------------------
    Attachment: PIG-4814-1.patch

It might be hard to come out a test cuz the reproduction involve namenode HA setting.

> AvroStorage does not take namenode HA as part of schema file url
> ----------------------------------------------------------------
>
>                 Key: PIG-4814
>                 URL: https://issues.apache.org/jira/browse/PIG-4814
>             Project: Pig
>          Issue Type: Bug
>            Reporter: Daniel Dai
>            Assignee: Daniel Dai
>             Fix For: 0.16.0
>
>         Attachments: PIG-4814-1.patch
>
>
> The following script fail:
> {code}
> in = LOAD 'recordsOfStringArrays.avro' USING AvroStorage('', '-d -f hdfs://ml-ha/user/aeckstein/notower-rw/streamRunwaysInputSchema.avsc');
> {code}
> Where ml-ha is the configured HA name for namenode.
> The reason is AvroStorage creates a new Configuration object in the backend, which missing
the namenode HA configuration in hdfs-site.xml:
> {code}
> <property>
>   <name>dfs.client.failover.proxy.provider.ml-ha</name>
>   <value>org.apache.hadoop.hdfs.server.namenode.ha.ConfiguredFailoverProxyProvider</value>
> </property>
> {code}
> It should use the configuration passed from frontend rather than construct a new one.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message