hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Arup Malakar (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-3645) RCFileWriter does not implement the right function to support Federation
Date Thu, 15 Nov 2012 19:27:12 GMT

    [ https://issues.apache.org/jira/browse/HIVE-3645?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13498262#comment-13498262
] 

Arup Malakar commented on HIVE-3645:
------------------------------------


Looking at PIG-2791 looks like the following needs to be done:

1. Use getDefaultBlockSize(Path) and getDefaultReplication(Path) instead of getDefaultBlockSize()
and getDefaultReplication(). As the ones without Path argument wont work in case of federated
namenode. These methods need to 
be shimmed.
 
2. Bump hadoop dependency to 2.0.0-alpha as  getDefaultBlockSize(Path)/getDefaultReplication(Path)
are  not available in 0.23.1

                
> RCFileWriter does not implement the right function to support Federation
> ------------------------------------------------------------------------
>
>                 Key: HIVE-3645
>                 URL: https://issues.apache.org/jira/browse/HIVE-3645
>             Project: Hive
>          Issue Type: Bug
>          Components: Serializers/Deserializers
>    Affects Versions: 0.9.0, 0.10.0
>         Environment: Hadoop 0.23.3 federation, Hive 0.9 and Pig 0.10
>            Reporter: Viraj Bhat
>
> Create a table using Hive DDL
> {code}
> CREATE TABLE tmp_hcat_federated_numbers_part_1 (
>   id       int,  
>   intnum   int,
>   floatnum float
> )partitioned by (
>   part1    string,
>   part2    string
> )
> STORED AS rcfile
> LOCATION 'viewfs:///database/tmp_hcat_federated_numbers_part_1';
> {code}
> Populate it using Pig:
> {code}
> A = load 'default.numbers_pig' using org.apache.hcatalog.pig.HCatLoader();
> B = filter A by id <=  500;
> C = foreach B generate (int)id, (int)intnum, (float)floatnum;
> store C into
>         'default.tmp_hcat_federated_numbers_part_1'
>         using org.apache.hcatalog.pig.HCatStorer
>        ('part1=pig, part2=hcat_pig_insert',
>         'id: int,intnum: int,floatnum: float');
> {code}
> Generates the following error when running on a Federated Cluster:
> {quote}
> 2012-10-29 20:40:25,011 [main] ERROR
> org.apache.pig.tools.pigstats.SimplePigStats - ERROR 2997: Unable to recreate
> exception from backed error: AttemptID:attempt_1348522594824_0846_m_000000_3
> Info:Error: org.apache.hadoop.fs.viewfs.NotInMountpointException:
> getDefaultReplication on empty path is invalid
>         at
> org.apache.hadoop.fs.viewfs.ViewFileSystem.getDefaultReplication(ViewFileSystem.java:479)
>         at org.apache.hadoop.hive.ql.io.RCFile$Writer.<init>(RCFile.java:723)
>         at org.apache.hadoop.hive.ql.io.RCFile$Writer.<init>(RCFile.java:705)
>         at
> org.apache.hadoop.hive.ql.io.RCFileOutputFormat.getRecordWriter(RCFileOutputFormat.java:86)
>         at
> org.apache.hcatalog.mapreduce.FileOutputFormatContainer.getRecordWriter(FileOutputFormatContainer.java:100)
>         at
> org.apache.hcatalog.mapreduce.HCatOutputFormat.getRecordWriter(HCatOutputFormat.java:228)
>         at
> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigOutputFormat.getRecordWriter(PigOutputFormat.java:84)
>         at
> org.apache.hadoop.mapred.MapTask$NewDirectOutputCollector.<init>(MapTask.java:587)
>         at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:706)
>         at org.apache.hadoop.mapred.MapTask.run(MapTask.java:332)
>         at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:157)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at javax.security.auth.Subject.doAs(Subject.java:396)
>         at
> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1212)
>         at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:152)
> {quote}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message