hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Xuefu Zhang (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-5318) Import Throws Error when Importing from a table export Hive 0.9 to Hive 0.10
Date Sat, 21 Sep 2013 23:48:51 GMT

    [ https://issues.apache.org/jira/browse/HIVE-5318?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13773932#comment-13773932
] 

Xuefu Zhang commented on HIVE-5318:
-----------------------------------

[~ashutoshc] Test was planed when patch is submitted.

As to the nullness of these lists, your assumption might be well correct, but that's not what
I see in the code.

Here is the caller code in ImportSemanticAnalyzer.

{code}
        tblDesc =  new CreateTableDesc(
            table.getTableName(),
            false, // isExternal: set to false here, can be overwritten by the
                   // IMPORT stmt
            table.getSd().getCols(),
            table.getPartitionKeys(),
            table.getSd().getBucketCols(),
            table.getSd().getSortCols(),
            table.getSd().getNumBuckets(),
            null, null, null, null, null, // these 5 delims passed as serde params
            null, // comment passed as table params
            table.getSd().getInputFormat(),
            table.getSd().getOutputFormat(),
            null, // location: set to null here, can be
                  // overwritten by the IMPORT stmt
            table.getSd().getSerdeInfo().getSerializationLib(),
            null, // storagehandler passed as table params
            table.getSd().getSerdeInfo().getParameters(),
            table.getParameters(), false,
            (null == table.getSd().getSkewedInfo()) ? null : table.getSd().getSkewedInfo()
                .getSkewedColNames(),
            (null == table.getSd().getSkewedInfo()) ? null : table.getSd().getSkewedInfo()
                .getSkewedColValues());

{code}

>From the snippet we can see that, it's possible that the last two lists can be null. Also,
the partition columns was passed from the thrift "table" object, for which null is clearly
a valid value.

For reference, this is the metadata for an exported, simple table with two columns and two
rows of data:

{code}
{"partitions":[],"table":"{\"1\":{\"str\":\"j1_41\"},\"2\":{\"str\":\"default\"},\"3\":{\"str\":\"johndee\"},\"4\":{\"i32\":1371900915},\"5\":{\"i32\":0},\"6\":{\"i32\":0},\"7\":{\"rec\":{\"1\":{\"lst\":[\"rec\",2,{\"1\":{\"str\":\"a\"},\"2\":{\"str\":\"string\"}},{\"1\":{\"str\":\"b\"},\"2\":{\"str\":\"int\"}}]},\"2\":{\"str\":\"hdfs://hivebase01:8020/user/hive/warehouse/j1_41\"},\"3\":{\"str\":\"org.apache.hadoop.mapred.TextInputFormat\"},\"4\":{\"str\":\"org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat\"},\"5\":{\"tf\":0},\"6\":{\"i32\":-1},\"7\":{\"rec\":{\"2\":{\"str\":\"org.apache.hadoop.hive.serde2.lazy.LazySimpleSerDe\"},\"3\":{\"map\":[\"str\",\"str\",2,{\"serialization.format\":\",\",\"field.delim\":\",\"}]}}},\"8\":{\"lst\":[\"str\",0]},\"9\":{\"lst\":[\"rec\",0]},\"10\":{\"map\":[\"str\",\"str\",0,{}]}}},\"8\":{\"lst\":[\"rec\",0]},\"9\":{\"map\":[\"str\",\"str\",1,{\"transient_lastDdlTime\":\"1371900931\"}]},\"12\":{\"str\":\"MANAGED_TABLE\"}}","version":"0.1"}
{code}

This piece of meta data contains no partition columns, or skewedkey/values, etc.

Could you clarify if you meant to say that the list should not null but with zero element?
For unknown reason, the code doesn't reflect that either. for instance, Utilities.getFieldSchemaString()
has code to handle a null list of partition columns.

Any further insight is appreciated.

                
> Import Throws Error when Importing from a table export Hive 0.9 to Hive 0.10
> ----------------------------------------------------------------------------
>
>                 Key: HIVE-5318
>                 URL: https://issues.apache.org/jira/browse/HIVE-5318
>             Project: Hive
>          Issue Type: Bug
>          Components: Import/Export
>    Affects Versions: 0.9.0, 0.10.0
>            Reporter: Brad Ruderman
>            Assignee: Xuefu Zhang
>            Priority: Critical
>         Attachments: HIVE-5318.patch
>
>
> When Exporting hive tables using the hive command in Hive 0.9 "EXPORT table TO 'hdfs_path'"
then importing to another hive 0.10 instance using "IMPORT FROM 'hdfs_path'", hive throws
this error:
> 13/09/18 13:14:02 ERROR ql.Driver: FAILED: SemanticException Exception while processing
> org.apache.hadoop.hive.ql.parse.SemanticException: Exception while processing
> 	at org.apache.hadoop.hive.ql.parse.ImportSemanticAnalyzer.analyzeInternal(ImportSemanticAnalyzer.java:277)
> 	at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:258)
> 	at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:459)
> 	at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:349)
> 	at org.apache.hadoop.hive.ql.Driver.run(Driver.java:938)
> 	at org.apache.hadoop.hive.ql.Driver.run(Driver.java:902)
> 	at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:259)
> 	at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:216)
> 	at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:412)
> 	at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:347)
> 	at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:706)
> 	at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:613)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
> 	at java.lang.reflect.Method.invoke(Method.java:597)
> 	at org.apache.hadoop.util.RunJar.main(RunJar.java:208)
> Caused by: java.lang.NullPointerException
> 	at java.util.ArrayList.<init>(ArrayList.java:131)
> 	at org.apache.hadoop.hive.ql.plan.CreateTableDesc.<init>(CreateTableDesc.java:128)
> 	at org.apache.hadoop.hive.ql.parse.ImportSemanticAnalyzer.analyzeInternal(ImportSemanticAnalyzer.java:99)
> 	... 16 more
> 13/09/18 13:14:02 INFO ql.Driver: </PERFLOG method=compile start=1379535241411 end=1379535242332
duration=921>
> 13/09/18 13:14:02 INFO ql.Driver: <PERFLOG method=releaseLocks>
> 13/09/18 13:14:02 INFO ql.Driver: </PERFLOG method=releaseLocks start=1379535242332
end=1379535242332 duration=0>
> 13/09/18 13:14:02 INFO ql.Driver: <PERFLOG method=releaseLocks>
> 13/09/18 13:14:02 INFO ql.Driver: </PERFLOG method=releaseLocks start=1379535242333
end=1379535242333 duration=0>
> This is probably a critical blocker for people who are trying to test Hive 0.10 in their
staging environments prior to the upgrade from 0.9

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message