hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Barnabas Maidics (JIRA)" <>
Subject [jira] [Commented] (HIVE-21072) NPE when running partitioned CTAS statements
Date Tue, 15 Jan 2019 13:46:00 GMT


Barnabas Maidics commented on HIVE-21072:

I've also tried running partitioned CTAS and ran into the same error. As I saw, if the execution
engine is TEZ, it works perfectly (that is why partition_ctas worked with TestMiniLlapLocalCliDriver). 

Using MR, the NPE was thrown because Hive tried to create a Map-only merge job (_GenMapRedUtils.createMRWorkForMergingFiles_),
but the _tableInfo_ of the _FileSinkDesc_ doesn't contain an entry with the key of "partition_columns"
and we try to call split on a null.
String[] partNames = properties.getProperty(
A possible quick fix is to *set _hive.merge.mapfiles_ to _false_* (as a default, it is true)
so these steps will be skipped (_GenMapRedUtils.__isMergeRequired_ will return _false_).
But maybe I miss something about this feature. [~jcamachorodriguez] what do you think the
long-term fix would be for this?

> NPE when running partitioned CTAS statements
> --------------------------------------------
>                 Key: HIVE-21072
>                 URL:
>             Project: Hive
>          Issue Type: Bug
>            Reporter: Liang-Chi Hsieh
>            Priority: Major
>              Labels: pull-request-available
> HIVE-20241 adds support of partitioned CTAS statements:
> {code:sql}
> CREATE TABLE partition_ctas_1 PARTITIONED BY (key) AS
> SELECT value, key FROM src where key > 200 and key < 300;{code}
> However, I've tried this feature by checking out latest branch-3, and encountered NPE:
> {code:java}
> hive> CREATE TABLE t PARTITIONED BY (part) AS SELECT 1 as id, "a" as part;
> FAILED: NullPointerException null
> {code}
> I also ran the query test partition_ctas.q. The test passes when using TestMiniLlapLocalCliDriver,
but when I go to test it with TestCliDriver manually, it also throws NullPointerException:
> {code:java}
> 2018-12-25T05:58:22,221 ERROR [a96009a7-3dda-4d95-9536-e2e16d976856 main] ql.Driver:
FAILED: NullPointerException null
> java.lang.NullPointerException
>     at org.apache.hadoop.hive.ql.optimizer.GenMapRedUtils.usePartitionColumns(
>     at org.apache.hadoop.hive.ql.optimizer.GenMapRedUtils.createMRWorkForMergingFiles(
>     at org.apache.hadoop.hive.ql.optimizer.GenMRFileSink1.process(
>     at org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(
>     at org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(
>     at org.apache.hadoop.hive.ql.parse.GenMapRedWalker.walk(
>     at org.apache.hadoop.hive.ql.parse.GenMapRedWalker.walk(
>     at org.apache.hadoop.hive.ql.parse.GenMapRedWalker.walk(
>     at org.apache.hadoop.hive.ql.parse.GenMapRedWalker.walk(
>     at org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.startWalking(
>     at org.apache.hadoop.hive.ql.parse.MapReduceCompiler.generateTaskTree(
>     at org.apache.hadoop.hive.ql.parse.TaskCompiler.compile(
>     at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(
>     at org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(
>     at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(
>     at org.apache.hadoop.hive.ql.parse.ExplainSemanticAnalyzer.analyzeInternal(
>     at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(
>     at org.apache.hadoop.hive.ql.Driver.compile(
>     at org.apache.hadoop.hive.ql.Driver.compileInternal(
>     at org.apache.hadoop.hive.ql.Driver.compileAndRespond(
>     at org.apache.hadoop.hive.ql.Driver.compileAndRespond(
>     at org.apache.hadoop.hive.ql.reexec.ReExecDriver.compileAndRespond(
>     at
>     at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(
>     at org.apache.hadoop.hive.cli.CliDriver.processCmd(
>     at org.apache.hadoop.hive.cli.CliDriver.processLine(
> {code}

This message was sent by Atlassian JIRA

View raw message