hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Gopal V (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-7396) BucketingSortingReduceSinkOptimizer throws NullPointException during ETL
Date Sat, 12 Jul 2014 05:31:05 GMT

    [ https://issues.apache.org/jira/browse/HIVE-7396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14059657#comment-14059657
] 

Gopal V commented on HIVE-7396:
-------------------------------

This is not just a simple NULL check missing.

I made modifications to test this out

{code}
--- a/ql/src/java/org/apache/hadoop/hive/ql/optimizer/BucketingSortingReduceSinkOptimizer.java
+++ b/ql/src/java/org/apache/hadoop/hive/ql/optimizer/BucketingSortingReduceSinkOptimizer.java
@@ -498,7 +498,7 @@ public Object process(Node nd, Stack<Node> stack, NodeProcessorCtx
procCtx,
 
             if (srcTable.isPartitioned()) {
               PrunedPartitionList prunedParts = pGraphContext.getOpToPartList().get(ts);
-              List<Partition> partitions = prunedParts.getNotDeniedPartns();
+              List<Partition> partitions = (prunedParts != null ) ? prunedParts.getNotDeniedPartns()
: null;
 
               // Support for dynamic partitions can be added later
               // The following is not optimized:
{code}

But then it generates a scan on only one partition instead of a full-table scan.

> BucketingSortingReduceSinkOptimizer throws NullPointException during ETL
> ------------------------------------------------------------------------
>
>                 Key: HIVE-7396
>                 URL: https://issues.apache.org/jira/browse/HIVE-7396
>             Project: Hive
>          Issue Type: Bug
>            Reporter: Gopal V
>            Priority: Trivial
>
> When a bucketed insert is performed from a partitioned table into a new table
> {code}
> 2014-07-11 22:11:30,176 ERROR [main]: ql.Driver (SessionState.java:printError(660)) -
FAILED: NullPointerException null
> java.lang.NullPointerException
>         at org.apache.hadoop.hive.ql.optimizer.BucketingSortingReduceSinkOptimizer$BucketSortReduceSinkProcessor.process(BucketingSortingReduceSinkOptimizer.java:501)
>         at org.apache.hadoop.hive.ql.lib.DefaultRuleDispatcher.dispatch(DefaultRuleDispatcher.java:90)
>         at org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatchAndReturn(DefaultGraphWalker.java:94)
>         at org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.dispatch(DefaultGraphWalker.java:78)
>         at org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.walk(DefaultGraphWalker.java:132)
>         at org.apache.hadoop.hive.ql.lib.DefaultGraphWalker.startWalking(DefaultGraphWalker.java:109)
>         at org.apache.hadoop.hive.ql.optimizer.BucketingSortingReduceSinkOptimizer.transform(BucketingSortingReduceSinkOptimizer.java:97)
>         at org.apache.hadoop.hive.ql.optimizer.Optimizer.optimize(Optimizer.java:146)
>         at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:9503)
>         at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:328)
> {code}



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message