hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jesus Camacho Rodriguez (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HIVE-10811) RelFieldTrimmer throws NoSuchElementException in some cases
Date Tue, 26 May 2015 14:28:20 GMT

     [ https://issues.apache.org/jira/browse/HIVE-10811?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Jesus Camacho Rodriguez updated HIVE-10811:
-------------------------------------------
    Attachment: HIVE-10811.02.patch

> RelFieldTrimmer throws NoSuchElementException in some cases
> -----------------------------------------------------------
>
>                 Key: HIVE-10811
>                 URL: https://issues.apache.org/jira/browse/HIVE-10811
>             Project: Hive
>          Issue Type: Bug
>          Components: CBO
>            Reporter: Jesus Camacho Rodriguez
>            Assignee: Jesus Camacho Rodriguez
>         Attachments: HIVE-10811.01.patch, HIVE-10811.02.patch, HIVE-10811.patch
>
>
> RelFieldTrimmer runs into NoSuchElementException in some cases.
> Stack trace:
> {noformat}
> Exception in thread "main" java.lang.AssertionError: Internal error: While invoking method
'public org.apache.calcite.sql2rel.RelFieldTrimmer$TrimResult org.apache.calcite.sql2rel.RelFieldTrimmer.trimFields(org.apache.calcite.rel.core.Sort,org.apache.calcite.util.ImmutableBitSet,java.util.Set)'
> 	at org.apache.calcite.util.Util.newInternal(Util.java:743)
> 	at org.apache.calcite.util.ReflectUtil$2.invoke(ReflectUtil.java:543)
> 	at org.apache.calcite.sql2rel.RelFieldTrimmer.dispatchTrimFields(RelFieldTrimmer.java:269)
> 	at org.apache.calcite.sql2rel.RelFieldTrimmer.trim(RelFieldTrimmer.java:175)
> 	at org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.applyPreJoinOrderingTransforms(CalcitePlanner.java:947)
> 	at org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.apply(CalcitePlanner.java:820)
> 	at org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction.apply(CalcitePlanner.java:768)
> 	at org.apache.calcite.tools.Frameworks$1.apply(Frameworks.java:109)
> 	at org.apache.calcite.prepare.CalcitePrepareImpl.perform(CalcitePrepareImpl.java:730)
> 	at org.apache.calcite.tools.Frameworks.withPrepare(Frameworks.java:145)
> 	at org.apache.calcite.tools.Frameworks.withPlanner(Frameworks.java:105)
> 	at org.apache.hadoop.hive.ql.parse.CalcitePlanner.getOptimizedAST(CalcitePlanner.java:607)
> 	at org.apache.hadoop.hive.ql.parse.CalcitePlanner.genOPTree(CalcitePlanner.java:244)
> 	at org.apache.hadoop.hive.ql.parse.SemanticAnalyzer.analyzeInternal(SemanticAnalyzer.java:10048)
> 	at org.apache.hadoop.hive.ql.parse.CalcitePlanner.analyzeInternal(CalcitePlanner.java:207)
> 	at org.apache.hadoop.hive.ql.parse.BaseSemanticAnalyzer.analyze(BaseSemanticAnalyzer.java:227)
> 	at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:424)
> 	at org.apache.hadoop.hive.ql.Driver.compile(Driver.java:308)
> 	at org.apache.hadoop.hive.ql.Driver.compileInternal(Driver.java:1122)
> 	at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1170)
> 	at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1059)
> 	at org.apache.hadoop.hive.ql.Driver.run(Driver.java:1049)
> 	at org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:213)
> 	at org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:165)
> 	at org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:376)
> 	at org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:736)
> 	at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:681)
> 	at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:621)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 	at java.lang.reflect.Method.invoke(Method.java:606)
> 	at org.apache.hadoop.util.RunJar.run(RunJar.java:221)
> 	at org.apache.hadoop.util.RunJar.main(RunJar.java:136)
> Caused by: java.lang.reflect.InvocationTargetException
> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 	at java.lang.reflect.Method.invoke(Method.java:606)
> 	at org.apache.calcite.util.ReflectUtil$2.invoke(ReflectUtil.java:536)
> 	... 32 more
> Caused by: java.lang.AssertionError: Internal error: While invoking method 'public org.apache.calcite.sql2rel.RelFieldTrimmer$TrimResult
org.apache.calcite.sql2rel.RelFieldTrimmer.trimFields(org.apache.calcite.rel.core.Sort,org.apache.calcite.util.ImmutableBitSet,java.util.Set)'
> 	at org.apache.calcite.util.Util.newInternal(Util.java:743)
> 	at org.apache.calcite.util.ReflectUtil$2.invoke(ReflectUtil.java:543)
> 	at org.apache.calcite.sql2rel.RelFieldTrimmer.dispatchTrimFields(RelFieldTrimmer.java:269)
> 	at org.apache.calcite.sql2rel.RelFieldTrimmer.trimChild(RelFieldTrimmer.java:210)
> 	at org.apache.calcite.sql2rel.RelFieldTrimmer.trimFields(RelFieldTrimmer.java:499)
> 	... 37 more
> Caused by: java.lang.reflect.InvocationTargetException
> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 	at java.lang.reflect.Method.invoke(Method.java:606)
> 	at org.apache.calcite.util.ReflectUtil$2.invoke(ReflectUtil.java:536)
> 	... 40 more
> Caused by: java.lang.AssertionError: Internal error: While invoking method 'public org.apache.calcite.sql2rel.RelFieldTrimmer$TrimResult
org.apache.calcite.sql2rel.RelFieldTrimmer.trimFields(org.apache.calcite.rel.core.Project,org.apache.calcite.util.ImmutableBitSet,java.util.Set)'
> 	at org.apache.calcite.util.Util.newInternal(Util.java:743)
> 	at org.apache.calcite.util.ReflectUtil$2.invoke(ReflectUtil.java:543)
> 	at org.apache.calcite.sql2rel.RelFieldTrimmer.dispatchTrimFields(RelFieldTrimmer.java:269)
> 	at org.apache.calcite.sql2rel.RelFieldTrimmer.trimChild(RelFieldTrimmer.java:210)
> 	at org.apache.calcite.sql2rel.RelFieldTrimmer.trimFields(RelFieldTrimmer.java:499)
> 	... 45 more
> Caused by: java.lang.reflect.InvocationTargetException
> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 	at java.lang.reflect.Method.invoke(Method.java:606)
> 	at org.apache.calcite.util.ReflectUtil$2.invoke(ReflectUtil.java:536)
> 	... 48 more
> Caused by: java.lang.AssertionError: Internal error: While invoking method 'public org.apache.calcite.sql2rel.RelFieldTrimmer$TrimResult
org.apache.calcite.sql2rel.RelFieldTrimmer.trimFields(org.apache.calcite.rel.core.Aggregate,org.apache.calcite.util.ImmutableBitSet,java.util.Set)'
> 	at org.apache.calcite.util.Util.newInternal(Util.java:743)
> 	at org.apache.calcite.util.ReflectUtil$2.invoke(ReflectUtil.java:543)
> 	at org.apache.calcite.sql2rel.RelFieldTrimmer.dispatchTrimFields(RelFieldTrimmer.java:269)
> 	at org.apache.calcite.sql2rel.RelFieldTrimmer.trimChild(RelFieldTrimmer.java:210)
> 	at org.apache.calcite.sql2rel.RelFieldTrimmer.trimFields(RelFieldTrimmer.java:345)
> 	... 53 more
> Caused by: java.lang.reflect.InvocationTargetException
> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 	at java.lang.reflect.Method.invoke(Method.java:606)
> 	at org.apache.calcite.util.ReflectUtil$2.invoke(ReflectUtil.java:536)
> 	... 56 more
> Caused by: java.lang.AssertionError: Internal error: While invoking method 'public org.apache.calcite.sql2rel.RelFieldTrimmer$TrimResult
org.apache.calcite.sql2rel.RelFieldTrimmer.trimFields(org.apache.calcite.rel.core.Project,org.apache.calcite.util.ImmutableBitSet,java.util.Set)'
> 	at org.apache.calcite.util.Util.newInternal(Util.java:743)
> 	at org.apache.calcite.util.ReflectUtil$2.invoke(ReflectUtil.java:543)
> 	at org.apache.calcite.sql2rel.RelFieldTrimmer.dispatchTrimFields(RelFieldTrimmer.java:269)
> 	at org.apache.calcite.sql2rel.RelFieldTrimmer.trimChild(RelFieldTrimmer.java:210)
> 	at org.apache.calcite.sql2rel.RelFieldTrimmer.trimFields(RelFieldTrimmer.java:772)
> 	... 61 more
> Caused by: java.lang.reflect.InvocationTargetException
> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 	at java.lang.reflect.Method.invoke(Method.java:606)
> 	at org.apache.calcite.util.ReflectUtil$2.invoke(ReflectUtil.java:536)
> 	... 64 more
> Caused by: java.lang.AssertionError: Internal error: While invoking method 'public org.apache.calcite.sql2rel.RelFieldTrimmer$TrimResult
org.apache.calcite.sql2rel.RelFieldTrimmer.trimFields(org.apache.calcite.rel.core.Project,org.apache.calcite.util.ImmutableBitSet,java.util.Set)'
> 	at org.apache.calcite.util.Util.newInternal(Util.java:743)
> 	at org.apache.calcite.util.ReflectUtil$2.invoke(ReflectUtil.java:543)
> 	at org.apache.calcite.sql2rel.RelFieldTrimmer.dispatchTrimFields(RelFieldTrimmer.java:269)
> 	at org.apache.calcite.sql2rel.RelFieldTrimmer.trimChild(RelFieldTrimmer.java:210)
> 	at org.apache.calcite.sql2rel.RelFieldTrimmer.trimFields(RelFieldTrimmer.java:345)
> 	... 69 more
> Caused by: java.lang.reflect.InvocationTargetException
> 	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
> 	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
> 	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 	at java.lang.reflect.Method.invoke(Method.java:606)
> 	at org.apache.calcite.util.ReflectUtil$2.invoke(ReflectUtil.java:536)
> 	... 72 more
> Caused by: java.util.NoSuchElementException
> 	at java.util.AbstractList$Itr.next(AbstractList.java:364)
> 	at java.util.AbstractList.hashCode(AbstractList.java:540)
> 	at org.apache.calcite.util.Util.hash(Util.java:230)
> 	at org.apache.calcite.util.Pair.hashCode(Pair.java:87)
> 	at com.google.common.base.Equivalences$Equals.doHash(Equivalences.java:70)
> 	at com.google.common.base.Equivalence.hash(Equivalence.java:105)
> 	at com.google.common.cache.LocalCache.hash(LocalCache.java:1888)
> 	at com.google.common.cache.LocalCache.getIfPresent(LocalCache.java:3953)
> 	at com.google.common.cache.LocalCache$LocalManualCache.getIfPresent(LocalCache.java:4758)
> 	at org.apache.calcite.rel.type.RelDataTypeFactoryImpl.canonize(RelDataTypeFactoryImpl.java:352)
> 	at org.apache.calcite.rel.type.RelDataTypeFactoryImpl.createStructType(RelDataTypeFactoryImpl.java:148)
> 	at org.apache.calcite.rel.type.RelDataTypeFactoryImpl.createStructType(RelDataTypeFactoryImpl.java:172)
> 	at org.apache.calcite.plan.RelOptUtil.permute(RelOptUtil.java:2589)
> 	at org.apache.calcite.sql2rel.RelFieldTrimmer.trimFields(RelFieldTrimmer.java:383)
> 	... 77 more
> {noformat}
> query:
> {noformat}
> SELECT w_warehouse_name, 
>        w_warehouse_sq_ft, 
>        w_city, 
>        w_county, 
>        w_state, 
>        w_country, 
>        ship_carriers, 
>        year, 
>        Sum(jan_sales)                     AS jan_sales,

>        Sum(feb_sales)                     AS feb_sales,

>        Sum(jan_sales / w_warehouse_sq_ft) AS jan_sales_per_sq_foot, 
>        Sum(feb_sales / w_warehouse_sq_ft) AS feb_sales_per_sq_foot, 
>        Sum(jan_net)                       AS jan_net,

>        Sum(feb_net)                       AS feb_net 
> FROM   (SELECT w_warehouse_name, 
>                w_warehouse_sq_ft, 
>                w_city, 
>                w_county, 
>                w_state, 
>                w_country, 
>                Concat('DHL', ',', 'BARIAN') AS ship_carriers, 
>                d_year                       AS year,

>                Sum(CASE 
>                      WHEN d_moy = 1 THEN ws_ext_sales_price * ws_quantity

>                      ELSE Cast(0 AS DECIMAL(7, 2)) 
>                    end)                     AS jan_sales,

>                Sum(CASE 
>                      WHEN d_moy = 2 THEN ws_ext_sales_price * ws_quantity

>                      ELSE Cast(0 AS DECIMAL(7, 2)) 
>                    end)                     AS feb_sales,

>                Sum(CASE 
>                      WHEN d_moy = 1 THEN ws_net_paid * ws_quantity

>                      ELSE Cast(0 AS DECIMAL(7, 2)) 
>                    end)                     AS jan_net,

>                Sum(CASE 
>                      WHEN d_moy = 2 THEN ws_net_paid * ws_quantity

>                      ELSE Cast(0 AS DECIMAL(7, 2)) 
>                    end)                     AS feb_net

>         FROM   web_sales 
>                JOIN warehouse 
>                  ON web_sales.ws_warehouse_sk = warehouse.w_warehouse_sk

>                JOIN date_dim 
>                  ON web_sales.ws_sold_date_sk = date_dim.d_date_sk

>                JOIN time_dim 
>                  ON web_sales.ws_sold_time_sk = time_dim.t_time_sk

>                JOIN ship_mode 
>                  ON web_sales.ws_ship_mode_sk = ship_mode.sm_ship_mode_sk

>         WHERE  d_year = 2001 
>                AND t_time BETWEEN 30838 AND 30838 + 28800 
>                AND sm_carrier IN ( 'DHL', 'BARIAN' ) 
>         GROUP  BY w_warehouse_name, 
>                   w_warehouse_sq_ft, 
>                   w_city, 
>                   w_county, 
>                   w_state, 
>                   w_country, 
>                   d_year 
>         UNION DISTINCT
>         SELECT w_warehouse_name, 
>                w_warehouse_sq_ft, 
>                w_city, 
>                w_county, 
>                w_state, 
>                w_country, 
>                Concat('DHL', ',', 'BARIAN') AS ship_carriers, 
>                d_year                       AS year,

>                Sum(CASE 
>                      WHEN d_moy = 1 THEN cs_sales_price * cs_quantity

>                      ELSE Cast(0 AS DECIMAL(7, 2)) 
>                    end)                     AS jan_sales,

>                Sum(CASE 
>                      WHEN d_moy = 2 THEN cs_sales_price * cs_quantity

>                      ELSE Cast(0 AS DECIMAL(7, 2)) 
>                    end)                     AS feb_sales,

>                Sum(CASE 
>                      WHEN d_moy = 1 THEN cs_net_paid_inc_tax * cs_quantity

>                      ELSE Cast(0 AS DECIMAL(7, 2)) 
>                    end)                     AS jan_net,

>                Sum(CASE 
>                      WHEN d_moy = 2 THEN cs_net_paid_inc_tax * cs_quantity

>                      ELSE Cast(0 AS DECIMAL(7, 2)) 
>                    end)                     AS feb_net

>         FROM   catalog_sales 
>                JOIN warehouse 
>                  ON catalog_sales.cs_warehouse_sk = warehouse.w_warehouse_sk

>                JOIN date_dim 
>                  ON catalog_sales.cs_sold_date_sk = date_dim.d_date_sk

>                JOIN time_dim 
>                  ON catalog_sales.cs_sold_time_sk = time_dim.t_time_sk

>                JOIN ship_mode 
>                  ON catalog_sales.cs_ship_mode_sk = ship_mode.sm_ship_mode_sk

>         WHERE  d_year = 2001 
>                AND t_time BETWEEN 30838 AND 30838 + 28800 
>                AND sm_carrier IN ( 'DHL', 'BARIAN' ) 
>         GROUP  BY w_warehouse_name, 
>                   w_warehouse_sq_ft, 
>                   w_city, 
>                   w_county, 
>                   w_state, 
>                   w_country, 
>                   d_year) x 
> GROUP  BY w_warehouse_name, 
>           w_warehouse_sq_ft, 
>           w_city, 
>           w_county, 
>           w_state, 
>           w_country, 
>           ship_carriers, 
>           year 
> ORDER  BY w_warehouse_name 
> LIMIT  100; 
> {noformat}
> Seems like use of Sum(jan_sales / w_warehouse_sq_ft) AS jan_sales_per_sq_foot, 
> Sum(feb_sales / w_warehouse_sq_ft) AS feb_sales_per_sq_foot in the select clause
causing the error.
> Query works fine if I turn off cbo or remove these two columns from select clause.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message