hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hari Sankar Sivarama Subramaniyan (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-13542) Missing stats for tables in TPCDS performance regression suite
Date Thu, 21 Apr 2016 20:29:12 GMT

    [ https://issues.apache.org/jira/browse/HIVE-13542?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15252613#comment-15252613
] 

Hari Sankar Sivarama Subramaniyan commented on HIVE-13542:
----------------------------------------------------------

[~jcamachorodriguez] I looked at this after I made the change. This time it looks like a metastore
client error and not a problem with the data.

{code}
16-04-21T11:45:55,201 DEBUG [f2995fd1-e4fc-4783-86cb-9264f715e540 main[]]: metastore.MetaStoreDirectSql
(MetaStoreDirectSql.java:timingTrace(864)) - Direct SQL query in 14.825303ms + 0.086336ms,
the query is [select "COLUMN_NAME", "COLUMN_TYPE", "LONG_LOW_VALUE", "LONG_HIGH_VALUE", "DOUBLE_LOW_VALU\
E", "DOUBLE_HIGH_VALUE", "BIG_DECIMAL_LOW_VALUE", "BIG_DECIMAL_HIGH_VALUE", "NUM_NULLS", "NUM_DISTINCTS",
"AVG_COL_LEN", "MAX_COL_LEN", "NUM_TRUES", "NUM_FALSES", "LAST_ANALYZED"  from "TAB_COL_STATS"
 where "DB_NAME" = ? and "TABLE_NAME" = ? and "COLUMN_NAME" in (...)]
2016-04-21T11:45:55,242 WARN  [f2995fd1-e4fc-4783-86cb-9264f715e540 main[]]: metastore.ObjectStore
(ObjectStore.java:handleDirectSqlError(2602)) - Direct SQL failed, falling back to ORM
org.apache.hadoop.hive.metastore.api.MetaException: The threadlocal Deadline is null, please
register it first.
        at org.apache.hadoop.hive.metastore.Deadline.newMetaException(Deadline.java:178) ~[hive-metastore-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
        at org.apache.hadoop.hive.metastore.Deadline.checkTimeout(Deadline.java:149) ~[hive-metastore-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
        at org.apache.hadoop.hive.metastore.MetaStoreDirectSql.makeColumnStats(MetaStoreDirectSql.java:1717)
~[hive-metastore-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
        at org.apache.hadoop.hive.metastore.MetaStoreDirectSql.getTableStats(MetaStoreDirectSql.java:1201)
~[hive-metastore-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
        at org.apache.hadoop.hive.metastore.ObjectStore$7.getSqlResult(ObjectStore.java:6849)
~[hive-metastore-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
        at org.apache.hadoop.hive.metastore.ObjectStore$7.getSqlResult(ObjectStore.java:6846)
~[hive-metastore-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
        at org.apache.hadoop.hive.metastore.ObjectStore$GetHelper.run(ObjectStore.java:2567)
[hive-metastore-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
        at org.apache.hadoop.hive.metastore.ObjectStore.getTableColumnStatisticsInternal(ObjectStore.java:6845)
[hive-metastore-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
        at org.apache.hadoop.hive.metastore.ObjectStore.getTableColumnStatistics(ObjectStore.java:6839)
[hive-metastore-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
        at org.apache.hadoop.hive.metastore.HiveMetaStore$HMSHandler.get_table_statistics_req(HiveMetaStore.java:4361)
[hive-metastore-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
        at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.getTableColumnStatistics(HiveMetaStoreClient.java:1613)
[hive-metastore-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
        at org.apache.hadoop.hive.ql.metadata.SessionHiveMetaStoreClient.getTableColumnStatistics(SessionHiveMetaStoreClient.java:347)
[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
        at org.apache.hadoop.hive.ql.metadata.Hive.getTableColumnStatistics(Hive.java:3317)
[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
        at org.apache.hadoop.hive.ql.stats.StatsUtils.getTableColumnStats(StatsUtils.java:758)
[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
        at org.apache.hadoop.hive.ql.optimizer.calcite.RelOptHiveTable.updateColStats(RelOptHiveTable.java:302)
[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
        at org.apache.hadoop.hive.ql.optimizer.calcite.RelOptHiveTable.getColStat(RelOptHiveTable.java:417)
[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
        at org.apache.hadoop.hive.ql.optimizer.calcite.RelOptHiveTable.getColStat(RelOptHiveTable.java:410)
[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
        at org.apache.hadoop.hive.ql.optimizer.calcite.reloperators.HiveTableScan.getColStat(HiveTableScan.java:158)
[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
        at org.apache.hadoop.hive.ql.optimizer.calcite.stats.HiveRelMdDistinctRowCount.getDistinctRowCount(HiveRelMdDistinctRowCount.java:77)
[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
        at org.apache.hadoop.hive.ql.optimizer.calcite.stats.HiveRelMdDistinctRowCount.getDistinctRowCount(HiveRelMdDistinctRowCount.java:64)
[hive-exec-2.1.0-SNAPSHOT.jar:2.1.0-SNAPSHOT]
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) ~[?:1.7.0_79]
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57) ~[?:1.7.0_79]
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
~[?:1.7.0_79]
        at java.lang.reflect.Method.invoke(Method.java:606) ~[?:1.7.0_79]
{code}

It seems  we need to register the Deadline before we can invoke the call on retrieving the
table statistics. [~sershe] , looking at the stack trace is it possible to suggest a way to
 make the fix at the client side.

Thanks
Hari

> Missing stats for tables in TPCDS performance regression suite
> --------------------------------------------------------------
>
>                 Key: HIVE-13542
>                 URL: https://issues.apache.org/jira/browse/HIVE-13542
>             Project: Hive
>          Issue Type: Bug
>            Reporter: Hari Sankar Sivarama Subramaniyan
>            Assignee: Hari Sankar Sivarama Subramaniyan
>         Attachments: HIVE-13542.1.patch
>
>
> These are the tables whose stats are missing in data/files/tpcds-perf/metastore_export/csv/TAB_COL_STATS.txt:
> * catalog_returns
> * catalog_sales
> * inventory
> * store_returns
> * store_sales
> * web_returns
> * web_sales
> Thanks to [~jcamachorodriguez] for discovering this issue.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message