hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hive QA (Jira)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-22725) Lazy evaluate HiveMetastore::fireReadTablePreEvent table computation
Date Tue, 14 Jan 2020 22:39:00 GMT

    [ https://issues.apache.org/jira/browse/HIVE-22725?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17015455#comment-17015455
] 

Hive QA commented on HIVE-22725:
--------------------------------



Here are the results of testing the latest attachment:
https://issues.apache.org/jira/secure/attachment/12990833/HIVE-22725.2.patch

{color:red}ERROR:{color} -1 due to no test(s) being added or modified.

{color:red}ERROR:{color} -1 due to 68 failed/errored test(s), 17876 tests executed
*Failed tests:*
{noformat}
org.apache.hadoop.hive.metastore.TestEmbeddedHiveMetaStore.testPartitionFilter (batchId=230)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStore.testPartitionFilter (batchId=232)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreCustomAuth.testPartitionFilter (batchId=236)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZK.testPartitionFilter (batchId=231)
org.apache.hadoop.hive.metastore.TestRemoteHiveMetaStoreZKBindHost.testPartitionFilter (batchId=238)
org.apache.hadoop.hive.metastore.TestSetUGIOnBothClientServer.testPartitionFilter (batchId=228)
org.apache.hadoop.hive.metastore.TestSetUGIOnOnlyClient.testPartitionFilter (batchId=226)
org.apache.hadoop.hive.metastore.TestSetUGIOnOnlyServer.testPartitionFilter (batchId=237)
org.apache.hadoop.hive.metastore.client.TestGetPartitions.getPartitionsByNamesBogusCatalog[Embedded]
(batchId=229)
org.apache.hadoop.hive.metastore.client.TestGetPartitions.getPartitionsByNamesBogusCatalog[Remote]
(batchId=229)
org.apache.hadoop.hive.metastore.client.TestGetPartitions.testGetPartitionWithAuthInfoNullDbName[Embedded]
(batchId=229)
org.apache.hadoop.hive.metastore.client.TestGetPartitions.testGetPartitionWithAuthInfoNullDbName[Remote]
(batchId=229)
org.apache.hadoop.hive.metastore.client.TestGetPartitions.testGetPartitionWithAuthInfoNullTblName[Embedded]
(batchId=229)
org.apache.hadoop.hive.metastore.client.TestGetPartitions.testGetPartitionWithAuthInfoNullTblName[Remote]
(batchId=229)
org.apache.hadoop.hive.metastore.client.TestGetPartitions.testGetPartitionsByNamesNoDb[Embedded]
(batchId=229)
org.apache.hadoop.hive.metastore.client.TestGetPartitions.testGetPartitionsByNamesNoDb[Remote]
(batchId=229)
org.apache.hadoop.hive.metastore.client.TestGetPartitions.testGetPartitionsByNamesNoTable[Embedded]
(batchId=229)
org.apache.hadoop.hive.metastore.client.TestGetPartitions.testGetPartitionsByNamesNoTable[Remote]
(batchId=229)
org.apache.hadoop.hive.metastore.client.TestListPartitions.listPartitionNamesBogusCatalog[Embedded]
(batchId=227)
org.apache.hadoop.hive.metastore.client.TestListPartitions.listPartitionNamesBogusCatalog[Remote]
(batchId=227)
org.apache.hadoop.hive.metastore.client.TestListPartitions.testListPartitionNamesByValuesNullDbName[Embedded]
(batchId=227)
org.apache.hadoop.hive.metastore.client.TestListPartitions.testListPartitionNamesByValuesNullDbName[Remote]
(batchId=227)
org.apache.hadoop.hive.metastore.client.TestListPartitions.testListPartitionNamesByValuesNullTblName[Embedded]
(batchId=227)
org.apache.hadoop.hive.metastore.client.TestListPartitions.testListPartitionNamesByValuesNullTblName[Remote]
(batchId=227)
org.apache.hadoop.hive.metastore.client.TestListPartitions.testListPartitionNamesNoDbName[Embedded]
(batchId=227)
org.apache.hadoop.hive.metastore.client.TestListPartitions.testListPartitionNamesNoDbName[Remote]
(batchId=227)
org.apache.hadoop.hive.metastore.client.TestListPartitions.testListPartitionNamesNoDb[Embedded]
(batchId=227)
org.apache.hadoop.hive.metastore.client.TestListPartitions.testListPartitionNamesNoDb[Remote]
(batchId=227)
org.apache.hadoop.hive.metastore.client.TestListPartitions.testListPartitionNamesNoTable[Embedded]
(batchId=227)
org.apache.hadoop.hive.metastore.client.TestListPartitions.testListPartitionNamesNoTable[Remote]
(batchId=227)
org.apache.hadoop.hive.metastore.client.TestListPartitions.testListPartitionNamesNoTblName[Embedded]
(batchId=227)
org.apache.hadoop.hive.metastore.client.TestListPartitions.testListPartitionNamesNoTblName[Remote]
(batchId=227)
org.apache.hadoop.hive.metastore.client.TestListPartitions.testListPartitionNamesNullDbName[Embedded]
(batchId=227)
org.apache.hadoop.hive.metastore.client.TestListPartitions.testListPartitionNamesNullDbName[Remote]
(batchId=227)
org.apache.hadoop.hive.metastore.client.TestListPartitions.testListPartitionNamesNullTblName[Embedded]
(batchId=227)
org.apache.hadoop.hive.metastore.client.TestListPartitions.testListPartitionNamesNullTblName[Remote]
(batchId=227)
org.apache.hadoop.hive.metastore.client.TestListPartitions.testListPartitionsAllNullDbName[Embedded]
(batchId=227)
org.apache.hadoop.hive.metastore.client.TestListPartitions.testListPartitionsAllNullDbName[Remote]
(batchId=227)
org.apache.hadoop.hive.metastore.client.TestListPartitions.testListPartitionsAllNullTblName[Embedded]
(batchId=227)
org.apache.hadoop.hive.metastore.client.TestListPartitions.testListPartitionsAllNullTblName[Remote]
(batchId=227)
org.apache.hadoop.hive.metastore.client.TestListPartitions.testListPartitionsByFilterNullDbName[Embedded]
(batchId=227)
org.apache.hadoop.hive.metastore.client.TestListPartitions.testListPartitionsByFilterNullDbName[Remote]
(batchId=227)
org.apache.hadoop.hive.metastore.client.TestListPartitions.testListPartitionsByFilterNullTblName[Embedded]
(batchId=227)
org.apache.hadoop.hive.metastore.client.TestListPartitions.testListPartitionsByFilterNullTblName[Remote]
(batchId=227)
org.apache.hadoop.hive.metastore.client.TestListPartitions.testListPartitionsWithAuthByValuesNullDbName[Embedded]
(batchId=227)
org.apache.hadoop.hive.metastore.client.TestListPartitions.testListPartitionsWithAuthByValuesNullDbName[Remote]
(batchId=227)
org.apache.hadoop.hive.metastore.client.TestListPartitions.testListPartitionsWithAuthByValuesNullTblName[Embedded]
(batchId=227)
org.apache.hadoop.hive.metastore.client.TestListPartitions.testListPartitionsWithAuthByValuesNullTblName[Remote]
(batchId=227)
org.apache.hadoop.hive.ql.TestTxnCommandsWithSplitUpdateAndVectorization.testParallelInsertAnalyzeStats
(batchId=341)
org.apache.hadoop.hive.ql.metadata.TestSessionHiveMetastoreClientGetPartitionsTempTable.testGetPartitionsByNamesNoTable[Embedded]
(batchId=336)
org.apache.hadoop.hive.ql.metadata.TestSessionHiveMetastoreClientGetPartitionsTempTable.testGetPartitionsByNamesNoTable[Remote]
(batchId=336)
org.apache.hadoop.hive.ql.metadata.TestSessionHiveMetastoreClientListPartitionsTempTable.testListPartitionNamesNoDbName[Embedded]
(batchId=336)
org.apache.hadoop.hive.ql.metadata.TestSessionHiveMetastoreClientListPartitionsTempTable.testListPartitionNamesNoDbName[Remote]
(batchId=336)
org.apache.hadoop.hive.ql.metadata.TestSessionHiveMetastoreClientListPartitionsTempTable.testListPartitionNamesNoTable[Embedded]
(batchId=336)
org.apache.hadoop.hive.ql.metadata.TestSessionHiveMetastoreClientListPartitionsTempTable.testListPartitionNamesNoTable[Remote]
(batchId=336)
org.apache.hadoop.hive.ql.metadata.TestSessionHiveMetastoreClientListPartitionsTempTable.testListPartitionNamesNoTblName[Embedded]
(batchId=336)
org.apache.hadoop.hive.ql.metadata.TestSessionHiveMetastoreClientListPartitionsTempTable.testListPartitionNamesNoTblName[Remote]
(batchId=336)
org.apache.hadoop.hive.ql.metadata.TestSessionHiveMetastoreClientListPartitionsTempTable.testListPartitionsAllNoTable[Embedded]
(batchId=336)
org.apache.hadoop.hive.ql.metadata.TestSessionHiveMetastoreClientListPartitionsTempTable.testListPartitionsAllNoTable[Remote]
(batchId=336)
org.apache.hadoop.hive.ql.metadata.TestSessionHiveMetastoreClientListPartitionsTempTable.testListPartitionsByExprEmptyDbName[Embedded]
(batchId=336)
org.apache.hadoop.hive.ql.metadata.TestSessionHiveMetastoreClientListPartitionsTempTable.testListPartitionsByExprEmptyDbName[Remote]
(batchId=336)
org.apache.hadoop.hive.ql.metadata.TestSessionHiveMetastoreClientListPartitionsTempTable.testListPartitionsByExprEmptyTblName[Embedded]
(batchId=336)
org.apache.hadoop.hive.ql.metadata.TestSessionHiveMetastoreClientListPartitionsTempTable.testListPartitionsByExprEmptyTblName[Remote]
(batchId=336)
org.apache.hadoop.hive.ql.metadata.TestSessionHiveMetastoreClientListPartitionsTempTable.testListPartitionsByExprNoTbl[Embedded]
(batchId=336)
org.apache.hadoop.hive.ql.metadata.TestSessionHiveMetastoreClientListPartitionsTempTable.testListPartitionsByExprNoTbl[Remote]
(batchId=336)
org.apache.hive.minikdc.TestRemoteHiveMetaStoreDualAuthCustom.testPartitionFilter (batchId=304)
org.apache.hive.minikdc.TestRemoteHiveMetaStoreDualAuthKerb.testPartitionFilter (batchId=304)
org.apache.hive.minikdc.TestRemoteHiveMetaStoreKerberos.testPartitionFilter (batchId=304)
{noformat}

Test results: https://builds.apache.org/job/PreCommit-HIVE-Build/20173/testReport
Console output: https://builds.apache.org/job/PreCommit-HIVE-Build/20173/console
Test logs: http://104.198.109.242/logs/PreCommit-HIVE-Build-20173/

Messages:
{noformat}
Executing org.apache.hive.ptest.execution.TestCheckPhase
Executing org.apache.hive.ptest.execution.PrepPhase
Executing org.apache.hive.ptest.execution.YetusPhase
Executing org.apache.hive.ptest.execution.ExecutionPhase
Executing org.apache.hive.ptest.execution.ReportingPhase
Tests exited with: TestsFailedException: 68 tests failed
{noformat}

This message is automatically generated.

ATTACHMENT ID: 12990833 - PreCommit-HIVE-Build

> Lazy evaluate HiveMetastore::fireReadTablePreEvent table computation
> --------------------------------------------------------------------
>
>                 Key: HIVE-22725
>                 URL: https://issues.apache.org/jira/browse/HIVE-22725
>             Project: Hive
>          Issue Type: Improvement
>            Reporter: Rajesh Balamohan
>            Assignee: Rajesh Balamohan
>            Priority: Minor
>         Attachments: HIVE-22725.1.patch, HIVE-22725.2.patch, image-2020-01-14-13-22-54-483.png
>
>
> "TransactionalValidationListener" gets added in the pre-event listeners of HMS by default.
> [https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java#L559]
> This causes issue in short select queries, as table details are computed for any partition
lookups.
> [https://github.com/apache/hive/blob/master/standalone-metastore/metastore-server/src/main/java/org/apache/hadoop/hive/metastore/HiveMetaStore.java#L4984]
>  
> !image-2020-01-14-13-22-54-483.png|width=579,height=202!
>  
> It would be good to lazy evaluate table lookup in this codepath.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Mime
View raw message