hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Zoltan Chovan (Jira)" <j...@apache.org>
Subject [jira] [Comment Edited] (HIVE-22741) Speed up ObjectStore method getTableMeta
Date Tue, 21 Jan 2020 12:23:00 GMT

    [ https://issues.apache.org/jira/browse/HIVE-22741?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17020181#comment-17020181
] 

Zoltan Chovan edited comment on HIVE-22741 at 1/21/20 12:22 PM:
----------------------------------------------------------------

Based on the DataNucelus logging, the following queries were generated for a call in TestGetTableMeta.testGetTableMeta()

 
{code:java}
DEBUG [DataNucleus.Datastore.Native] - SELECT DISTINCT 'org.apache.hadoop.hive.metastore.model.MTable'
AS NUCLEUS_TYPE,A0.CREATE_TIME,C0.CTLG_NAME,C0.CREATE_TIME,C0."DESC",C0.DB_LOCATION_URI,C0."NAME",C0.OWNER_NAME,C0.OWNER_TYPE,C0.DB_ID,A0.TBL_ID,A0.LAST_ACCESS_TIME,A0.OWNER,A0.OWNER_TYPE,A0.RETENTION,A0.IS_REWRITE_ENABLED,A0.TBL_NAME,A0.TBL_TYPE,A0.WRITE_ID
FROM TBLS A0 LEFT OUTER JOIN DBS B0 ON A0.DB_ID = B0.DB_ID LEFT OUTER JOIN DBS C0 ON A0.DB_ID
= C0.DB_ID WHERE B0.CTLG_NAME = <'hive'> AND LOWER(B0."NAME") LIKE '%one' ESCAPE '\'
DEBUG [DataNucleus.Datastore.Native] - SELECT A0.PARAM_KEY,A0.PARAM_VALUE FROM TABLE_PARAMS
A0 WHERE A0.TBL_ID = <1> AND A0.PARAM_KEY IS NOT NULL
DEBUG [DataNucleus.Datastore.Native] - SELECT A0.PARAM_KEY,A0.PARAM_VALUE FROM TABLE_PARAMS
A0 WHERE A0.TBL_ID = <2> AND A0.PARAM_KEY IS NOT NULL
DEBUG [DataNucleus.Datastore.Native] - SELECT A0.PARAM_KEY,A0.PARAM_VALUE FROM TABLE_PARAMS
A0 WHERE A0.TBL_ID = <3> AND A0.PARAM_KEY IS NOT NULL
{code}
With the patch applied the generated query is as follows:
{code:java}
DEBUG [DataNucleus.Datastore.Native] - SELECT B0."NAME",A0.TBL_NAME,A0.TBL_TYPE,(SELECT A0_SUB.PARAM_VALUE
FROM TABLE_PARAMS A0_SUB WHERE A0_SUB.TBL_ID = A0.TBL_ID AND A0_SUB.PARAM_KEY = 'comment')
FROM TBLS A0 LEFT OUTER JOIN DBS B0 ON A0.DB_ID = B0.DB_ID WHERE B0.CTLG_NAME = <'hive'>
AND LOWER(B0."NAME") LIKE '%one' ESCAPE '\'
{code}
 

Attached the full logs:

[^patched_datanucleus.log]
[^no_patch_datanucleus.log]

Running
{noformat}
cat <logname> | cut -d" " -f3- > stripped.log{noformat}
on the log files allows for running diff for comparison.

 

 


was (Author: zchovan):
Based on the DataNucelus logging, the following queries were generated for a call in TestGetTableMeta.testGetTableMeta()

 
{code:java}
DEBUG [DataNucleus.Datastore.Native] - SELECT DISTINCT 'org.apache.hadoop.hive.metastore.model.MTable'
AS NUCLEUS_TYPE,A0.CREATE_TIME,C0.CTLG_NAME,C0.CREATE_TIME,C0."DESC",C0.DB_LOCATION_URI,C0."NAME",C0.OWNER_NAME,C0.OWNER_TYPE,C0.DB_ID,A0.TBL_ID,A0.LAST_ACCESS_TIME,A0.OWNER,A0.OWNER_TYPE,A0.RETENTION,A0.IS_REWRITE_ENABLED,A0.TBL_NAME,A0.TBL_TYPE,A0.WRITE_ID
FROM TBLS A0 LEFT OUTER JOIN DBS B0 ON A0.DB_ID = B0.DB_ID LEFT OUTER JOIN DBS C0 ON A0.DB_ID
= C0.DB_ID WHERE B0.CTLG_NAME = <'hive'> AND LOWER(B0."NAME") LIKE '%one' ESCAPE '\'
DEBUG [DataNucleus.Datastore.Native] - SELECT A0.PARAM_KEY,A0.PARAM_VALUE FROM TABLE_PARAMS
A0 WHERE A0.TBL_ID = <1> AND A0.PARAM_KEY IS NOT NULL
DEBUG [DataNucleus.Datastore.Native] - SELECT A0.PARAM_KEY,A0.PARAM_VALUE FROM TABLE_PARAMS
A0 WHERE A0.TBL_ID = <2> AND A0.PARAM_KEY IS NOT NULL
DEBUG [DataNucleus.Datastore.Native] - SELECT A0.PARAM_KEY,A0.PARAM_VALUE FROM TABLE_PARAMS
A0 WHERE A0.TBL_ID = <3> AND A0.PARAM_KEY IS NOT NULL
{code}
With the patch applied the generated query is as follows:
{code:java}
DEBUG [DataNucleus.Datastore.Native] - SELECT B0."NAME",A0.TBL_NAME,A0.TBL_TYPE,(SELECT A0_SUB.PARAM_VALUE
FROM TABLE_PARAMS A0_SUB WHERE A0_SUB.TBL_ID = A0.TBL_ID AND A0_SUB.PARAM_KEY = 'comment')
FROM TBLS A0 LEFT OUTER JOIN DBS B0 ON A0.DB_ID = B0.DB_ID WHERE B0.CTLG_NAME = <'hive'>
AND LOWER(B0."NAME") LIKE '%one' ESCAPE '\'
{code}

> Speed up ObjectStore method getTableMeta 
> -----------------------------------------
>
>                 Key: HIVE-22741
>                 URL: https://issues.apache.org/jira/browse/HIVE-22741
>             Project: Hive
>          Issue Type: Improvement
>            Reporter: Zoltan Chovan
>            Assignee: Zoltan Chovan
>            Priority: Major
>         Attachments: HIVE-22741.patch, no_patch_datanucleus.log, patched_datanucleus.log
>
>




--
This message was sent by Atlassian Jira
(v8.3.4#803005)

Mime
View raw message