hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jesus Camacho Rodriguez (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-13757) Add PerfLog information to genLogicalPlan in CalcitePlanner
Date Thu, 19 May 2016 11:41:12 GMT

    [ https://issues.apache.org/jira/browse/HIVE-13757?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15290958#comment-15290958
] 

Jesus Camacho Rodriguez commented on HIVE-13757:
------------------------------------------------

[~ashutoshc], log looks like this:

{noformat}
...
2016-05-19T04:25:14,196 DEBUG [141c5742-9b33-4e3f-a1d8-55c52a74d5cb main[]]: log.PerfLogger
(PerfLogger.java:PerfLogBegin(130)) - <PERFLOG method=optimizer from=org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction>
2016-05-19T04:25:14,196 DEBUG [141c5742-9b33-4e3f-a1d8-55c52a74d5cb main[]]: hive.log (MetaStoreUtils.java:getDDLFromFieldSchema(878))
- DDL: struct src { string key, string value}
2016-05-19T04:25:14,197 DEBUG [141c5742-9b33-4e3f-a1d8-55c52a74d5cb main[]]: parse.CalcitePlanner
(CalcitePlanner.java:genLogicalPlan(3299)) - Created Plan for Query Block t1_v
2016-05-19T04:25:14,198 DEBUG [141c5742-9b33-4e3f-a1d8-55c52a74d5cb main[]]: log.PerfLogger
(PerfLogger.java:PerfLogEnd(170)) - </PERFLOG method=optimizer start=1463657114196 end=1463657114198
duration=2 from=org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction Calcite:
Plan generation - Build RelNode for subqueries>
2016-05-19T04:25:14,198 DEBUG [141c5742-9b33-4e3f-a1d8-55c52a74d5cb main[]]: log.PerfLogger
(PerfLogger.java:PerfLogBegin(130)) - <PERFLOG method=optimizer from=org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction>
2016-05-19T04:25:14,198 DEBUG [141c5742-9b33-4e3f-a1d8-55c52a74d5cb main[]]: log.PerfLogger
(PerfLogger.java:PerfLogEnd(170)) - </PERFLOG method=optimizer start=1463657114198 end=1463657114198
duration=0 from=org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction Calcite:
Plan generation - Build RelNode for table scan>
2016-05-19T04:25:14,198 DEBUG [141c5742-9b33-4e3f-a1d8-55c52a74d5cb main[]]: log.PerfLogger
(PerfLogger.java:PerfLogBegin(130)) - <PERFLOG method=optimizer from=org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction>
2016-05-19T04:25:14,198 DEBUG [141c5742-9b33-4e3f-a1d8-55c52a74d5cb main[]]: log.PerfLogger
(PerfLogger.java:PerfLogEnd(170)) - </PERFLOG method=optimizer start=1463657114198 end=1463657114198
duration=0 from=org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction Calcite:
Plan generation - Build RelNode for join>
2016-05-19T04:25:14,198 DEBUG [141c5742-9b33-4e3f-a1d8-55c52a74d5cb main[]]: log.PerfLogger
(PerfLogger.java:PerfLogBegin(130)) - <PERFLOG method=optimizer from=org.apache.hadoop.hive.ql.parse.CalcitePlanner$CalcitePlannerAction>
...
{noformat}

If we want to obtain more meaningful information e.g. recursive methods, multiple calls to
same method, etc., PerfLogger infra should be extended. For instance, allow to aggregate the
time spent in a certain method using keys to recognize them. Then at the end of query execution,
dump all that measurements. Or even at the end of a given session.

> Add PerfLog information to genLogicalPlan in CalcitePlanner
> -----------------------------------------------------------
>
>                 Key: HIVE-13757
>                 URL: https://issues.apache.org/jira/browse/HIVE-13757
>             Project: Hive
>          Issue Type: Improvement
>          Components: CBO
>    Affects Versions: 2.1.0
>            Reporter: Jesus Camacho Rodriguez
>            Assignee: Jesus Camacho Rodriguez
>            Priority: Minor
>         Attachments: HIVE-13757.patch
>
>
> This will help us to have more accurate measurements of the different methods used to
generate the RelNode tree for the query.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message