hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chao (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (HIVE-9136) Profile query compiler [Spark Branch]
Date Wed, 17 Dec 2014 21:33:16 GMT

     [ https://issues.apache.org/jira/browse/HIVE-9136?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Chao updated HIVE-9136:
-----------------------
    Status: Patch Available  (was: Open)

Patch v1. I added several spark-specific log events to {{PerfLogger}}. The correspondence
agains Tez is:

|| In Tez || In Spark ||
| TEZ_SUBMIT_TO_RUNNING | SPARK_SUBMIT_TO_RUNNING |
| TEZ_BUILD_DAG | SPARK_BUILD_PLAN + SPARK_BUILD_RDD_GRAPH|
| TEZ_SUBMIT_DAG | SPARK_SUBMIT_JOB |
| TEZ_RUN_DAG | SPARK_RUN_JOB |
| TEZ_CREATE_VERTEX | SPARK_CREATE_TRAN |
| TEZ_RUN_VERTEX | SPARK_RUN_STAGE |
| TEZ_INIITIALIZE_PROCESSOR | ? |
| TEZ_RUN_PROCESSOR | ? |
| TEZ_INITIALIZE_OPERATORS | SPARK_INITIALIZE_OPERATORS |

For TEZ_INITIALIZE_PROCESSOR and TEZ_RUN_PROCESSOR, I didn't find correspondence in our Spark
branch. Any idea? Maybe log the {{SparkBaseFunctionResultList}}?

In addition, I added SPARK_FLUSH_HASHTABLE, to track perf on Spark hash table sink, and SPARK_GENERATE_OPERATOR_TREE,
to track perf on, as the name suggests, generating operator tree.

I'm also open to any kind of suggestions.



> Profile query compiler [Spark Branch]
> -------------------------------------
>
>                 Key: HIVE-9136
>                 URL: https://issues.apache.org/jira/browse/HIVE-9136
>             Project: Hive
>          Issue Type: Sub-task
>          Components: Spark
>    Affects Versions: spark-branch
>            Reporter: Brock Noland
>            Assignee: Chao
>         Attachments: HIVE-9136.1.patch
>
>
> We should put some performance counters around the compiler and evaluate how long it
takes to compile a query in Spark versus the other execution frameworks. Query 28 is a good
one to use for testing.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message