hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Rui Li (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-15259) The deserialization time of HOS20 is longer than what in HOS16
Date Tue, 22 Nov 2016 10:07:58 GMT

    [ https://issues.apache.org/jira/browse/HIVE-15259?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15686282#comment-15686282
] 

Rui Li commented on HIVE-15259:
-------------------------------

With Spark 2.0, you don't have to copy all the jars to Hive lib. Please refer to our wiki
https://cwiki.apache.org/confluence/display/Hive/Hive+on+Spark%3A+Getting+Started.

The log you posted is from Spark. I guess it uploads the required jars to prepare the classpath
for containers. Not sure if it's related to the deserialization time. And on Hive side we
don't have much control on it - we basically just specify that hive-exec is needed. Others
are up to Spark.

> The deserialization time of HOS20 is longer than what in  HOS16
> ---------------------------------------------------------------
>
>                 Key: HIVE-15259
>                 URL: https://issues.apache.org/jira/browse/HIVE-15259
>             Project: Hive
>          Issue Type: Improvement
>            Reporter: liyunzhang_intel
>         Attachments: Deserialization_HOS16.PNG, Deserialization_HOS20.PNG
>
>
> deploy Hive on Spark on spark 1.6 version and spark 2.0 version.
> run query and in latest code(with spark2.0) the deserialization time of a task is 4 sec
while the deserialization time of spark1.6 is 1 sec. The detail is in attached picture.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message