hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Venki Korukanti (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-7747) Submitting a query to Spark from HiveServer2 fails
Date Wed, 20 Aug 2014 10:03:26 GMT

    [ https://issues.apache.org/jira/browse/HIVE-7747?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14103706#comment-14103706
] 

Venki Korukanti commented on HIVE-7747:
---------------------------------------

Test failure here is related to the change. Failure is complicated. It turns out that output
of {{HiveConf(srcHiveConf, SessionState.class)}} is not same as srcHiveConf in terms of (property,
value) pairs. Executed as part of constructor, the {{HiveConf.initialize}} method applies
system properties on top of copied properties from srcHiveConf. So from the moment srcHiveConf
is created to the moment of cloning HiveConf if there are any System properties set, cloned
HiveConf inherits those properties. In the test case ({{MiniHS2}}) scratchdir property is
modified in System properties (See [here|https://github.com/apache/hive/blob/trunk/itests/hive-unit/src/main/java/org/apache/hive/jdbc/miniHS2/MiniHS2.java#L184]),
but the default scratchdir value is {{$\{test.tmp.dir\}/scratchdir}} from hive-site.xml. Scrathdir
set in {{MiniHS2}} is never used before, but with this change HS2 started using it. Scratchdir
created in {{MiniHS2}} (See [here|https://github.com/apache/hive/blob/trunk/itests/hive-unit/src/main/java/org/apache/hive/jdbc/miniHS2/MiniHS2.java#L183])
doesn't have 777 permissions, so whenever we have user impersonation there are issues (thats
where the test is failing). Before this change, scratchdir is always {{$\{test.tmp.dir\}/scratchdir}}
which is created in HS2 with 777 permissions (See [here|https://github.com/apache/hive/blob/trunk/ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java#L3458]),
so there were no issues with the impersonation. 

I think it is better to fix this in SparkClient by fetching the jar directly than through
HiveConf, to avoid unexpected issues.



> Submitting a query to Spark from HiveServer2 fails
> --------------------------------------------------
>
>                 Key: HIVE-7747
>                 URL: https://issues.apache.org/jira/browse/HIVE-7747
>             Project: Hive
>          Issue Type: Bug
>          Components: Spark
>    Affects Versions: 0.13.1
>            Reporter: Venki Korukanti
>            Assignee: Venki Korukanti
>         Attachments: HIVE-7747.1.patch
>
>
> {{spark.serializer}} is set to {{org.apache.spark.serializer.KryoSerializer}}. Same configuration
works fine from Hive CLI.
> Spark tasks fails with following error:
> {code}
> Job aborted due to stage failure: Task 0 in stage 1.0 failed 4 times, most recent failure:
Lost task 0.3 in stage 1.0 (TID 9, 192.168.168.216): java.lang.IllegalStateException: unread
block data
>         java.io.ObjectInputStream$BlockDataInputStream.setBlockDataMode(ObjectInputStream.java:2421)
>         java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1382)
>         java.io.ObjectInputStream.defaultReadFields(ObjectInputStream.java:1990)
>         java.io.ObjectInputStream.readSerialData(ObjectInputStream.java:1915)
>         java.io.ObjectInputStream.readOrdinaryObject(ObjectInputStream.java:1798)
>         java.io.ObjectInputStream.readObject0(ObjectInputStream.java:1350)
>         java.io.ObjectInputStream.readObject(ObjectInputStream.java:370)
>         org.apache.spark.serializer.JavaDeserializationStream.readObject(JavaSerializer.scala:62)
>         org.apache.spark.serializer.JavaSerializerInstance.deserialize(JavaSerializer.scala:84)
>         org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:181)
>         java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>         java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>         java.lang.Thread.run(Thread.java:744)
> {code}



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message