hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Chengxiang Li (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-8855) Automatic calculate reduce number for spark job[Spark Branch]
Date Tue, 25 Nov 2014 09:58:13 GMT

    [ https://issues.apache.org/jira/browse/HIVE-8855?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14224306#comment-14224306
] 

Chengxiang Li commented on HIVE-8855:
-------------------------------------

Hi, [~jxiang], I hit this exception with the path.
{noformat}
onReceive(314)) - Received hello from akka.tcp://79ec2601-f6e9-40de-b7bf-6b5760126a28@192.168.150.114:33902/user/RemoteDriver
2014-11-25 17:40:01,502 ERROR [94df45b9-4960-487f-ab39-6b659fc6b899-akka.actor.default-dispatcher-4]:
remote.EndpointWriter (Slf4jLogger.scala:apply$mcV$sp(66)) - Transient association error (association
remains live)
java.io.NotSerializableException: java.lang.Thread
        at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1183)
        at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1547)
        at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1508)
        at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1431)
        at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177)
        at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1547)
        at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1508)
        at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1431)
        at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177)
        at java.io.ObjectOutputStream.defaultWriteFields(ObjectOutputStream.java:1547)
        at java.io.ObjectOutputStream.writeSerialData(ObjectOutputStream.java:1508)
        at java.io.ObjectOutputStream.writeOrdinaryObject(ObjectOutputStream.java:1431)
        at java.io.ObjectOutputStream.writeObject0(ObjectOutputStream.java:1177)
        at java.io.ObjectOutputStream.writeObject(ObjectOutputStream.java:347)
        at akka.serialization.JavaSerializer$$anonfun$toBinary$1.apply$mcV$sp(Serializer.scala:129)
        at akka.serialization.JavaSerializer$$anonfun$toBinary$1.apply(Serializer.scala:129)
        at akka.serialization.JavaSerializer$$anonfun$toBinary$1.apply(Serializer.scala:129)
        at scala.util.DynamicVariable.withValue(DynamicVariable.scala:57)
        at akka.serialization.JavaSerializer.toBinary(Serializer.scala:129)
        at akka.remote.serialization.MessageContainerSerializer.serializeSelection(MessageContainerSerializer.scala:36)
        at akka.remote.serialization.MessageContainerSerializer.toBinary(MessageContainerSerializer.scala:25)
        at akka.remote.MessageSerializer$.serialize(MessageSerializer.scala:36)
        at akka.remote.EndpointWriter$$anonfun$serializeMessage$1.apply(Endpoint.scala:845)
        at akka.remote.EndpointWriter$$anonfun$serializeMessage$1.apply(Endpoint.scala:845)
        at scala.util.DynamicVariable.withValue(DynamicVariable.scala:57)
        at akka.remote.EndpointWriter.serializeMessage(Endpoint.scala:844)
        at akka.remote.EndpointWriter.writeSend(Endpoint.scala:747)
        at akka.remote.EndpointWriter$$anonfun$2.applyOrElse(Endpoint.scala:722)
        at akka.actor.Actor$class.aroundReceive(Actor.scala:465)
        at akka.remote.EndpointActor.aroundReceive(Endpoint.scala:415)
        at akka.actor.ActorCell.receiveMessage(ActorCell.scala:516)
        at akka.actor.ActorCell.invoke(ActorCell.scala:487)
        at akka.dispatch.Mailbox.processMailbox(Mailbox.scala:238)
        at akka.dispatch.Mailbox.run(Mailbox.scala:220)
        at akka.dispatch.ForkJoinExecutorConfigurator$AkkaForkJoinTask.exec(AbstractDispatcher.scala:393)
        at scala.concurrent.forkjoin.ForkJoinTask.doExec(ForkJoinTask.java:260)
        at scala.concurrent.forkjoin.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1339)
        at scala.concurrent.forkjoin.ForkJoinPool.runWorker(ForkJoinPool.java:1979)
        at scala.concurrent.forkjoin.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:107)

{noformat}
It should because SparkClientImpl::getExecutorCount submit inner class job which make akka
serializer SparkClientImpl instance with Java Serializer, I think you may move the logic of
getExecutorCount to RemoteSparkClient.

> Automatic calculate reduce number for spark job[Spark Branch]
> -------------------------------------------------------------
>
>                 Key: HIVE-8855
>                 URL: https://issues.apache.org/jira/browse/HIVE-8855
>             Project: Hive
>          Issue Type: Sub-task
>          Components: Spark
>            Reporter: Chengxiang Li
>            Assignee: Jimmy Xiang
>              Labels: Spark-M3
>             Fix For: spark-branch
>
>         Attachments: HIVE-8855.1-spark.patch, HIVE-8855.2-spark.patch
>
>
> As the following up work of HIVE-8649, we should enable reduce number automatic calculation
for both local spark client and remote spark client.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message