hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Rui Li (JIRA)" <>
Subject [jira] [Updated] (HIVE-8991) Fix custom_input_output_format [Spark Branch]
Date Fri, 28 Nov 2014 15:27:12 GMT


Rui Li updated HIVE-8991:
    Attachment: HIVE-8991.1-spark.patch

This patch can fix the test on my machine.
Strange thing is that, if I add hive-it-util to {{spark.driver.extraClassPath}}, I'll have
to add hive-exec to it as well, which I suppose should be always added to driver's class path
by default.
Looked a little bit into this. It seems spark will use {{SparkSubmitDriverBootstrapper}} to
launch {{SparkSubmit}} if there're {{spark.driver.extra*}} properties. So I suspect {{SparkSubmitDriverBootstrapper}}
somehow doesn't set CP properly for the driver.
Also tried setting {{--driver-class-path}} in {{SparkClientImpl}}, but it will override {{spark.driver.extraClassPath}}.
Another thing is that {{SparkSubmitDriverBootstrapper}} just hangs after client and driver
have shut down. Others may help me verify this.

> Fix custom_input_output_format [Spark Branch]
> ---------------------------------------------
>                 Key: HIVE-8991
>                 URL:
>             Project: Hive
>          Issue Type: Sub-task
>          Components: Spark
>            Reporter: Rui Li
>            Assignee: Rui Li
>         Attachments: HIVE-8991.1-spark.patch
> After HIVE-8836, {{custom_input_output_format}} fails because of missing hive-it-util
in remote driver's class path.

This message was sent by Atlassian JIRA

View raw message