hive-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Prasanth Jayachandran (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (HIVE-19494) Accept shade prefix during reflective instantiation of output format
Date Thu, 10 May 2018 22:00:00 GMT

    [ https://issues.apache.org/jira/browse/HIVE-19494?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16471192#comment-16471192
] 

Prasanth Jayachandran commented on HIVE-19494:
----------------------------------------------

[~sershe] can you please take a look? small patch that accepts a prefix for classloading.


> Accept shade prefix during reflective instantiation of output format
> --------------------------------------------------------------------
>
>                 Key: HIVE-19494
>                 URL: https://issues.apache.org/jira/browse/HIVE-19494
>             Project: Hive
>          Issue Type: Sub-task
>          Components: Streaming
>    Affects Versions: 3.0.0, 3.1.0
>            Reporter: Prasanth Jayachandran
>            Assignee: Prasanth Jayachandran
>            Priority: Major
>         Attachments: HIVE-19494.1.patch
>
>
> Hive Streaming API jars are sometimes shaded with a different prefix when used in environments
where another version of hive already exists (spark for example). In most cases, shading is
done with rename of classes with some prefix. If an uber/assembly jar is generated with renamed
prefix, Hive Streaming API will not work as Hive Streaming API will reflectively instantiate
outputformat class using FQCN string provided by metastore table storage descriptor object.

> For example: 
> RecordWriter will create instance of OutputFormat using string "org.apache.hadoop.hive.ql.io.orc.OrcOutputFormat".
When a shaded jar with renamed class references are used, this class will not be found by
the classloader. 
> We can optionally accept a shade prefix from user via config which will be tried (as
fallback) when ClassNotFoundException is thrown.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

Mime
View raw message