hive-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Shuaishuai Nie (JIRA)" <>
Subject [jira] [Updated] (HIVE-5072) [WebHCat]Enable directly invoke Sqoop job through Templeton
Date Tue, 15 Apr 2014 01:18:14 GMT


Shuaishuai Nie updated HIVE-5072:

    Attachment: HIVE-5072.4.patch

Updated the patch based on the comment.
[~ekoifman], here are the comments for the questions:
Hi [~ekoifman].
5. Your implementation supports automatically shipping of Sqoop tar file to the target node
where the job will be executed. The tar file doesn't contain JDBC drivers for all DB types
so you can't expect it to be in the "lib" folder on target node.
If the user is using the archive, They need to add the jdbc in the archive file before passing
it. JDBC driver jar is a third party add-on for Sqoop,and also provide the flexibility for
Sqoop to connect to any DB supporting JDBC, so user need to provide the JDBC driver jar based
on the DB they use.

7. Since this API would mostly be used by Sqoop users, I think they'll find it confusing that
items with the same name in Sqoop have different meaning in WebHCat/Sqoop. I would try to
either change the names of parameters or make the definition in the 2 systems match.
This make sense, I will change the name of the parameter. Also I will create a separate JIRA
for the documentation after the patch got committed.

9. Does your implementation work with Sqoop 1 and Sqoop 2? If not, it should probably assert
the right version is installed/tar file is available and produce a meaningful error to client.
The implementation only works with Sqoop 1. Since the usage of Sqoop 1 and Sqoop 2 differs
a lot (Sqoop 2 need to create connection, create job for import/export) and the way to pass
command in Sqoop 1 is not working in Sqoop 2, so it is not likely user will use this function
against Sqoop 2. I think the best way is to specify that this endpoint only support Sqoop
1 in the documentation.

> [WebHCat]Enable directly invoke Sqoop job through Templeton
> -----------------------------------------------------------
>                 Key: HIVE-5072
>                 URL:
>             Project: Hive
>          Issue Type: Improvement
>          Components: WebHCat
>    Affects Versions: 0.12.0
>            Reporter: Shuaishuai Nie
>            Assignee: Shuaishuai Nie
>         Attachments: HIVE-5072.1.patch, HIVE-5072.2.patch, HIVE-5072.3.patch, HIVE-5072.4.patch,
> Now it is hard to invoke a Sqoop job through templeton. The only way is to use the classpath
jar generated by a sqoop job and use the jar delegator in Templeton. We should implement Sqoop
Delegator to enable directly invoke Sqoop job through Templeton.

This message was sent by Atlassian JIRA

View raw message