sqoop-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jarek Jarcec Cecho (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SQOOP-1821) DOC: Add instructions on how-to create a external connector and make it work in Sqoop
Date Wed, 03 Dec 2014 19:15:13 GMT

    [ https://issues.apache.org/jira/browse/SQOOP-1821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14233373#comment-14233373
] 

Jarek Jarcec Cecho commented on SQOOP-1821:
-------------------------------------------

Tomcat is doing a lot of hairy stuff with classpath, so I'm not that surprised that we might
be having troubles to load additional connectors correctly. That is probably something that
we should have looked into before :-/

Here is my guess why altering {{common.loader}} in Tomcat configuration is not working. We're
loading connectors by exploring [our own {{ClassLoader}}|https://github.com/apache/sqoop/blob/sqoop2/core/src/main/java/org/apache/sqoop/connector/ConnectorManagerUtils.java#L44],
but the {{common.loader}} is one of the parent {{ClassLoader}} and I'm not sure whether the
call {{getResources()}} is recursively trying to search the parent {{ClassLoader}} as well.
If my guess is correct then I doubt that we have good way how to load extra connectors without
doing code changes or putting the extra connectors directly into our war archive.

If the {{getResource()}} call is not recursive, then we could perhaps try to traverse {{ClassLoader}}
chain ourselves. I'm afraid that it might not work correctly though as I would expect that
instantiating connector from parent {{ClassLoader}} will fail on missing dependencies that
are in the child {{ClassLoader}} and hence they are not available. I've dealt with similar
issues back in SQOOP-1232 when I was introducing the tools.

Another option might be to provide configuration property that will contain list of file system
paths that Sqoop should actively search for jar files that are valid connectors. This way
we could simply load the jars ourselves to the correct {{ClassLoader}} and be "done" with
it (in addition to the current way of exploring Classpath).

I'm pretty sure that there are other ways how to go about it, so don't hesitate to add your
own suggestion!



> DOC: Add instructions on how-to create a external connector and make it work in Sqoop
> -------------------------------------------------------------------------------------
>
>                 Key: SQOOP-1821
>                 URL: https://issues.apache.org/jira/browse/SQOOP-1821
>             Project: Sqoop
>          Issue Type: Bug
>    Affects Versions: 1.99.5
>            Reporter: Veena Basavaraj
>             Fix For: 1.99.5
>
>
> Edit this wiki to begin with
> https://cwiki.apache.org/confluence/display/SQOOP/Setting+up+Sqoop+2#SettingupSqoop2-InstallinganewconnectortoSqoop2
> Also, add the same to the .rst of need be
> relevant code that loads all connectors 
> https://github.com/apache/sqoop/blob/sqoop2/core/src/main/java/org/apache/sqoop/connector/ConnectorManagerUtils.java#L44
> NOTE : External contributors had issues with getting a new sqoop connector working seamlessly
with the Sqoop2. They had to do some hacks to include this new jar into the class path. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message