sqoop-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Alvin Ge (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (SQOOP-2496) Sqoop2: Provide a way to inject external connectors
Date Thu, 21 Apr 2016 15:17:25 GMT

    [ https://issues.apache.org/jira/browse/SQOOP-2496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15252039#comment-15252039
] 

Alvin Ge commented on SQOOP-2496:
---------------------------------

[file:///E:/sqoop-branch-1.99.6/submission/mapreduce/src/main/java/org/apache/sqoop/submission/mapreduce/MapreduceSubmissionEngine.java]
Just like this?

> Sqoop2: Provide a way to inject external connectors
> ---------------------------------------------------
>
>                 Key: SQOOP-2496
>                 URL: https://issues.apache.org/jira/browse/SQOOP-2496
>             Project: Sqoop
>          Issue Type: Bug
>    Affects Versions: 1.99.6
>            Reporter: Jarek Jarcec Cecho
>             Fix For: 1.99.7
>
>
> On internal hackathon we we're hacking Sqoop 2 connector with [~singhashish] and we went
through few troubles that we should address.
> We have a a [configuration property|https://github.com/apache/sqoop/blob/sqoop2/dist/src/main/server/conf/sqoop.properties#L173]
for extra directory from which we will load jar files. We were able to use this configuration
property to load our hacked connector to Server, but we were not able to get it working through
job submission. Here is the exception that we hit:
> {code} 
> java.lang.IllegalArgumentException: java.net.URISyntaxException: Relative path in absolute
URI: jar:file://var/lib/sqoop/connectors/Connector-1.0-SNAPSHOT.jar!
>         at org.apache.hadoop.fs.Path.initialize(Path.java:206)
>         at org.apache.hadoop.fs.Path.<init>(Path.java:172)
>         at org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:215)
>         at org.apache.hadoop.mapreduce.JobSubmitter.copyAndConfigureFiles(JobSubmitter.java:390)
>         at org.apache.hadoop.mapreduce.JobSubmitter.submitJobInternal(JobSubmitter.java:483)
>         at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1306)
>         at org.apache.hadoop.mapreduce.Job$10.run(Job.java:1303)
>         at java.security.AccessController.doPrivileged(Native Method)
>         at javax.security.auth.Subject.doAs(Subject.java:415)
>         at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1671)
>         at org.apache.hadoop.mapreduce.Job.submit(Job.java:1303)
>         at org.apache.sqoop.submission.mapreduce.MapreduceSubmissionEngine.submitToCluster(MapreduceSubmissionEngine.java:274)
>         at org.apache.sqoop.submission.mapreduce.MapreduceSubmissionEngine.submit(MapreduceSubmissionEngine.java:255)
>         at org.apache.sqoop.driver.JobManager.start(JobManager.java:288)
>         at org.apache.sqoop.handler.JobRequestHandler.startJob(JobRequestHandler.java:380)
>         at org.apache.sqoop.handler.JobRequestHandler.handleEvent(JobRequestHandler.java:116)
>         at org.apache.sqoop.server.v1.JobServlet.handlePutRequest(JobServlet.java:96)
>         at org.apache.sqoop.server.SqoopProtocolServlet.doPut(SqoopProtocolServlet.java:79)
>         at javax.servlet.http.HttpServlet.service(HttpServlet.java:646)
>         at javax.servlet.http.HttpServlet.service(HttpServlet.java:723)
> {code}
> To put into nutshell, [ClassUtils.jarForClass|https://github.com/apache/sqoop/blob/sqoop2/common/src/main/java/org/apache/sqoop/utils/ClassUtils.java#L136]
returns for external connectors path starting with prefix {{jar:file}} and suffix {{!}} that
breaks mapreduce code. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message