flink-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Abhishek Agarwal (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (FLINK-3271) Using webhdfs in a flink topology throws classnotfound exception
Date Thu, 21 Jan 2016 18:53:40 GMT

     [ https://issues.apache.org/jira/browse/FLINK-3271?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Abhishek Agarwal updated FLINK-3271:
------------------------------------
    Description: 
I was just trying to run a storm topology on flink using flink-storm. I got this exception
- 

{noformat}
Caused by: java.lang.NoClassDefFoundError: org/mortbay/util/ajax/JSON
	at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.jsonParse(WebHdfsFileSystem.java:325)
	at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$FsPathResponseRunner.getResponse(WebHdfsFileSystem.java:727)
	at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.runWithRetry(WebHdfsFileSystem.java:610)
	at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.access$100(WebHdfsFileSystem.java:458)
	at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner$1.run(WebHdfsFileSystem.java:487)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
	at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.run(WebHdfsFileSystem.java:483)
	at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.listStatus(WebHdfsFileSystem.java:1277)
{noformat}

My topology list some files on hdfs using webhdfs API. org.mortbay.util.ajax.JSON was included
in the application uber jar. I noticed that flink loads the application jar in a child classloader.
This is what most likely happened - 

1. WebHdfsFileSystem class was loaded through parent class loader since it is included in
flink-dist.jar.
2. WebHdfsFileSystem has reference to the org.mortbay.util.ajax.JSON but since it is loaded
through parent class loader, WebHdfsFileSystem can't read a class in child class loader. 

Ideally all the referenced classes should be available in the distribution jar so that these
sort of issues may not occur. 

  was:
I was just trying to run a storm topology on flink using flink-storm. I got this exception
- 

{noformat}
Caused by: java.lang.NoClassDefFoundError: org/mortbay/util/ajax/JSON
	at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.jsonParse(WebHdfsFileSystem.java:325)
	at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$FsPathResponseRunner.getResponse(WebHdfsFileSystem.java:727)
	at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.runWithRetry(WebHdfsFileSystem.java:610)
	at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.access$100(WebHdfsFileSystem.java:458)
	at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner$1.run(WebHdfsFileSystem.java:487)
	at java.security.AccessController.doPrivileged(Native Method)
	at javax.security.auth.Subject.doAs(Subject.java:422)
	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
	at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.run(WebHdfsFileSystem.java:483)
	at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.listStatus(WebHdfsFileSystem.java:1277)
{noformat}

My topology list some files on hdfs using webhdfs API. org.mortbay.util.ajax.JSON is included
in the uber jar. I noticed that flink loads the application jar in a child classloader. This
is what most likely happened - 

1. WebHdfsFileSystem class was loaded through parent class loader since it is included in
flink-dist.jar.
2. WebHdfsFileSystem has reference to the org.mortbay.util.ajax.JSON but since it is loaded
through parent class loader, WebHdfsFileSystem can't read a class in child class loader. 

Ideally all the referenced classes should be available in the distribution jar so that these
sort of issues may not occur. 


> Using webhdfs in a flink topology throws classnotfound exception
> ----------------------------------------------------------------
>
>                 Key: FLINK-3271
>                 URL: https://issues.apache.org/jira/browse/FLINK-3271
>             Project: Flink
>          Issue Type: Bug
>          Components: Core
>    Affects Versions: 0.10.1
>            Reporter: Abhishek Agarwal
>
> I was just trying to run a storm topology on flink using flink-storm. I got this exception
- 
> {noformat}
> Caused by: java.lang.NoClassDefFoundError: org/mortbay/util/ajax/JSON
> 	at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.jsonParse(WebHdfsFileSystem.java:325)
> 	at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$FsPathResponseRunner.getResponse(WebHdfsFileSystem.java:727)
> 	at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.runWithRetry(WebHdfsFileSystem.java:610)
> 	at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.access$100(WebHdfsFileSystem.java:458)
> 	at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner$1.run(WebHdfsFileSystem.java:487)
> 	at java.security.AccessController.doPrivileged(Native Method)
> 	at javax.security.auth.Subject.doAs(Subject.java:422)
> 	at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1628)
> 	at org.apache.hadoop.hdfs.web.WebHdfsFileSystem$AbstractRunner.run(WebHdfsFileSystem.java:483)
> 	at org.apache.hadoop.hdfs.web.WebHdfsFileSystem.listStatus(WebHdfsFileSystem.java:1277)
> {noformat}
> My topology list some files on hdfs using webhdfs API. org.mortbay.util.ajax.JSON was
included in the application uber jar. I noticed that flink loads the application jar in a
child classloader. This is what most likely happened - 
> 1. WebHdfsFileSystem class was loaded through parent class loader since it is included
in flink-dist.jar.
> 2. WebHdfsFileSystem has reference to the org.mortbay.util.ajax.JSON but since it is
loaded through parent class loader, WebHdfsFileSystem can't read a class in child class loader.

> Ideally all the referenced classes should be available in the distribution jar so that
these sort of issues may not occur. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message