incubator-oozie-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jeremy Hanna <jeremy.hanna1...@gmail.com>
Subject Re: using oozie with datastax brisk
Date Sat, 22 Oct 2011 14:26:05 GMT
Thanks for the response Mayank.  Is there a particular fix on trunk or something you're talking
about?

The full exception in the logs from the error I mentioned is:
2011-10-21 21:03:55,627  INFO BaseJobServlet:536 - USER[?] GROUP[users] TOKEN[-] APP[-] JOB[-]
ACTION[-] AuthorizationException
org.apache.oozie.service.AuthorizationException: E0902: Exception occured: [java.io.IOException:
No FileSystem for scheme: cfs]
        at org.apache.oozie.service.AuthorizationService.authorizeForApp(AuthorizationService.java:320)
        at org.apache.oozie.servlet.BaseJobServlet.checkAuthorizationForApp(BaseJobServlet.java:185)
        at org.apache.oozie.servlet.BaseJobsServlet.doPost(BaseJobsServlet.java:89)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:637)
        at org.apache.oozie.servlet.JsonRestServlet.service(JsonRestServlet.java:281)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:717)
        at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:290)
        at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:206)
        at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:233)
        at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:191)
        at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
        at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:102)
        at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:109)
        at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:298)
        at org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:859)
        at org.apache.coyote.http11.Http11Protocol$Http11ConnectionHandler.process(Http11Protocol.java:588)
        at org.apache.tomcat.util.net.JIoEndpoint$Worker.run(JIoEndpoint.java:489)
        at java.lang.Thread.run(Thread.java:662)
Caused by: org.apache.oozie.service.HadoopAccessorException: E0902: Exception occured: [java.io.IOException:
No FileSystem for scheme: cfs]
        at org.apache.oozie.service.KerberosHadoopAccessorService.createFileSystem(KerberosHadoopAccessorService.java:208)
        at org.apache.oozie.service.AuthorizationService.authorizeForApp(AuthorizationService.java:285)
        ... 17 more
Caused by: java.io.IOException: No FileSystem for scheme: cfs
        at org.apache.hadoop.fs.FileSystem.createFileSystem(FileSystem.java:1307)
        at org.apache.hadoop.fs.FileSystem.access$100(FileSystem.java:65)
        at org.apache.hadoop.fs.FileSystem$Cache.get(FileSystem.java:1328)
        at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:226)
        at org.apache.oozie.service.KerberosHadoopAccessorService$3.run(KerberosHadoopAccessorService.java:200)
        at org.apache.oozie.service.KerberosHadoopAccessorService$3.run(KerberosHadoopAccessorService.java:192)
        at java.security.AccessController.doPrivileged(Native Method)
        at javax.security.auth.Subject.doAs(Subject.java:396)
        at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1059)
        at org.apache.oozie.service.KerberosHadoopAccessorService.createFileSystem(KerberosHadoopAccessorService.java:192)
        ... 18 more

I seem to have gotten past that part, though I've needed to get some of the Cassandra libs
in there that cfs is dependent on.  But I may not be seeing all the errors yet.  Is there
a release planned with the fix that you're talking about or a revision or ticket or patch
I can apply?

Thanks,

Jeremy

On Oct 22, 2011, at 12:32 AM, Mayank Bansal wrote:

> I am not sure whether you will be able to run any other file system except
> hdfs. I do not think we have put the configurable file systems fix in 3.0.2.
> 
> Send the Oozie log for more details.
> 
> Thanks,
> Mayank
> 
> On Fri, Oct 21, 2011 at 7:37 PM, Jeremy Hanna <jeremy.hanna1234@gmail.com>wrote:
> 
>> Looks like I need to put that core-site.xml and the jar that contains
>> CassandraFileSystem into the oozie/oozie-server/lib directory.  Still trying
>> to track down a 500 error now, but at least I've gotten past that initial
>> issue.
>> 
>> On Oct 21, 2011, at 4:20 PM, Jeremy Hanna wrote:
>> 
>>> I've been trying to setup oozie with datastax's brisk product.  I am
>> using oozie 3.0.2 and set it up by using:
>>> bin/oozie-setup.sh -hadoop 0.20.200 /opt/brisk/resources/hadoop -extjs
>> ./ext-2.2.zip
>>> 
>>> Technically brisk uses hadoop 0.20.203 but oozie didn't like that.
>>> 
>>> So in the brisk core-site.xml it defines:
>>> <property>
>>>   <name>fs.cfs.impl</name>
>>>   <value>com.datastax.bdp.hadoop.cfs.CassandraFileSystem</value>
>>> </property>
>>> 
>>> and then I use that in my fs.default.name
>>> <property>
>>>   <name>fs.default.name</name>
>>>   <value>cfs://myserver.com</value>
>>> </property>
>>> 
>>> however when I try to run one of the examples such as this (with nameNode
>> set to cfs://myserver.com):
>>> oozie job -oozie http://localhost:11000/oozie -config
>> examples/apps/map-reduce/job.properties -run
>>> 
>>> it gives the error:
>>> Error: E0902 : E0902: Exception occured: [java.io.IOException: No
>> FileSystem for scheme: cfs]
>>> 
>>> Shouldn't oozie have picked up the core-site.xml from the hadoop that it
>> picked up when running oozie-setup.sh and therefore been able to pick up the
>> fs.cfs.impl property?
>>> 
>>> Thanks!
>>> 
>>> Jeremy
>> 
>> 
> 
> 
> -- 
> Thanks and Regards,
> Mayank
> Cell: 408-718-9370


Mime
View raw message