hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jani Arvonen <j...@floobs.com>
Subject Re: newbie problem: Using DFS from Java program
Date Wed, 22 Aug 2007 07:09:06 GMT
Oh yes, of course. Our web application throws this exception when trying
to use copyFromLocalFile() through DistributedFileSystem. The local file
im trying to copy to the HDFS do exists...

And we have also tried to use LocalFileSystem which works fine but that
is not exactly what we are trying to achieve :)

stacktrace:

java.lang.NullPointerException
	at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:155)
	at org.apache.hadoop.fs.FileSystem.getNamed(FileSystem.java:119)
	at
org.apache.hadoop.fs.ChecksumFileSystem.copyFromLocalFile(ChecksumFileSystem.java:576)
	at
com.floobs.service.media.DefaultMediaStorage.copyFromLocalFile(DefaultMediaStorage.java:23)
	at
com.floobs.service.media.DefaultMediaStorage.copyFromLocalFile(DefaultMediaStorage.java:18)
	at
com.floobs.web.media.MediaUploadFormController.storeFile(MediaUploadFormController.java:155)
	at com.floobs.web.media.MediaUploadFormController.access
$500(MediaUploadFormController.java:36)
	at com.floobs.web.media.MediaUploadFormController
$2.processFile(MediaUploadFormController.java:120)
	at com.floobs.web.media.MediaUploadFormController
$UploadFileProcessor.execute(MediaUploadFormController.java:144)
	at
com.floobs.web.media.MediaUploadFormController.processFileUploads(MediaUploadFormController.java:114)
	at
com.floobs.web.media.MediaUploadFormController.onSubmit(MediaUploadFormController.java:65)
	at
org.springframework.web.servlet.mvc.SimpleFormController.processFormSubmission(SimpleFormController.java:267)
	at
org.springframework.web.servlet.mvc.AbstractFormController.handleRequestInternal(AbstractFormController.java:250)
	at
org.springframework.web.servlet.mvc.AbstractController.handleRequest(AbstractController.java:153)
	at
org.riotfamily.cachius.spring.CacheableControllerHandlerAdapter.handle(CacheableControllerHandlerAdapter.java:118)
	at
org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:839)
	at
org.riotfamily.common.web.servlet.ReloadableDispatcherServlet.doDispatch(ReloadableDispatcherServlet.java:107)
	at
org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:774)
	at
org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:460)
	at
org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:425)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:710)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
	at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:269)
	at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
	at
org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:691)
	at
org.apache.catalina.core.ApplicationDispatcher.doInclude(ApplicationDispatcher.java:594)
	at
org.apache.catalina.core.ApplicationDispatcher.include(ApplicationDispatcher.java:505)
	at
org.riotfamily.pages.component.impl.IncludeComponent.renderInternal(IncludeComponent.java:66)
	at
org.riotfamily.pages.component.impl.AbstractComponent.render(AbstractComponent.java:98)
	at
org.riotfamily.pages.component.render.AbstractRenderStrategy.renderComponent(AbstractRenderStrategy.java:243)
	at
org.riotfamily.pages.component.render.AbstractRenderStrategy.renderComponentVersion(AbstractRenderStrategy.java:207)
	at
org.riotfamily.pages.component.render.EditModeRenderStrategy.renderContainer(EditModeRenderStrategy.java:169)
	at
org.riotfamily.pages.component.render.AbstractRenderStrategy.renderComponents(AbstractRenderStrategy.java:156)
	at
org.riotfamily.pages.component.render.AbstractRenderStrategy.renderComponentList(AbstractRenderStrategy.java:137)
	at
org.riotfamily.pages.component.render.EditModeRenderStrategy.renderComponentList(EditModeRenderStrategy.java:93)
	at
org.riotfamily.pages.component.render.AbstractRenderStrategy.render(AbstractRenderStrategy.java:87)
	at
org.riotfamily.pages.component.render.AbstractRenderStrategy.render(AbstractRenderStrategy.java:81)
	at org.riotfamily.pages.component.ComponentListController
$1.doInTransactionWithoutResult(ComponentListController.java:202)
	at
org.springframework.transaction.support.TransactionCallbackWithoutResult.doInTransaction(TransactionCallbackWithoutResult.java:33)
	at
org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:127)
	at
org.riotfamily.pages.component.ComponentListController.handleRequest(ComponentListController.java:199)
	at
org.riotfamily.cachius.spring.CacheableControllerHandlerAdapter.handle(CacheableControllerHandlerAdapter.java:118)
	at
org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:839)
	at
org.riotfamily.common.web.servlet.ReloadableDispatcherServlet.doDispatch(ReloadableDispatcherServlet.java:107)
	at
org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:774)
	at
org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:460)
	at
org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:425)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:710)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
	at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:269)
	at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
	at
org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:691)
	at
org.apache.catalina.core.ApplicationDispatcher.doInclude(ApplicationDispatcher.java:594)
	at
org.apache.catalina.core.ApplicationDispatcher.include(ApplicationDispatcher.java:505)
	at
org.riotfamily.common.web.view.freemarker.IncludeMethod.exec(IncludeMethod.java:37)
	at
org.riotfamily.common.web.view.freemarker.AbstractSimpleMethod.exec(AbstractSimpleMethod.java:38)
	at freemarker.core.MethodCall._getAsTemplateModel(MethodCall.java:93)
	at freemarker.core.Expression.getAsTemplateModel(Expression.java:89)
	at freemarker.core.Expression.getStringValue(Expression.java:93)
	at freemarker.core.DollarVariable.accept(DollarVariable.java:76)
	at freemarker.core.Environment.visit(Environment.java:196)
	at freemarker.core.MixedContent.accept(MixedContent.java:92)
	at freemarker.core.Environment.visit(Environment.java:196)
	at freemarker.core.Macro$Context.runMacro(Macro.java:164)
	at freemarker.core.Environment.visit(Environment.java:537)
	at freemarker.core.UnifiedCall.accept(UnifiedCall.java:128)
	at freemarker.core.Environment.visit(Environment.java:196)
	at freemarker.core.MixedContent.accept(MixedContent.java:92)
	at freemarker.core.Environment.visit(Environment.java:196)
	at freemarker.core.Environment.process(Environment.java:176)
	at freemarker.template.Template.process(Template.java:232)
	at
org.springframework.web.servlet.view.freemarker.FreeMarkerView.processTemplate(FreeMarkerView.java:333)
	at
org.riotfamily.common.web.view.freemarker.RiotFreeMarkerView.doRender(RiotFreeMarkerView.java:113)
	at org.riotfamily.common.web.view.freemarker.RiotFreeMarkerView
$1.doInContext(RiotFreeMarkerView.java:93)
	at
org.riotfamily.common.web.view.ViewContext.execute(ViewContext.java:45)
	at
org.riotfamily.common.web.view.freemarker.RiotFreeMarkerView.renderMergedTemplateModel(RiotFreeMarkerView.java:91)
	at
org.springframework.web.servlet.view.AbstractTemplateView.renderMergedOutputModel(AbstractTemplateView.java:179)
	at
org.springframework.web.servlet.view.AbstractView.render(AbstractView.java:247)
	at
org.riotfamily.common.web.view.freemarker.RiotFreeMarkerView.render(RiotFreeMarkerView.java:75)
	at
org.springframework.web.servlet.DispatcherServlet.render(DispatcherServlet.java:1123)
	at
org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:860)
	at
org.riotfamily.common.web.servlet.ReloadableDispatcherServlet.doDispatch(ReloadableDispatcherServlet.java:107)
	at
org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:774)
	at
org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:460)
	at
org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:425)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:710)
	at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
	at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:269)
	at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
	at
org.riotfamily.common.web.filter.PluginChain.doFilter(PluginChain.java:60)
	at
org.riotfamily.pages.page.support.FolderFilterPlugin.doFilter(FolderFilterPlugin.java:82)
	at
org.riotfamily.common.web.filter.PluginChain.doFilter(PluginChain.java:57)
	at
org.riotfamily.riot.security.AccessControlFilterPlugin.doFilter(AccessControlFilterPlugin.java:58)
	at
org.riotfamily.common.web.filter.PluginChain.doFilter(PluginChain.java:57)
	at
org.riotfamily.common.web.filter.PluginFilter.doFilterInternal(PluginFilter.java:123)
	at
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:75)
	at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215)
	at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
	at
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:70)
	at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215)
	at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
	at
org.riotfamily.common.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:95)
	at
org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:75)
	at
org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215)
	at
org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
	at
org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:210)
	at
org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:174)
	at
org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
	at
org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117)
	at
org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:108)
	at
org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:151)
	at
org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:870)
	at org.apache.coyote.http11.Http11BaseProtocol
$Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:665)
	at
org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:528)
	at
org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:81)
	at org.apache.tomcat.util.threads.ThreadPool
$ControlRunnable.run(ThreadPool.java:685)
	at java.lang.Thread.run(Thread.java:619)



And our hadoop server logfile (datanode log) contains this several times
with different timestamp:

2007-08-21 00:00:23,393 INFO org.apache.hadoop.dfs.DataNode: using
BLOCKREPORT_INTERVAL of 3356962msec
2007-08-21 00:00:23,407 ERROR org.apache.hadoop.dfs.DataNode: Exception:
java.lang.NullPointerException
        at org.apache.hadoop.dfs.FSDataset
$FSDir.getBlockInfo(FSDataset.java:148)
        at org.apache.hadoop.dfs.FSDataset
$FSVolume.getBlockInfo(FSDataset.java:353)
        at org.apache.hadoop.dfs.FSDataset
$FSVolumeSet.getBlockInfo(FSDataset.java:419)
        at
org.apache.hadoop.dfs.FSDataset.getBlockReport(FSDataset.java:636)
        at
org.apache.hadoop.dfs.DataNode.offerService(DataNode.java:523)
        at org.apache.hadoop.dfs.DataNode.run(DataNode.java:1310)
        at java.lang.Thread.run(Thread.java:619)


but im not sure if this has anything to do with this particular problem.
Any help would be really appreciated!

Best regards,
Jani

On Tue, 2007-08-21 at 11:53 -0500, Sagar Naik wrote:
> Hello
> Can u provide some more information.
> Like a stackstrace from log files
> 
> - Sagar
> Jani Arvonen wrote:
> > Hello everybody!
> >
> > I've been trying to use hadoop distributed file system from my java
> > spring web application but without any good results :). We have one
> > server where hadoop namenode are datanode are succesfully running (so
> > they all are running on a single node). I managed to configure it with
> > the help of documentation page found from hadoop website.
> >
> > Now i need to make my web application to use this distributed file
> > system running on the other server. I've tried to place the same
> > hadoop-site.xml found from the hadoop master server in the programs
> > classpath and using DistributedFileSystem but without no luck. Always
> > when trying to use for example copyFromLocalFile it throws a
> > NullPointerException. Im getting really lost with this issue because i
> > havent found any examples or tutorials related to this :)
> >
> > I would really appreciate if somebody could help me so that i can
> > connect to this hadoop master server from my java spring web application
> > and to use distributed file system. 
> >
> >
> > best regards,
> > Jani
> >
> >
> >   
> 
> 
> -- 
> Always vizz it us @ visvo.com
> 
> 


Mime
View raw message