hadoop-common-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Sagar Naik <sa...@visvo.com>
Subject Re: newbie problem: Using DFS from Java program
Date Wed, 22 Aug 2007 15:44:19 GMT
Hey

I think its more of a classpath issue.
Can u pl check whether u r hadoop config (hadoop-site.xml , etc)  files 
are present in classpath when u start from webserver.
I dont think u wud have errors in hadoop log files, but its good to chk


Jani Arvonen wrote:
> Oh yes, of course. Our web application throws this exception when trying
> to use copyFromLocalFile() through DistributedFileSystem. The local file
> im trying to copy to the HDFS do exists...
>
> And we have also tried to use LocalFileSystem which works fine but that
> is not exactly what we are trying to achieve :)
>
> stacktrace:
>
> java.lang.NullPointerException
> 	at org.apache.hadoop.fs.FileSystem.get(FileSystem.java:155)
> 	at org.apache.hadoop.fs.FileSystem.getNamed(FileSystem.java:119)
> 	at
> org.apache.hadoop.fs.ChecksumFileSystem.copyFromLocalFile(ChecksumFileSystem.java:576)
> 	at
> com.floobs.service.media.DefaultMediaStorage.copyFromLocalFile(DefaultMediaStorage.java:23)
> 	at
> com.floobs.service.media.DefaultMediaStorage.copyFromLocalFile(DefaultMediaStorage.java:18)
> 	at
> com.floobs.web.media.MediaUploadFormController.storeFile(MediaUploadFormController.java:155)
> 	at com.floobs.web.media.MediaUploadFormController.access
> $500(MediaUploadFormController.java:36)
> 	at com.floobs.web.media.MediaUploadFormController
> $2.processFile(MediaUploadFormController.java:120)
> 	at com.floobs.web.media.MediaUploadFormController
> $UploadFileProcessor.execute(MediaUploadFormController.java:144)
> 	at
> com.floobs.web.media.MediaUploadFormController.processFileUploads(MediaUploadFormController.java:114)
> 	at
> com.floobs.web.media.MediaUploadFormController.onSubmit(MediaUploadFormController.java:65)
> 	at
> org.springframework.web.servlet.mvc.SimpleFormController.processFormSubmission(SimpleFormController.java:267)
> 	at
> org.springframework.web.servlet.mvc.AbstractFormController.handleRequestInternal(AbstractFormController.java:250)
> 	at
> org.springframework.web.servlet.mvc.AbstractController.handleRequest(AbstractController.java:153)
> 	at
> org.riotfamily.cachius.spring.CacheableControllerHandlerAdapter.handle(CacheableControllerHandlerAdapter.java:118)
> 	at
> org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:839)
> 	at
> org.riotfamily.common.web.servlet.ReloadableDispatcherServlet.doDispatch(ReloadableDispatcherServlet.java:107)
> 	at
> org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:774)
> 	at
> org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:460)
> 	at
> org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:425)
> 	at javax.servlet.http.HttpServlet.service(HttpServlet.java:710)
> 	at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
> 	at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:269)
> 	at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
> 	at
> org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:691)
> 	at
> org.apache.catalina.core.ApplicationDispatcher.doInclude(ApplicationDispatcher.java:594)
> 	at
> org.apache.catalina.core.ApplicationDispatcher.include(ApplicationDispatcher.java:505)
> 	at
> org.riotfamily.pages.component.impl.IncludeComponent.renderInternal(IncludeComponent.java:66)
> 	at
> org.riotfamily.pages.component.impl.AbstractComponent.render(AbstractComponent.java:98)
> 	at
> org.riotfamily.pages.component.render.AbstractRenderStrategy.renderComponent(AbstractRenderStrategy.java:243)
> 	at
> org.riotfamily.pages.component.render.AbstractRenderStrategy.renderComponentVersion(AbstractRenderStrategy.java:207)
> 	at
> org.riotfamily.pages.component.render.EditModeRenderStrategy.renderContainer(EditModeRenderStrategy.java:169)
> 	at
> org.riotfamily.pages.component.render.AbstractRenderStrategy.renderComponents(AbstractRenderStrategy.java:156)
> 	at
> org.riotfamily.pages.component.render.AbstractRenderStrategy.renderComponentList(AbstractRenderStrategy.java:137)
> 	at
> org.riotfamily.pages.component.render.EditModeRenderStrategy.renderComponentList(EditModeRenderStrategy.java:93)
> 	at
> org.riotfamily.pages.component.render.AbstractRenderStrategy.render(AbstractRenderStrategy.java:87)
> 	at
> org.riotfamily.pages.component.render.AbstractRenderStrategy.render(AbstractRenderStrategy.java:81)
> 	at org.riotfamily.pages.component.ComponentListController
> $1.doInTransactionWithoutResult(ComponentListController.java:202)
> 	at
> org.springframework.transaction.support.TransactionCallbackWithoutResult.doInTransaction(TransactionCallbackWithoutResult.java:33)
> 	at
> org.springframework.transaction.support.TransactionTemplate.execute(TransactionTemplate.java:127)
> 	at
> org.riotfamily.pages.component.ComponentListController.handleRequest(ComponentListController.java:199)
> 	at
> org.riotfamily.cachius.spring.CacheableControllerHandlerAdapter.handle(CacheableControllerHandlerAdapter.java:118)
> 	at
> org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:839)
> 	at
> org.riotfamily.common.web.servlet.ReloadableDispatcherServlet.doDispatch(ReloadableDispatcherServlet.java:107)
> 	at
> org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:774)
> 	at
> org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:460)
> 	at
> org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:425)
> 	at javax.servlet.http.HttpServlet.service(HttpServlet.java:710)
> 	at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
> 	at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:269)
> 	at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
> 	at
> org.apache.catalina.core.ApplicationDispatcher.invoke(ApplicationDispatcher.java:691)
> 	at
> org.apache.catalina.core.ApplicationDispatcher.doInclude(ApplicationDispatcher.java:594)
> 	at
> org.apache.catalina.core.ApplicationDispatcher.include(ApplicationDispatcher.java:505)
> 	at
> org.riotfamily.common.web.view.freemarker.IncludeMethod.exec(IncludeMethod.java:37)
> 	at
> org.riotfamily.common.web.view.freemarker.AbstractSimpleMethod.exec(AbstractSimpleMethod.java:38)
> 	at freemarker.core.MethodCall._getAsTemplateModel(MethodCall.java:93)
> 	at freemarker.core.Expression.getAsTemplateModel(Expression.java:89)
> 	at freemarker.core.Expression.getStringValue(Expression.java:93)
> 	at freemarker.core.DollarVariable.accept(DollarVariable.java:76)
> 	at freemarker.core.Environment.visit(Environment.java:196)
> 	at freemarker.core.MixedContent.accept(MixedContent.java:92)
> 	at freemarker.core.Environment.visit(Environment.java:196)
> 	at freemarker.core.Macro$Context.runMacro(Macro.java:164)
> 	at freemarker.core.Environment.visit(Environment.java:537)
> 	at freemarker.core.UnifiedCall.accept(UnifiedCall.java:128)
> 	at freemarker.core.Environment.visit(Environment.java:196)
> 	at freemarker.core.MixedContent.accept(MixedContent.java:92)
> 	at freemarker.core.Environment.visit(Environment.java:196)
> 	at freemarker.core.Environment.process(Environment.java:176)
> 	at freemarker.template.Template.process(Template.java:232)
> 	at
> org.springframework.web.servlet.view.freemarker.FreeMarkerView.processTemplate(FreeMarkerView.java:333)
> 	at
> org.riotfamily.common.web.view.freemarker.RiotFreeMarkerView.doRender(RiotFreeMarkerView.java:113)
> 	at org.riotfamily.common.web.view.freemarker.RiotFreeMarkerView
> $1.doInContext(RiotFreeMarkerView.java:93)
> 	at
> org.riotfamily.common.web.view.ViewContext.execute(ViewContext.java:45)
> 	at
> org.riotfamily.common.web.view.freemarker.RiotFreeMarkerView.renderMergedTemplateModel(RiotFreeMarkerView.java:91)
> 	at
> org.springframework.web.servlet.view.AbstractTemplateView.renderMergedOutputModel(AbstractTemplateView.java:179)
> 	at
> org.springframework.web.servlet.view.AbstractView.render(AbstractView.java:247)
> 	at
> org.riotfamily.common.web.view.freemarker.RiotFreeMarkerView.render(RiotFreeMarkerView.java:75)
> 	at
> org.springframework.web.servlet.DispatcherServlet.render(DispatcherServlet.java:1123)
> 	at
> org.springframework.web.servlet.DispatcherServlet.doDispatch(DispatcherServlet.java:860)
> 	at
> org.riotfamily.common.web.servlet.ReloadableDispatcherServlet.doDispatch(ReloadableDispatcherServlet.java:107)
> 	at
> org.springframework.web.servlet.DispatcherServlet.doService(DispatcherServlet.java:774)
> 	at
> org.springframework.web.servlet.FrameworkServlet.processRequest(FrameworkServlet.java:460)
> 	at
> org.springframework.web.servlet.FrameworkServlet.doPost(FrameworkServlet.java:425)
> 	at javax.servlet.http.HttpServlet.service(HttpServlet.java:710)
> 	at javax.servlet.http.HttpServlet.service(HttpServlet.java:803)
> 	at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:269)
> 	at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
> 	at
> org.riotfamily.common.web.filter.PluginChain.doFilter(PluginChain.java:60)
> 	at
> org.riotfamily.pages.page.support.FolderFilterPlugin.doFilter(FolderFilterPlugin.java:82)
> 	at
> org.riotfamily.common.web.filter.PluginChain.doFilter(PluginChain.java:57)
> 	at
> org.riotfamily.riot.security.AccessControlFilterPlugin.doFilter(AccessControlFilterPlugin.java:58)
> 	at
> org.riotfamily.common.web.filter.PluginChain.doFilter(PluginChain.java:57)
> 	at
> org.riotfamily.common.web.filter.PluginFilter.doFilterInternal(PluginFilter.java:123)
> 	at
> org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:75)
> 	at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215)
> 	at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
> 	at
> org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:70)
> 	at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215)
> 	at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
> 	at
> org.riotfamily.common.web.filter.CharacterEncodingFilter.doFilterInternal(CharacterEncodingFilter.java:95)
> 	at
> org.springframework.web.filter.OncePerRequestFilter.doFilter(OncePerRequestFilter.java:75)
> 	at
> org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:215)
> 	at
> org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:188)
> 	at
> org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:210)
> 	at
> org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:174)
> 	at
> org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:127)
> 	at
> org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:117)
> 	at
> org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:108)
> 	at
> org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:151)
> 	at
> org.apache.coyote.http11.Http11Processor.process(Http11Processor.java:870)
> 	at org.apache.coyote.http11.Http11BaseProtocol
> $Http11ConnectionHandler.processConnection(Http11BaseProtocol.java:665)
> 	at
> org.apache.tomcat.util.net.PoolTcpEndpoint.processSocket(PoolTcpEndpoint.java:528)
> 	at
> org.apache.tomcat.util.net.LeaderFollowerWorkerThread.runIt(LeaderFollowerWorkerThread.java:81)
> 	at org.apache.tomcat.util.threads.ThreadPool
> $ControlRunnable.run(ThreadPool.java:685)
> 	at java.lang.Thread.run(Thread.java:619)
>
>
>
> And our hadoop server logfile (datanode log) contains this several times
> with different timestamp:
>
> 2007-08-21 00:00:23,393 INFO org.apache.hadoop.dfs.DataNode: using
> BLOCKREPORT_INTERVAL of 3356962msec
> 2007-08-21 00:00:23,407 ERROR org.apache.hadoop.dfs.DataNode: Exception:
> java.lang.NullPointerException
>         at org.apache.hadoop.dfs.FSDataset
> $FSDir.getBlockInfo(FSDataset.java:148)
>         at org.apache.hadoop.dfs.FSDataset
> $FSVolume.getBlockInfo(FSDataset.java:353)
>         at org.apache.hadoop.dfs.FSDataset
> $FSVolumeSet.getBlockInfo(FSDataset.java:419)
>         at
> org.apache.hadoop.dfs.FSDataset.getBlockReport(FSDataset.java:636)
>         at
> org.apache.hadoop.dfs.DataNode.offerService(DataNode.java:523)
>         at org.apache.hadoop.dfs.DataNode.run(DataNode.java:1310)
>         at java.lang.Thread.run(Thread.java:619)
>
>
> but im not sure if this has anything to do with this particular problem.
> Any help would be really appreciated!
>
> Best regards,
> Jani
>
> On Tue, 2007-08-21 at 11:53 -0500, Sagar Naik wrote:
>   
>> Hello
>> Can u provide some more information.
>> Like a stackstrace from log files
>>
>> - Sagar
>> Jani Arvonen wrote:
>>     
>>> Hello everybody!
>>>
>>> I've been trying to use hadoop distributed file system from my java
>>> spring web application but without any good results :). We have one
>>> server where hadoop namenode are datanode are succesfully running (so
>>> they all are running on a single node). I managed to configure it with
>>> the help of documentation page found from hadoop website.
>>>
>>> Now i need to make my web application to use this distributed file
>>> system running on the other server. I've tried to place the same
>>> hadoop-site.xml found from the hadoop master server in the programs
>>> classpath and using DistributedFileSystem but without no luck. Always
>>> when trying to use for example copyFromLocalFile it throws a
>>> NullPointerException. Im getting really lost with this issue because i
>>> havent found any examples or tutorials related to this :)
>>>
>>> I would really appreciate if somebody could help me so that i can
>>> connect to this hadoop master server from my java spring web application
>>> and to use distributed file system. 
>>>
>>>
>>> best regards,
>>> Jani
>>>
>>>
>>>   
>>>       
>>
>>
>>
>>     
>
>
>   



-- 
This message has been scanned for viruses and
dangerous content and is believed to be clean.


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message