apex-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From David Yan <da...@datatorrent.com>
Subject Re: datatorrent gateway application package problem
Date Thu, 29 Sep 2016 00:25:26 GMT
FYI, I'm getting the exact same error after I set fs.defaultFS to file:///
in core-site.xml in the hadoop installation and restarted yarn.

Wrong FS: hdfs://localhost:9000/user/dtadmin/datatorrent, expected: file:///

For now, dtgateway only works if fs.defaultFS is the same FS as the "DFS
location" in the dtgateway configuration. I'll log this as a bug.

Please double check the correct value is picked up for fs.defaultFS in your
installation. I think that value should be in core-site.xml, not
hdfs-site.xml, and if not specified in core-site.xml, it would default to
file:///

David


On Wed, Sep 28, 2016 at 4:02 PM, <dbs@softiron.co.uk> wrote:

> On 2016-09-28 14:56, Vlad Rozov wrote:
>
>> Check namenode logs. Are there any exceptions/errors raised at the
>> same time as the gateway exception?
>>
>
> Nope.  Nothing interesting there.
>
>
>
>
>> On 9/28/16 14:43, dbs@softiron.co.uk wrote:
>>
>>> On 2016-09-28 14:09, David Yan wrote:
>>>
>>>> dtgateway does not use the native libraries.
>>>>
>>>> I just tried a clean install of RTS 3.5.0 community edition on plain
>>>> vanilla hadoop 2.7.3 on ubuntu 14.04 and was able to launch the pi
>>>> demo from the web interface. I was able to do the same even with the
>>>> native libraries removed.
>>>>
>>>
>>> Thanks.  I thought it was a long shot.
>>>
>>> What is your fs.defaultFS property in hadoop? Can you verify that it
>>>> points to your hdfs (hdfs://namenode:9000/) ?
>>>>
>>>
>>> In $HADOOP_CONF_DIR/hdfs-site.xml, fs.defaultFS is hdfs://namenode:9000
>>>
>>> Note also that if I try to give a URL to the dtgateway Installation
>>> Wizard
>>> that isn't the namenode host/port, it complains immediately, instead of
>>> proceeding to the next page.
>>>
>>> Unfortunately I don't have access to an ARM64 so any finding you post
>>>> here would be helpful.
>>>>
>>>
>>> I'm not surprised.  A classic chicken-and-egg problem.  There aren't a
>>> lot of machines around, so the software doesn't get a lot of attention, so
>>> people don't buy machines.
>>>
>>>
>>>
>>>> David
>>>>
>>>> On Wed, Sep 28, 2016 at 11:04 AM, <dbs@softiron.co.uk> wrote:
>>>>
>>>> I had a thought of what might be the problem:
>>>>> the hadoop native library doesn't build on SUSE/SLES ARM64,
>>>>> so I'm running with the pure Java library.
>>>>>
>>>>> Is it possible that the dtgateway is trying to use the native
>>>>> library
>>>>> directly?
>>>>>
>>>>> Thanks,
>>>>> -david
>>>>>
>>>>> On 2016-09-26 14:45, David Yan wrote:
>>>>>
>>>>> Do you see any exception stacktrace in the log when this error
>>>>> occurred:
>>>>>
>>>>> Wrong FS: hdfs://namenode:9000/user/dtadmin/datatorrent, expected:
>>>>> file:///
>>>>>
>>>>> David
>>>>>
>>>>> On Mon, Sep 26, 2016 at 2:39 PM, <dbs@softiron.co.uk> wrote:
>>>>>
>>>>> From    David Yan <david@datatorrent.com>
>>>>> Date    Sat 00:12
>>>>>
>>>>> Can you please provide any exception stacktrace in the
>>>>> dtgateway.log file when that happens?
>>>>>
>>>>> I reran the dtgateway installation wizard, which failed (same as
>>>>> before) with error msg:
>>>>>
>>>>> | DFS directory cannot be written to with error message "Mkdirs
>>>>> failed to create /user/dtadmin/datatorrent (exists=false,
>>>>> cwd=file:/opt/datatorrent/releases/3.4.0)"
>>>>>
>>>>> The corresponding exception in dtgateway.log is:
>>>>>
>>>>> | 2016-09-26 13:43:54,767 ERROR com.datatorrent.gateway.I: DFS
>>>>> Directory cannot be written to with exception:
>>>>> | java.io.IOException: Mkdirs failed to create
>>>>> /user/dtadmin/datatorrent (exists=false,
>>>>> cwd=file:/opt/datatorrent/releases/3.4.0)
>>>>> |     at
>>>>>
>>>>
>>>> org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileS
>>>> ystem.java:455)
>>>>
>>>>> |     at
>>>>>
>>>>
>>>> org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileS
>>>> ystem.java:440)
>>>>
>>>>> |     at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:911)
>>>>> |     at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:892)
>>>>> |     at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:789)
>>>>> |     at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:778)
>>>>> |     at
>>>>> com.datatorrent.stram.client.FSAgent.createFile(FSAgent.java:77)
>>>>> |     at com.datatorrent.gateway.I.h(gc:324)
>>>>> |     at com.datatorrent.gateway.I.h(gc:284)
>>>>> |     at
>>>>> com.datatorrent.gateway.resources.ws.v2.ConfigResource.h(fc:136)
>>>>> |     at
>>>>> com.datatorrent.gateway.resources.ws.v2.ConfigResource.h(fc:171)
>>>>> |     at
>>>>>
>>>>
>>>> com.datatorrent.gateway.resources.ws.v2.ConfigResource.setCo
>>>> nfigProperty(fc:34)
>>>>
>>>>> |     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
>>>>> |     at
>>>>>
>>>>
>>>> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAcce
>>>> ssorImpl.java:62)
>>>>
>>>>> |     at
>>>>>
>>>>
>>>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe
>>>> thodAccessorImpl.java:43)
>>>>
>>>>> |     at java.lang.reflect.Method.invoke(Method.java:498)
>>>>> |     at
>>>>>
>>>>
>>>> com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invo
>>>> ke(JavaMethodInvokerFactory.java:60)
>>>>
>>>>> |     at
>>>>>
>>>>
>>>> com.sun.jersey.server.impl.model.method.dispatch.AbstractRes
>>>> ourceMethodDispatchProvider$ResponseOutInvoker._dispatch(A
>>>> bstractResourceMethodDispatchProvider.java:205)
>>>>
>>>>> |     at
>>>>>
>>>>
>>>> com.sun.jersey.server.impl.model.method.dispatch.ResourceJav
>>>> aMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
>>>>
>>>>> |     at
>>>>>
>>>>
>>>> com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(
>>>> HttpMethodRule.java:288)
>>>>
>>>>> |     at
>>>>>
>>>>
>>>> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accep
>>>> t(RightHandPathRule.java:147)
>>>>
>>>>> |     at
>>>>>
>>>>
>>>> com.sun.jersey.server.impl.uri.rules.SubLocatorRule.accept(
>>>> SubLocatorRule.java:134)
>>>>
>>>>> |     at
>>>>>
>>>>
>>>> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accep
>>>> t(RightHandPathRule.java:147)
>>>>
>>>>> |     at
>>>>>
>>>>
>>>> com.sun.jersey.server.impl.uri.rules.SubLocatorRule.accept(
>>>> SubLocatorRule.java:134)
>>>>
>>>>> |     at
>>>>>
>>>>
>>>> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accep
>>>> t(RightHandPathRule.java:147)
>>>>
>>>>> |     at
>>>>>
>>>>
>>>> com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accep
>>>> t(ResourceClassRule.java:108)
>>>>
>>>>> |     at
>>>>>
>>>>
>>>> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accep
>>>> t(RightHandPathRule.java:147)
>>>>
>>>>> |     at
>>>>>
>>>>
>>>> com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule
>>>> .accept(RootResourceClassesRule.java:84)
>>>>
>>>>> |     at
>>>>>
>>>>
>>>> com.sun.jersey.server.impl.application.WebApplicationImpl._h
>>>> andleRequest(WebApplicationImpl.java:1469)
>>>>
>>>>> |     at
>>>>>
>>>>
>>>> com.sun.jersey.server.impl.application.WebApplicationImpl._h
>>>> andleRequest(WebApplicationImpl.java:1400)
>>>>
>>>>> |     at
>>>>>
>>>>
>>>> com.sun.jersey.server.impl.application.WebApplicationImpl.ha
>>>> ndleRequest(WebApplicationImpl.java:1349)
>>>>
>>>>> |     at
>>>>>
>>>>
>>>> com.sun.jersey.server.impl.application.WebApplicationImpl.ha
>>>> ndleRequest(WebApplicationImpl.java:1339)
>>>>
>>>>> |     at
>>>>>
>>>>
>>>> com.sun.jersey.spi.container.servlet.WebComponent.service(We
>>>> bComponent.java:416)
>>>>
>>>>> |     at
>>>>>
>>>>
>>>> com.sun.jersey.spi.container.servlet.ServletContainer.servic
>>>> e(ServletContainer.java:537)
>>>>
>>>>> |     at
>>>>>
>>>>
>>>> com.sun.jersey.spi.container.servlet.ServletContainer.servic
>>>> e(ServletContainer.java:699)
>>>>
>>>>> |     at
>>>>> javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
>>>>> |     at
>>>>>
>>>>  org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:669)
>>>>
>>>> |     at
>>>>>
>>>>
>>>> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHan
>>>> dler.java:457)
>>>>
>>>>> |     at
>>>>>
>>>>
>>>> org.eclipse.jetty.server.session.SessionHandler.doHandle(
>>>> SessionHandler.java:229)
>>>>
>>>>> |     at
>>>>>
>>>>
>>>> org.eclipse.jetty.server.handler.ContextHandler.doHandle(
>>>> ContextHandler.java:1075)
>>>>
>>>>> |     at
>>>>>
>>>>
>>>> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHand
>>>> ler.java:384)
>>>>
>>>>> |     at
>>>>>
>>>>
>>>> org.eclipse.jetty.server.session.SessionHandler.doScope(
>>>> SessionHandler.java:193)
>>>>
>>>>> |     at
>>>>>
>>>>
>>>> org.eclipse.jetty.server.handler.ContextHandler.doScope(
>>>> ContextHandler.java:1009)
>>>>
>>>>> |     at
>>>>>
>>>>
>>>> org.eclipse.jetty.server.handler.ScopedHandler.handle(Scoped
>>>> Handler.java:135)
>>>>
>>>>> |     at
>>>>>
>>>>
>>>> org.eclipse.jetty.server.handler.HandlerCollection.handle(
>>>> HandlerCollection.java:154)
>>>>
>>>>> |     at
>>>>>
>>>>
>>>> org.eclipse.jetty.server.handler.HandlerWrapper.handle(Handl
>>>> erWrapper.java:116)
>>>>
>>>>> |     at org.eclipse.jetty.server.Server.handle(Server.java:368)
>>>>> |     at
>>>>>
>>>>
>>>> org.eclipse.jetty.server.AbstractHttpConnection.handleReques
>>>> t(AbstractHttpConnection.java:489)
>>>>
>>>>> |     at
>>>>>
>>>>
>>>> org.eclipse.jetty.server.AbstractHttpConnection.content(Abst
>>>> ractHttpConnection.java:953)
>>>>
>>>>> |     at
>>>>>
>>>>
>>>> org.eclipse.jetty.server.AbstractHttpConnection$RequestHandl
>>>> er.content(AbstractHttpConnection.java:1014)
>>>>
>>>>> |     at
>>>>> org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:861)
>>>>> |     at
>>>>>
>>>>  org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:240)
>>>>
>>>> |     at
>>>>>
>>>>
>>>> org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHtt
>>>> pConnection.java:82)
>>>>
>>>>> |     at
>>>>>
>>>>
>>>> org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(Select
>>>> ChannelEndPoint.java:628)
>>>>
>>>>> |     at
>>>>>
>>>>
>>>> org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectC
>>>> hannelEndPoint.java:52)
>>>>
>>>>> |     at
>>>>>
>>>>
>>>> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(Queued
>>>> ThreadPool.java:608)
>>>>
>>>>> |     at
>>>>>
>>>>
>>>> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedT
>>>> hreadPool.java:543)
>>>>
>>>>> |     at java.lang.Thread.run(Thread.java:745)
>>>>> | 2016-09-26 13:43:54,772 INFO
>>>>> com.datatorrent.gateway.resources.ws.v2.WSResource: Caught exception
>>>>> in processing web service:
>>>>> com.datatorrent.stram.client.DTConfiguration$ConfigException: DFS
>>>>> directory cannot be written to with error message "Mkdirs failed to
>>>>> create /user/dtadmin/datatorrent (exists=false,
>>>>> cwd=file:/opt/datatorrent/releases/3.4.0)"
>>>>>
>>>>> Also, when I shutdown the gateway, I get a different exception in
>>>>> dtgateway.log:
>>>>>
>>>>> | 2016-09-26 13:40:44,366 INFO com.datatorrent.gateway.DTGateway:
>>>>> Shutting down
>>>>> | 2016-09-26 13:40:44,427 ERROR com.datatorrent.gateway.I: DFS
>>>>> Directory cannot be written to with exception:
>>>>> | java.io.IOException: Filesystem closed
>>>>> |     at
>>>>> org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:808)
>>>>> |     at
>>>>> org.apache.hadoop.hdfs.DFSClient.delete(DFSClient.java:2041)
>>>>> |     at
>>>>>
>>>>
>>>> org.apache.hadoop.hdfs.DistributedFileSystem$14.doCall(
>>>> DistributedFileSystem.java:707)
>>>>
>>>>> |     at
>>>>>
>>>>
>>>> org.apache.hadoop.hdfs.DistributedFileSystem$14.doCall(
>>>> DistributedFileSystem.java:703)
>>>>
>>>>> |     at
>>>>>
>>>>
>>>> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSyst
>>>> emLinkResolver.java:81)
>>>>
>>>>> |     at
>>>>>
>>>>
>>>> org.apache.hadoop.hdfs.DistributedFileSystem.delete(Distribu
>>>> tedFileSystem.java:714)
>>>>
>>>>> |     at
>>>>> com.datatorrent.stram.client.FSAgent.deleteFile(FSAgent.java:94)
>>>>> |     at com.datatorrent.gateway.I.h(gc:30)
>>>>> |     at com.datatorrent.gateway.l.K(pc:158)
>>>>> |     at com.datatorrent.gateway.l.h(pc:65)
>>>>> |     at com.datatorrent.gateway.K.run(yc:156)
>>>>> |     at java.lang.Thread.run(Thread.java:745)
>>>>> | 2016-09-26 13:40:44,444 INFO com.datatorrent.gateway.DTGateway:
>>>>> Shutdown complete
>>>>>
>>>>> Thanks much.
>>>>> -david
>>>>>
>>>>> On Tue, Sep 13, 2016 at 4:06 PM, <dbs@softiron.co.uk> wrote:
>>>>>
>>>>> I have the dtgateway running (community edition) with
>>>>> apex/malhar 3.4 and hadoop 2.7.4 on SuSE SLES 12.1 on an ARM64
>>>>> server (aarch64).
>>>>>
>>>>> In the Installation Wizard, I set the DFS location to:
>>>>>
>>>>> hdfs://namenode:9000/user/dtadmin/datatorrent
>>>>>
>>>>> and the gateway saves the hadoop configuration and restarts
>>>>> successfully, but  I get an error:
>>>>>
>>>>> Wrong FS: hdfs://namenode:9000/user/dtadmin/datatorrent,
>>>>> expected: file:///
>>>>>
>>>>> the 'service dtgateway status' command says it's running.
>>>>> The log file has the same error as above, but nothing else
>>>>> useful.
>>>>>
>>>>> Is there a manual way to move the demo apps into the
>>>>> gateway?
>>>>>
>>>>> If I ignore this error and try to upload or import a demo
>>>>> app, I get the same error.
>>>>>
>>>>> What's odd is that the gateway does write into the HDFS
>>>>> directory:
>>>>>
>>>>> Found 6 items
>>>>> drwxrwxrwt   - dtadmin hadoop          0 2016-09-13 15:35
>>>>> /user/dtadmin/datatorrent/appPackages
>>>>> drwxrwxrwt   - dtadmin hadoop          0 2016-09-13 15:35
>>>>> /user/dtadmin/datatorrent/apps
>>>>> drwxrwxrwt   - dtadmin hadoop          0 2016-09-13 15:35
>>>>> /user/dtadmin/datatorrent/conf
>>>>> drwxrwxrwt   - dtadmin hadoop          0 2016-09-13 15:35
>>>>> /user/dtadmin/datatorrent/dashboards
>>>>> drwxrwxrwt   - dtadmin hadoop          0 2016-09-13 15:35
>>>>> /user/dtadmin/datatorrent/licenses
>>>>> drwxrwxrwt   - dtadmin hadoop          0 2016-09-13 15:35
>>>>> /user/dtadmin/datatorrent/systemAlerts
>>>>>
>>>>> I've tried playing with different ways to configure the DFS
>>>>> location, but nothing works.
>>>>> If I use an actual local filesystem directory, I get a
>>>>> different errro from the installation wizard (Non-DFS file system
>>>>> is used: org.apache.hadoop.fs.LocalFileSystem), which makes sense.
>>>>> But if I ignore this error, the app wizard successsfully writes
>>>>> packages into the local directory, but I can't launch them (no
>>>>> surprise there).
>>>>>
>>>>> Anybody have any ideas what I can do?
>>>>>
>>>>> thanks much.
>>>>> -david
>>>>>
>>>>
>>>
>

Mime
View raw message