apex-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From d..@softiron.co.uk
Subject Re: datatorrent gateway application package problem
Date Thu, 29 Sep 2016 21:48:10 GMT
David,

I double-checked and I was wrong in my previous email.
The fs.defaultFS setting is in core-site.xml, not hdfs-site, so that 
wasn't the problem.

But you were essentially right about the cause.

I have my hadoop xml files in a directory pointed to by the 
$HADOOP_CONF_DIR environment variable, but this directory is separate 
from the hadoop installation tree (ie, it's not
a subdir of HADOOP_HOME or HADOOP_PREFIX).

I copied all my xml files into $HADOOP_HOME/etc/hadoop/, restarted 
dtgateway, and ran the
instalation wizard again, and this time it didn't get the error.  Also, 
I can import
packages from dtHub successfully, which didn't work before either.

So that fixes the problem.  My guess is that dtgateway doesn't use 
HADOOP_CONF_DIR to find the hadoop xml files.

Thanks very much for your help!

-david


> On Wed, Sep 28, 2016 at 9:15 PM, <dbs@softiron.co.uk> wrote:
> 
>> On 2016-09-28 17:25, David Yan wrote:
>> 
>>> FYI, I'm getting the exact same error after I set fs.defaultFS to
>>> file:/// in core-site.xml in the hadoop installation and restarted
>>> yarn.
>>> 
>>> Wrong FS: hdfs://localhost:9000/user/dtadmin/datatorrent,
>>> expected:
>>> file:///
>>> 
>>> For now, dtgateway only works if fs.defaultFS is the same FS as
>>> the
>>> "DFS location" in the dtgateway configuration. I'll log this as a
>>> bug.
>>> 
>>> Please double check the correct value is picked up for
>>> fs.defaultFS in
>>> your installation. I think that value should be in core-site.xml,
>>> not
>>> hdfs-site.xml, and if not specified in core-site.xml, it would
>>> default
>>> to file:///
>>> 
>>> David
>> 
>> I'll check the next time I have access to the hardware.  Thanks
>> very much.
>> There's no way I would have found this by myself.  I really
>> appreciate
>> you spending the time on this.
>> 
>> -david
>> 
>> On Wed, Sep 28, 2016 at 4:02 PM, <dbs@softiron.co.uk> wrote:
>> 
>> On 2016-09-28 14:56, Vlad Rozov wrote:
>> 
>> Check namenode logs. Are there any exceptions/errors raised at the
>> same time as the gateway exception?
>> 
>> Nope.  Nothing interesting there.
>> 
>> On 9/28/16 14:43, dbs@softiron.co.uk wrote:
>> On 2016-09-28 14:09, David Yan wrote:
>> dtgateway does not use the native libraries.
>> 
>> I just tried a clean install of RTS 3.5.0 community edition on plain
>> vanilla hadoop 2.7.3 on ubuntu 14.04 and was able to launch the pi
>> demo from the web interface. I was able to do the same even with the
>> native libraries removed.
>> 
>> Thanks.  I thought it was a long shot.
>> 
>> What is your fs.defaultFS property in hadoop? Can you verify that
>> it
>> points to your hdfs (hdfs://namenode:9000/) ?
>> 
>> In $HADOOP_CONF_DIR/hdfs-site.xml, fs.defaultFS is
>> hdfs://namenode:9000
>> 
>> Note also that if I try to give a URL to the dtgateway Installation
>> Wizard
>> that isn't the namenode host/port, it complains immediately, instead
>> of
>> proceeding to the next page.
>> 
>> Unfortunately I don't have access to an ARM64 so any finding you
>> post
>> here would be helpful.
>> 
>> I'm not surprised.  A classic chicken-and-egg problem.  There aren't
>> a lot of machines around, so the software doesn't get a lot of
>> attention, so people don't buy machines.
>> 
>> David
>> 
>> On Wed, Sep 28, 2016 at 11:04 AM, <dbs@softiron.co.uk> wrote:
>> 
>> I had a thought of what might be the problem:
>> the hadoop native library doesn't build on SUSE/SLES ARM64,
>> so I'm running with the pure Java library.
>> 
>> Is it possible that the dtgateway is trying to use the native
>> library
>> directly?
>> 
>> Thanks,
>> -david
>> 
>> On 2016-09-26 14:45, David Yan wrote:
>> 
>> Do you see any exception stacktrace in the log when this error
>> occurred:
>> 
>> Wrong FS: hdfs://namenode:9000/user/dtadmin/datatorrent, expected:
>> file:///
>> 
>> David
>> 
>> On Mon, Sep 26, 2016 at 2:39 PM, <dbs@softiron.co.uk> wrote:
>> 
>> From    David Yan <david@datatorrent.com>
>> Date    Sat 00:12
>> 
>> Can you please provide any exception stacktrace in the
>> dtgateway.log file when that happens?
>> 
>> I reran the dtgateway installation wizard, which failed (same as
>> before) with error msg:
>> 
>> | DFS directory cannot be written to with error message "Mkdirs
>> failed to create /user/dtadmin/datatorrent (exists=false,
>> cwd=file:/opt/datatorrent/releases/3.4.0)"
>> 
>> The corresponding exception in dtgateway.log is:
>> 
>> | 2016-09-26 13:43:54,767 ERROR com.datatorrent.gateway.I: DFS
>> Directory cannot be written to with exception:
>> | java.io.IOException: Mkdirs failed to create
>> /user/dtadmin/datatorrent (exists=false,
>> cwd=file:/opt/datatorrent/releases/3.4.0)
>> |     at
>  
> org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:455)
> 
>> |     at
>  
> org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFileSystem.java:440)
> 
>> |     at
>> org.apache.hadoop.fs.FileSystem.create(FileSystem.java:911)
>> |     at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:892)
>> |     at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:789)
>> |     at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:778)
>> |     at
>> com.datatorrent.stram.client.FSAgent.createFile(FSAgent.java:77)
>> |     at com.datatorrent.gateway.I.h(gc:324)
>> |     at com.datatorrent.gateway.I.h(gc:284)
>> |     at
>> com.datatorrent.gateway.resources.ws.v2.ConfigResource.h(fc:136)
>> |     at
>> com.datatorrent.gateway.resources.ws.v2.ConfigResource.h(fc:171)
>> |     at
>  
> com.datatorrent.gateway.resources.ws.v2.ConfigResource.setConfigProperty(fc:34)
> 
>> |     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
>> Method)
>> |     at
>  
> sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
> 
>> |     at
> 
> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
> 
>> |     at java.lang.reflect.Method.invoke(Method.java:498)
>> |     at
> 
> com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
> 
>> |     at
> 
> com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$ResponseOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:205)
> 
>> |     at
> 
> com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
> 
>> |     at
> 
> com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288)
> 
>> |     at
> 
> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
> 
>> |     at
> 
> com.sun.jersey.server.impl.uri.rules.SubLocatorRule.accept(SubLocatorRule.java:134)
> 
>> |     at
> 
> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
> 
>> |     at
> 
> com.sun.jersey.server.impl.uri.rules.SubLocatorRule.accept(SubLocatorRule.java:134)
> 
>> |     at
> 
> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
> 
>> |     at
> 
> com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
> 
>> |     at
> 
> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
> 
>> |     at
> 
> com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
> 
>> |     at
> 
> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469)
> 
>> |     at
> 
> com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400)
> 
>> |     at
> 
> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349)
> 
>> |     at
> 
> com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339)
> 
>> |     at
> 
> com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416)
> 
>> |     at
> 
> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537)
> 
>> |     at
> 
> com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:699)
> 
>> |     at
>> javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
>> |     at
>  org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:669)
> 
>> |     at
>  
> org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:457)
> 
>> |     at
> 
> org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:229)
> 
>> |     at
> 
> org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1075)
> 
>> |     at
>  
> org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:384)
> 
>> |     at
> 
> org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:193)
> 
>> |     at
> 
> org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1009)
> 
>> |     at
>  
> org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:135)
> 
>> |     at
> 
> org.eclipse.jetty.server.handler.HandlerCollection.handle(HandlerCollection.java:154)
> 
>> |     at
>  
> org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:116)
> 
>> |     at org.eclipse.jetty.server.Server.handle(Server.java:368)
>> |     at
> 
> org.eclipse.jetty.server.AbstractHttpConnection.handleRequest(AbstractHttpConnection.java:489)
> 
>> |     at
> 
> org.eclipse.jetty.server.AbstractHttpConnection.content(AbstractHttpConnection.java:953)
> 
>> |     at
> 
> org.eclipse.jetty.server.AbstractHttpConnection$RequestHandler.content(AbstractHttpConnection.java:1014)
> 
>> |     at
>> org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:861)
>> |     at
>  org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:240)
> 
>> |     at
> 
> org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHttpConnection.java:82)
> 
>> |     at
> 
> org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(SelectChannelEndPoint.java:628)
> 
>> |     at
> 
> org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectChannelEndPoint.java:52)
> 
>> |     at
> 
> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:608)
> 
>> |     at
>  
> org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(QueuedThreadPool.java:543)
> 
>> |     at java.lang.Thread.run(Thread.java:745)
>> | 2016-09-26 13:43:54,772 INFO
>> com.datatorrent.gateway.resources.ws.v2.WSResource: Caught exception
>> in processing web service:
>> com.datatorrent.stram.client.DTConfiguration$ConfigException: DFS
>> directory cannot be written to with error message "Mkdirs failed to
>> create /user/dtadmin/datatorrent (exists=false,
>> cwd=file:/opt/datatorrent/releases/3.4.0)"
>> 
>> Also, when I shutdown the gateway, I get a different exception in
>> dtgateway.log:
>> 
>> | 2016-09-26 13:40:44,366 INFO com.datatorrent.gateway.DTGateway:
>> Shutting down
>> | 2016-09-26 13:40:44,427 ERROR com.datatorrent.gateway.I: DFS
>> Directory cannot be written to with exception:
>> | java.io.IOException: Filesystem closed
>> |     at
>> org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:808)
>> |     at
>> org.apache.hadoop.hdfs.DFSClient.delete(DFSClient.java:2041)
>> |     at
> 
> org.apache.hadoop.hdfs.DistributedFileSystem$14.doCall(DistributedFileSystem.java:707)
> 
>> |     at
> 
> org.apache.hadoop.hdfs.DistributedFileSystem$14.doCall(DistributedFileSystem.java:703)
> 
>> |     at
> 
> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSystemLinkResolver.java:81)
> 
>> |     at
> 
> org.apache.hadoop.hdfs.DistributedFileSystem.delete(DistributedFileSystem.java:714)
> 
>> |     at
>> com.datatorrent.stram.client.FSAgent.deleteFile(FSAgent.java:94)
>> |     at com.datatorrent.gateway.I.h(gc:30)
>> |     at com.datatorrent.gateway.l.K(pc:158)
>> |     at com.datatorrent.gateway.l.h(pc:65)
>> |     at com.datatorrent.gateway.K.run(yc:156)
>> |     at java.lang.Thread.run(Thread.java:745)
>> | 2016-09-26 13:40:44,444 INFO com.datatorrent.gateway.DTGateway:
>> Shutdown complete
>> 
>> Thanks much.
>> -david
>> 
>> On Tue, Sep 13, 2016 at 4:06 PM, <dbs@softiron.co.uk> wrote:
>> 
>> I have the dtgateway running (community edition) with
>> apex/malhar 3.4 and hadoop 2.7.4 on SuSE SLES 12.1 on an ARM64
>> server (aarch64).
>> 
>> In the Installation Wizard, I set the DFS location to:
>> 
>> hdfs://namenode:9000/user/dtadmin/datatorrent
>> 
>> and the gateway saves the hadoop configuration and restarts
>> successfully, but  I get an error:
>> 
>> Wrong FS: hdfs://namenode:9000/user/dtadmin/datatorrent,
>> expected: file:///
>> 
>> the 'service dtgateway status' command says it's running.
>> The log file has the same error as above, but nothing else
>> useful.
>> 
>> Is there a manual way to move the demo apps into the
>> gateway?
>> 
>> If I ignore this error and try to upload or import a demo
>> app, I get the same error.
>> 
>> What's odd is that the gateway does write into the HDFS
>> directory:
>> 
>> Found 6 items
>> drwxrwxrwt   - dtadmin hadoop          0 2016-09-13 15:35
>> /user/dtadmin/datatorrent/appPackages
>> drwxrwxrwt   - dtadmin hadoop          0 2016-09-13 15:35
>> /user/dtadmin/datatorrent/apps
>> drwxrwxrwt   - dtadmin hadoop          0 2016-09-13 15:35
>> /user/dtadmin/datatorrent/conf
>> drwxrwxrwt   - dtadmin hadoop          0 2016-09-13 15:35
>> /user/dtadmin/datatorrent/dashboards
>> drwxrwxrwt   - dtadmin hadoop          0 2016-09-13 15:35
>> /user/dtadmin/datatorrent/licenses
>> drwxrwxrwt   - dtadmin hadoop          0 2016-09-13 15:35
>> /user/dtadmin/datatorrent/systemAlerts
>> 
>> I've tried playing with different ways to configure the DFS
>> location, but nothing works.
>> If I use an actual local filesystem directory, I get a
>> different errro from the installation wizard (Non-DFS file system
>> is used: org.apache.hadoop.fs.LocalFileSystem), which makes sense.
>> But if I ignore this error, the app wizard successsfully writes
>> packages into the local directory, but I can't launch them (no
>> surprise there).
>> 
>> Anybody have any ideas what I can do?
>> 
>> thanks much.
>> -david


Mime
View raw message