apex-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Thomas Weise <...@apache.org>
Subject Re: datatorrent gateway application package problem
Date Thu, 29 Sep 2016 22:09:00 GMT
Glad you have it working now. And yes, the gateway (like the apex CLI) only
depends on the hadoop script. Different distributions of Hadoop have
different ways of managing their bits and we want to be agnostic. So if you
would like to use a different conf dir, then you would make that change
inside your Hadoop setup and verify it works with just the hadoop script.

Thomas


On Thu, Sep 29, 2016 at 2:48 PM, <dbs@softiron.co.uk> wrote:

> David,
>
> I double-checked and I was wrong in my previous email.
> The fs.defaultFS setting is in core-site.xml, not hdfs-site, so that
> wasn't the problem.
>
> But you were essentially right about the cause.
>
> I have my hadoop xml files in a directory pointed to by the
> $HADOOP_CONF_DIR environment variable, but this directory is separate from
> the hadoop installation tree (ie, it's not
> a subdir of HADOOP_HOME or HADOOP_PREFIX).
>
> I copied all my xml files into $HADOOP_HOME/etc/hadoop/, restarted
> dtgateway, and ran the
> instalation wizard again, and this time it didn't get the error.  Also, I
> can import
> packages from dtHub successfully, which didn't work before either.
>
> So that fixes the problem.  My guess is that dtgateway doesn't use
> HADOOP_CONF_DIR to find the hadoop xml files.
>
> Thanks very much for your help!
>
> -david
>
>
> On Wed, Sep 28, 2016 at 9:15 PM, <dbs@softiron.co.uk> wrote:
>>
>> On 2016-09-28 17:25, David Yan wrote:
>>>
>>> FYI, I'm getting the exact same error after I set fs.defaultFS to
>>>> file:/// in core-site.xml in the hadoop installation and restarted
>>>> yarn.
>>>>
>>>> Wrong FS: hdfs://localhost:9000/user/dtadmin/datatorrent,
>>>> expected:
>>>> file:///
>>>>
>>>> For now, dtgateway only works if fs.defaultFS is the same FS as
>>>> the
>>>> "DFS location" in the dtgateway configuration. I'll log this as a
>>>> bug.
>>>>
>>>> Please double check the correct value is picked up for
>>>> fs.defaultFS in
>>>> your installation. I think that value should be in core-site.xml,
>>>> not
>>>> hdfs-site.xml, and if not specified in core-site.xml, it would
>>>> default
>>>> to file:///
>>>>
>>>> David
>>>>
>>>
>>> I'll check the next time I have access to the hardware.  Thanks
>>> very much.
>>> There's no way I would have found this by myself.  I really
>>> appreciate
>>> you spending the time on this.
>>>
>>> -david
>>>
>>>
>>> On Wed, Sep 28, 2016 at 4:02 PM, <dbs@softiron.co.uk> wrote:
>>>
>>> On 2016-09-28 14:56, Vlad Rozov wrote:
>>>
>>> Check namenode logs. Are there any exceptions/errors raised at the
>>> same time as the gateway exception?
>>>
>>> Nope.  Nothing interesting there.
>>>
>>> On 9/28/16 14:43, dbs@softiron.co.uk wrote:
>>> On 2016-09-28 14:09, David Yan wrote:
>>> dtgateway does not use the native libraries.
>>>
>>> I just tried a clean install of RTS 3.5.0 community edition on plain
>>> vanilla hadoop 2.7.3 on ubuntu 14.04 and was able to launch the pi
>>> demo from the web interface. I was able to do the same even with the
>>> native libraries removed.
>>>
>>> Thanks.  I thought it was a long shot.
>>>
>>> What is your fs.defaultFS property in hadoop? Can you verify that
>>> it
>>> points to your hdfs (hdfs://namenode:9000/) ?
>>>
>>> In $HADOOP_CONF_DIR/hdfs-site.xml, fs.defaultFS is
>>> hdfs://namenode:9000
>>>
>>> Note also that if I try to give a URL to the dtgateway Installation
>>> Wizard
>>> that isn't the namenode host/port, it complains immediately, instead
>>> of
>>> proceeding to the next page.
>>>
>>> Unfortunately I don't have access to an ARM64 so any finding you
>>> post
>>> here would be helpful.
>>>
>>> I'm not surprised.  A classic chicken-and-egg problem.  There aren't
>>> a lot of machines around, so the software doesn't get a lot of
>>> attention, so people don't buy machines.
>>>
>>> David
>>>
>>> On Wed, Sep 28, 2016 at 11:04 AM, <dbs@softiron.co.uk> wrote:
>>>
>>> I had a thought of what might be the problem:
>>> the hadoop native library doesn't build on SUSE/SLES ARM64,
>>> so I'm running with the pure Java library.
>>>
>>> Is it possible that the dtgateway is trying to use the native
>>> library
>>> directly?
>>>
>>> Thanks,
>>> -david
>>>
>>> On 2016-09-26 14:45, David Yan wrote:
>>>
>>> Do you see any exception stacktrace in the log when this error
>>> occurred:
>>>
>>> Wrong FS: hdfs://namenode:9000/user/dtadmin/datatorrent, expected:
>>> file:///
>>>
>>> David
>>>
>>> On Mon, Sep 26, 2016 at 2:39 PM, <dbs@softiron.co.uk> wrote:
>>>
>>> From    David Yan <david@datatorrent.com>
>>> Date    Sat 00:12
>>>
>>> Can you please provide any exception stacktrace in the
>>> dtgateway.log file when that happens?
>>>
>>> I reran the dtgateway installation wizard, which failed (same as
>>> before) with error msg:
>>>
>>> | DFS directory cannot be written to with error message "Mkdirs
>>> failed to create /user/dtadmin/datatorrent (exists=false,
>>> cwd=file:/opt/datatorrent/releases/3.4.0)"
>>>
>>> The corresponding exception in dtgateway.log is:
>>>
>>> | 2016-09-26 13:43:54,767 ERROR com.datatorrent.gateway.I: DFS
>>> Directory cannot be written to with exception:
>>> | java.io.IOException: Mkdirs failed to create
>>> /user/dtadmin/datatorrent (exists=false,
>>> cwd=file:/opt/datatorrent/releases/3.4.0)
>>> |     at
>>>
>>  org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFile
>> System.java:455)
>>
>> |     at
>>>
>>  org.apache.hadoop.fs.ChecksumFileSystem.create(ChecksumFile
>> System.java:440)
>>
>> |     at
>>> org.apache.hadoop.fs.FileSystem.create(FileSystem.java:911)
>>> |     at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:892)
>>> |     at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:789)
>>> |     at org.apache.hadoop.fs.FileSystem.create(FileSystem.java:778)
>>> |     at
>>> com.datatorrent.stram.client.FSAgent.createFile(FSAgent.java:77)
>>> |     at com.datatorrent.gateway.I.h(gc:324)
>>> |     at com.datatorrent.gateway.I.h(gc:284)
>>> |     at
>>> com.datatorrent.gateway.resources.ws.v2.ConfigResource.h(fc:136)
>>> |     at
>>> com.datatorrent.gateway.resources.ws.v2.ConfigResource.h(fc:171)
>>> |     at
>>>
>>  com.datatorrent.gateway.resources.ws.v2.ConfigResource.setC
>> onfigProperty(fc:34)
>>
>> |     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
>>> Method)
>>> |     at
>>>
>>  sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAcc
>> essorImpl.java:62)
>>
>> |     at
>>>
>>
>> sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMe
>> thodAccessorImpl.java:43)
>>
>> |     at java.lang.reflect.Method.invoke(Method.java:498)
>>> |     at
>>>
>>
>> com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invo
>> ke(JavaMethodInvokerFactory.java:60)
>>
>> |     at
>>>
>>
>> com.sun.jersey.server.impl.model.method.dispatch.AbstractRes
>> ourceMethodDispatchProvider$ResponseOutInvoker._dispatch(A
>> bstractResourceMethodDispatchProvider.java:205)
>>
>> |     at
>>>
>>
>> com.sun.jersey.server.impl.model.method.dispatch.ResourceJav
>> aMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
>>
>> |     at
>>>
>>
>> com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(
>> HttpMethodRule.java:288)
>>
>> |     at
>>>
>>
>> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accep
>> t(RightHandPathRule.java:147)
>>
>> |     at
>>>
>>
>> com.sun.jersey.server.impl.uri.rules.SubLocatorRule.accept(
>> SubLocatorRule.java:134)
>>
>> |     at
>>>
>>
>> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accep
>> t(RightHandPathRule.java:147)
>>
>> |     at
>>>
>>
>> com.sun.jersey.server.impl.uri.rules.SubLocatorRule.accept(
>> SubLocatorRule.java:134)
>>
>> |     at
>>>
>>
>> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accep
>> t(RightHandPathRule.java:147)
>>
>> |     at
>>>
>>
>> com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accep
>> t(ResourceClassRule.java:108)
>>
>> |     at
>>>
>>
>> com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accep
>> t(RightHandPathRule.java:147)
>>
>> |     at
>>>
>>
>> com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule
>> .accept(RootResourceClassesRule.java:84)
>>
>> |     at
>>>
>>
>> com.sun.jersey.server.impl.application.WebApplicationImpl._h
>> andleRequest(WebApplicationImpl.java:1469)
>>
>> |     at
>>>
>>
>> com.sun.jersey.server.impl.application.WebApplicationImpl._h
>> andleRequest(WebApplicationImpl.java:1400)
>>
>> |     at
>>>
>>
>> com.sun.jersey.server.impl.application.WebApplicationImpl.ha
>> ndleRequest(WebApplicationImpl.java:1349)
>>
>> |     at
>>>
>>
>> com.sun.jersey.server.impl.application.WebApplicationImpl.ha
>> ndleRequest(WebApplicationImpl.java:1339)
>>
>> |     at
>>>
>>
>> com.sun.jersey.spi.container.servlet.WebComponent.service(We
>> bComponent.java:416)
>>
>> |     at
>>>
>>
>> com.sun.jersey.spi.container.servlet.ServletContainer.servic
>> e(ServletContainer.java:537)
>>
>> |     at
>>>
>>
>> com.sun.jersey.spi.container.servlet.ServletContainer.servic
>> e(ServletContainer.java:699)
>>
>> |     at
>>> javax.servlet.http.HttpServlet.service(HttpServlet.java:848)
>>> |     at
>>>
>>  org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:669)
>>
>> |     at
>>>
>>  org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHa
>> ndler.java:457)
>>
>> |     at
>>>
>>
>> org.eclipse.jetty.server.session.SessionHandler.doHandle(
>> SessionHandler.java:229)
>>
>> |     at
>>>
>>
>> org.eclipse.jetty.server.handler.ContextHandler.doHandle(
>> ContextHandler.java:1075)
>>
>> |     at
>>>
>>  org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHan
>> dler.java:384)
>>
>> |     at
>>>
>>
>> org.eclipse.jetty.server.session.SessionHandler.doScope(
>> SessionHandler.java:193)
>>
>> |     at
>>>
>>
>> org.eclipse.jetty.server.handler.ContextHandler.doScope(
>> ContextHandler.java:1009)
>>
>> |     at
>>>
>>  org.eclipse.jetty.server.handler.ScopedHandler.handle(Scope
>> dHandler.java:135)
>>
>> |     at
>>>
>>
>> org.eclipse.jetty.server.handler.HandlerCollection.handle(
>> HandlerCollection.java:154)
>>
>> |     at
>>>
>>  org.eclipse.jetty.server.handler.HandlerWrapper.handle(Hand
>> lerWrapper.java:116)
>>
>> |     at org.eclipse.jetty.server.Server.handle(Server.java:368)
>>> |     at
>>>
>>
>> org.eclipse.jetty.server.AbstractHttpConnection.handleReques
>> t(AbstractHttpConnection.java:489)
>>
>> |     at
>>>
>>
>> org.eclipse.jetty.server.AbstractHttpConnection.content(Abst
>> ractHttpConnection.java:953)
>>
>> |     at
>>>
>>
>> org.eclipse.jetty.server.AbstractHttpConnection$RequestHandl
>> er.content(AbstractHttpConnection.java:1014)
>>
>> |     at
>>> org.eclipse.jetty.http.HttpParser.parseNext(HttpParser.java:861)
>>> |     at
>>>
>>  org.eclipse.jetty.http.HttpParser.parseAvailable(HttpParser.java:240)
>>
>> |     at
>>>
>>
>> org.eclipse.jetty.server.AsyncHttpConnection.handle(AsyncHtt
>> pConnection.java:82)
>>
>> |     at
>>>
>>
>> org.eclipse.jetty.io.nio.SelectChannelEndPoint.handle(Select
>> ChannelEndPoint.java:628)
>>
>> |     at
>>>
>>
>> org.eclipse.jetty.io.nio.SelectChannelEndPoint$1.run(SelectC
>> hannelEndPoint.java:52)
>>
>> |     at
>>>
>>
>> org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(Queued
>> ThreadPool.java:608)
>>
>> |     at
>>>
>>  org.eclipse.jetty.util.thread.QueuedThreadPool$3.run(
>> QueuedThreadPool.java:543)
>>
>> |     at java.lang.Thread.run(Thread.java:745)
>>> | 2016-09-26 13:43:54,772 INFO
>>> com.datatorrent.gateway.resources.ws.v2.WSResource: Caught exception
>>> in processing web service:
>>> com.datatorrent.stram.client.DTConfiguration$ConfigException: DFS
>>> directory cannot be written to with error message "Mkdirs failed to
>>> create /user/dtadmin/datatorrent (exists=false,
>>> cwd=file:/opt/datatorrent/releases/3.4.0)"
>>>
>>> Also, when I shutdown the gateway, I get a different exception in
>>> dtgateway.log:
>>>
>>> | 2016-09-26 13:40:44,366 INFO com.datatorrent.gateway.DTGateway:
>>> Shutting down
>>> | 2016-09-26 13:40:44,427 ERROR com.datatorrent.gateway.I: DFS
>>> Directory cannot be written to with exception:
>>> | java.io.IOException: Filesystem closed
>>> |     at
>>> org.apache.hadoop.hdfs.DFSClient.checkOpen(DFSClient.java:808)
>>> |     at
>>> org.apache.hadoop.hdfs.DFSClient.delete(DFSClient.java:2041)
>>> |     at
>>>
>>
>> org.apache.hadoop.hdfs.DistributedFileSystem$14.doCall(
>> DistributedFileSystem.java:707)
>>
>> |     at
>>>
>>
>> org.apache.hadoop.hdfs.DistributedFileSystem$14.doCall(
>> DistributedFileSystem.java:703)
>>
>> |     at
>>>
>>
>> org.apache.hadoop.fs.FileSystemLinkResolver.resolve(FileSyst
>> emLinkResolver.java:81)
>>
>> |     at
>>>
>>
>> org.apache.hadoop.hdfs.DistributedFileSystem.delete(Distribu
>> tedFileSystem.java:714)
>>
>> |     at
>>> com.datatorrent.stram.client.FSAgent.deleteFile(FSAgent.java:94)
>>> |     at com.datatorrent.gateway.I.h(gc:30)
>>> |     at com.datatorrent.gateway.l.K(pc:158)
>>> |     at com.datatorrent.gateway.l.h(pc:65)
>>> |     at com.datatorrent.gateway.K.run(yc:156)
>>> |     at java.lang.Thread.run(Thread.java:745)
>>> | 2016-09-26 13:40:44,444 INFO com.datatorrent.gateway.DTGateway:
>>> Shutdown complete
>>>
>>> Thanks much.
>>> -david
>>>
>>> On Tue, Sep 13, 2016 at 4:06 PM, <dbs@softiron.co.uk> wrote:
>>>
>>> I have the dtgateway running (community edition) with
>>> apex/malhar 3.4 and hadoop 2.7.4 on SuSE SLES 12.1 on an ARM64
>>> server (aarch64).
>>>
>>> In the Installation Wizard, I set the DFS location to:
>>>
>>> hdfs://namenode:9000/user/dtadmin/datatorrent
>>>
>>> and the gateway saves the hadoop configuration and restarts
>>> successfully, but  I get an error:
>>>
>>> Wrong FS: hdfs://namenode:9000/user/dtadmin/datatorrent,
>>> expected: file:///
>>>
>>> the 'service dtgateway status' command says it's running.
>>> The log file has the same error as above, but nothing else
>>> useful.
>>>
>>> Is there a manual way to move the demo apps into the
>>> gateway?
>>>
>>> If I ignore this error and try to upload or import a demo
>>> app, I get the same error.
>>>
>>> What's odd is that the gateway does write into the HDFS
>>> directory:
>>>
>>> Found 6 items
>>> drwxrwxrwt   - dtadmin hadoop          0 2016-09-13 15:35
>>> /user/dtadmin/datatorrent/appPackages
>>> drwxrwxrwt   - dtadmin hadoop          0 2016-09-13 15:35
>>> /user/dtadmin/datatorrent/apps
>>> drwxrwxrwt   - dtadmin hadoop          0 2016-09-13 15:35
>>> /user/dtadmin/datatorrent/conf
>>> drwxrwxrwt   - dtadmin hadoop          0 2016-09-13 15:35
>>> /user/dtadmin/datatorrent/dashboards
>>> drwxrwxrwt   - dtadmin hadoop          0 2016-09-13 15:35
>>> /user/dtadmin/datatorrent/licenses
>>> drwxrwxrwt   - dtadmin hadoop          0 2016-09-13 15:35
>>> /user/dtadmin/datatorrent/systemAlerts
>>>
>>> I've tried playing with different ways to configure the DFS
>>> location, but nothing works.
>>> If I use an actual local filesystem directory, I get a
>>> different errro from the installation wizard (Non-DFS file system
>>> is used: org.apache.hadoop.fs.LocalFileSystem), which makes sense.
>>> But if I ignore this error, the app wizard successsfully writes
>>> packages into the local directory, but I can't launch them (no
>>> surprise there).
>>>
>>> Anybody have any ideas what I can do?
>>>
>>> thanks much.
>>> -david
>>>
>>
>

Mime
View raw message