apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chetan Narsude <che...@datatorrent.com>
Subject Re: ConnectExceptions when running Apex
Date Thu, 27 Aug 2015 22:20:16 GMT
Ilya,

  The error you are seeing is coming from the networking library
underneath. It's async networking library. It throws that error after the
tcp/ip socket connection is being shutdown(closed) and something tries to
write to it.

  The reason for that is that your FeatureGenerator operator is unaware
that it's being shutdown and is still trying to process the data. Is your
code ignoring(swallowing) the InterruptedException?

--
Chetan





On Thu, Aug 27, 2015 at 3:09 PM, Ganelin, Ilya <Ilya.Ganelin@capitalone.com>
wrote:

> Not from my code, I see a bunch of these:
> java.io.IOException: Connection is not open
>         at
> com.datatorrent.common.util.PubSubWebSocketClient.assertUsable(PubSubWebSoc
> ketClient.java:280)
>         at
> com.datatorrent.common.util.PubSubWebSocketClient.publish(PubSubWebSocketCl
> ient.java:303)
>         at
> com.datatorrent.stram.util.SharedPubSubWebSocketClient.publish(SharedPubSub
> WebSocketClient.java:113)
>         at
> com.datatorrent.stram.FSEventRecorder$EventRecorderThread.run(FSEventRecord
> er.java:73)
>
> And then the error below, which crashes my operator and causes a restart.
>
> On 8/27/15, 2:52 PM, "Gaurav Gupta" <gaurav@datatorrent.com> wrote:
>
> >Do you see any other exception in your operator?
> >
> >Thanks
> >- Gaurav
> >
> >> On Aug 27, 2015, at 2:42 PM, Ganelin, Ilya
> >><Ilya.Ganelin@capitalone.com> wrote:
> >>
> >> Hi all - one more cropping up a lot is this one:
> >>
> >> Apex Exceptions:2015-08-26 19:33:13,547 INFO
> >> com.datatorrent.stram.StreamingContainerParent: child msg: Stopped
> >>running
> >> due to an exception.
> >> com.datatorrent.netlet.NetletThrowable$NetletRuntimeException:
> >> java.lang.UnsupportedOperationException: Client does not own the socket
> >> any longer!
> >>        at
> >> com.datatorrent.netlet.AbstractClient$1.offer(AbstractClient.java:316)
> >>        at
> >> com.datatorrent.netlet.AbstractClient$1.offer(AbstractClient.java:306)
> >>        at
> >> com.datatorrent.netlet.AbstractClient.send(AbstractClient.java:252)
> >>        at
> >>
> >>com.datatorrent.netlet.AbstractLengthPrependerClient.write(AbstractLength
> >>Pr
> >> ependerClient.java:236)
> >>        at
> >>
> >>com.datatorrent.netlet.AbstractLengthPrependerClient.write(AbstractLength
> >>Pr
> >> ependerClient.java:190)
> >>        at
> >>
> >>com.datatorrent.stram.stream.BufferServerPublisher.put(BufferServerPublis
> >>he
> >> r.java:125)
> >>        at
> >> com.datatorrent.api.DefaultOutputPort.emit(DefaultOutputPort.java:48)
> >>        at
> >>
> >>com.capitalone.vault8.citadel.operators.h20.apex_demo.FeatureGenerator$1.
> >>pr
> >> ocess(FeatureGenerator.java:102)
> >>        at
> >>
> >>com.capitalone.vault8.citadel.operators.h20.apex_demo.FeatureGenerator$1.
> >>pr
> >> ocess(FeatureGenerator.java:31)
> >>        at
> >> com.datatorrent.api.DefaultInputPort.put(DefaultInputPort.java:67)
> >>        at
> >>
> >>com.datatorrent.stram.stream.BufferServerSubscriber$BufferReservoir.sweep
> >>(B
> >> ufferServerSubscriber.java:244)
> >>        at
> >> com.datatorrent.stram.engine.GenericNode.run(GenericNode.java:226)
> >>        at
> >>
> >>com.datatorrent.stram.engine.StreamingContainer$2.run(StreamingContainer.
> >>ja
> >> va:1363)
> >> Caused by: java.lang.UnsupportedOperationException: Client does not own
> >> the socket any longer!
> >>
> >>
> >> Any hints as to what¹s causing this would be great!
> >>
> >>
> >>
> >> On 8/26/15, 1:13 PM, "Ganelin, Ilya" <Ilya.Ganelin@capitalone.com
> >><mailto:Ilya.Ganelin@capitalone.com>> wrote:
> >>
> >>> Hi all ­ I¹m seeing the following exceptions constantly in my YARN logs
> >>> all seeming to do with network connectivity:
> >>>
> >>> I¹m wondering if you could shed some light on what is happening here.
> >>> Thanks!
> >>>
> >>> 2015-08-26 16:07:23,403 ERROR com.datatorrent.stram.FSEventRecorder:
> >>> Caught Exception
> >>> java.lang.RuntimeException: java.util.concurrent.TimeoutException: No
> >>> response received after 500
> >>>       at
> >>>
> >>>com.datatorrent.netlet.util.DTThrowable.wrapIfChecked(DTThrowable.java:5
> >>>0)
> >>>       at
> >>> com.datatorrent.netlet.util.DTThrowable.rethrow(DTThrowable.java:31)
> >>>       at
> >>>
> >>>com.datatorrent.common.util.PubSubWebSocketClient.assertUsable(PubSubWeb
> >>>So
> >>> cketClient.java:291)
> >>>       at
> >>>
> >>>com.datatorrent.common.util.PubSubWebSocketClient.publish(PubSubWebSocke
> >>>tC
> >>> lient.java:303)
> >>>       at
> >>>
> >>>com.datatorrent.stram.util.SharedPubSubWebSocketClient.publish(SharedPub
> >>>Su
> >>> bWebSocketClient.java:113)
> >>>       at
> >>>
> >>>com.datatorrent.stram.FSEventRecorder$EventRecorderThread.run(FSEventRec
> >>>or
> >>> der.java:73)
> >>> Caused by: java.util.concurrent.TimeoutException: No response received
> >>> after 500
> >>>       at
> >>>
> >>>com.ning.http.client.providers.netty.NettyResponseFuture.get(NettyRespon
> >>>se
> >>> Future.java:244)
> >>>       at
> >>>
> >>>com.datatorrent.common.util.PubSubWebSocketClient.openConnection(PubSubW
> >>>eb
> >>> SocketClient.java:188)
> >>>       at
> >>>
> >>>com.datatorrent.stram.util.SharedPubSubWebSocketClient.setup(SharedPubSu
> >>>bW
> >>> ebSocketClient.java:66)
> >>>       at
> >>>
> >>>com.datatorrent.stram.StreamingContainerManager.setupWsClient(StreamingC
> >>>on
> >>> tainerManager.java:476)
> >>>       at
> >>>
> >>>com.datatorrent.stram.StreamingContainerManager.<init>(StreamingContaine
> >>>rM
> >>> anager.java:331)
> >>>       at
> >>>
> >>>com.datatorrent.stram.StreamingContainerManager.getInstance(StreamingCon
> >>>ta
> >>> inerManager.java:2828)
> >>>       at
> >>>
> >>>com.datatorrent.stram.StreamingAppMasterService.serviceInit(StreamingApp
> >>>Ma
> >>> sterService.java:516)
> >>>       at
> >>>
> >>>org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
> >>>       at
> >>>
> >>>com.datatorrent.stram.StreamingAppMaster.main(StreamingAppMaster.java:98
> >>>)
> >>>
> >>> 015-08-26 16:08:24,721 ERROR com.datatorrent.stram.FSEventRecorder:
> >>> Caught Exception
> >>> java.net.ConnectException: connection timed out to
> >>> ws://10.0.2.15:9090/pubsub
> >>>       at
> >>>
> >>>com.ning.http.client.providers.netty.NettyConnectListener.operationCompl
> >>>et
> >>> e(NettyConnectListener.java:102)
> >>>       at
> >>>
> >>>org.jboss.netty.channel.DefaultChannelFuture.notifyListener(DefaultChann
> >>>el
> >>> Future.java:427)
> >>>       at
> >>>
> >>>org.jboss.netty.channel.DefaultChannelFuture.notifyListeners(DefaultChan
> >>>ne
> >>> lFuture.java:418)
> >>>       at
> >>>
> >>>org.jboss.netty.channel.DefaultChannelFuture.setFailure(DefaultChannelFu
> >>>tu
> >>> re.java:380)
> >>>       at
> >>>
> >>>org.jboss.netty.channel.socket.nio.NioClientBoss.processConnectTimeout(N
> >>>io
> >>> ClientBoss.java:139)
> >>>       at
> >>>
> >>>org.jboss.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.j
> >>>av
> >>> a:82)
> >>>       at
> >>>
> >>>org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSe
> >>>le
> >>> ctor.java:312)
> >>>       at
> >>>
> >>>org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:
> >>>41
> >>> )
> >>>       at
> >>>
> >>>org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.j
> >>>av
> >>> a:108)
> >>>       at
> >>>
> >>>org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWor
> >>>ke
> >>> r.java:42)
> >>>       at
> >>>
> >>>java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav
> >>>a:
> >>> 1145)
> >>>       at
> >>>
> >>>java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja
> >>>va
> >>> :615)
> >>>       at java.lang.Thread.run(Thread.java:745)
> >>> Caused by: java.net.ConnectException: connection timed out
> >>>       at
> >>>
> >>>org.jboss.netty.channel.socket.nio.NioClientBoss.processConnectTimeout(N
> >>>io
> >>> ClientBoss.java:136)
> >>>       at
> >>>
> >>>org.jboss.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.j
> >>>av
> >>> a:82)
> >>>       at
> >>>
> >>>org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSe
> >>>le
> >>> ctor.java:312)
> >>>       at
> >>>
> >>>org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:
> >>>41
> >>> )
> >>>       ... 3 more
> >>>
> >>> 2015-08-26 16:08:14,560 ERROR com.datatorrent.stram.FSEventRecorder:
> >>> Caught Exception
> >>> java.io.IOException: Connection is not open
> >>>       at
> >>>
> >>>com.datatorrent.common.util.PubSubWebSocketClient.assertUsable(PubSubWeb
> >>>So
> >>> cketClient.java:280)
> >>>       at
> >>>
> >>>com.datatorrent.common.util.PubSubWebSocketClient.publish(PubSubWebSocke
> >>>tC
> >>> lient.java:303)
> >>>       at
> >>>
> >>>com.datatorrent.stram.util.SharedPubSubWebSocketClient.publish(SharedPub
> >>>Su
> >>> bWebSocketClient.java:113)
> >>>       at
> >>>
> >>>com.datatorrent.stram.FSEventRecorder$EventRecorderThread.run(FSEventRec
> >>>or
> >>> der.java:73)
> >>> ________________________________________________________
> >>>
> >>> The information contained in this e-mail is confidential and/or
> >>> proprietary to Capital One and/or its affiliates and may only be used
> >>> solely in performance of work or services for Capital One. The
> >>> information transmitted herewith is intended only for use by the
> >>> individual or entity to which it is addressed. If the reader of this
> >>> message is not the intended recipient, you are hereby notified that any
> >>> review, retransmission, dissemination, distribution, copying or other
> >>>use
> >>> of, or taking of any action in reliance upon this information is
> >>>strictly
> >>> prohibited. If you have received this communication in error, please
> >>> contact the sender and delete the material from your computer.
> >>
> >> ________________________________________________________
> >>
> >> The information contained in this e-mail is confidential and/or
> >>proprietary to Capital One and/or its affiliates and may only be used
> >>solely in performance of work or services for Capital One. The
> >>information transmitted herewith is intended only for use by the
> >>individual or entity to which it is addressed. If the reader of this
> >>message is not the intended recipient, you are hereby notified that any
> >>review, retransmission, dissemination, distribution, copying or other
> >>use of, or taking of any action in reliance upon this information is
> >>strictly prohibited. If you have received this communication in error,
> >>please contact the sender and delete the material from your computer.
> >
>
> ________________________________________________________
>
> The information contained in this e-mail is confidential and/or
> proprietary to Capital One and/or its affiliates and may only be used
> solely in performance of work or services for Capital One. The information
> transmitted herewith is intended only for use by the individual or entity
> to which it is addressed. If the reader of this message is not the intended
> recipient, you are hereby notified that any review, retransmission,
> dissemination, distribution, copying or other use of, or taking of any
> action in reliance upon this information is strictly prohibited. If you
> have received this communication in error, please contact the sender and
> delete the material from your computer.
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message