apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Ganelin, Ilya" <Ilya.Gane...@capitalone.com>
Subject Re: ConnectExceptions when running Apex
Date Thu, 27 Aug 2015 22:09:45 GMT
Not from my code, I see a bunch of these:
java.io.IOException: Connection is not open
	at 
com.datatorrent.common.util.PubSubWebSocketClient.assertUsable(PubSubWebSoc
ketClient.java:280)
	at 
com.datatorrent.common.util.PubSubWebSocketClient.publish(PubSubWebSocketCl
ient.java:303)
	at 
com.datatorrent.stram.util.SharedPubSubWebSocketClient.publish(SharedPubSub
WebSocketClient.java:113)
	at 
com.datatorrent.stram.FSEventRecorder$EventRecorderThread.run(FSEventRecord
er.java:73)

And then the error below, which crashes my operator and causes a restart.

On 8/27/15, 2:52 PM, "Gaurav Gupta" <gaurav@datatorrent.com> wrote:

>Do you see any other exception in your operator?
>
>Thanks
>- Gaurav
>
>> On Aug 27, 2015, at 2:42 PM, Ganelin, Ilya
>><Ilya.Ganelin@capitalone.com> wrote:
>> 
>> Hi all - one more cropping up a lot is this one:
>> 
>> Apex Exceptions:2015-08-26 19:33:13,547 INFO
>> com.datatorrent.stram.StreamingContainerParent: child msg: Stopped
>>running
>> due to an exception.
>> com.datatorrent.netlet.NetletThrowable$NetletRuntimeException:
>> java.lang.UnsupportedOperationException: Client does not own the socket
>> any longer!
>>        at 
>> com.datatorrent.netlet.AbstractClient$1.offer(AbstractClient.java:316)
>>        at 
>> com.datatorrent.netlet.AbstractClient$1.offer(AbstractClient.java:306)
>>        at 
>> com.datatorrent.netlet.AbstractClient.send(AbstractClient.java:252)
>>        at 
>> 
>>com.datatorrent.netlet.AbstractLengthPrependerClient.write(AbstractLength
>>Pr
>> ependerClient.java:236)
>>        at 
>> 
>>com.datatorrent.netlet.AbstractLengthPrependerClient.write(AbstractLength
>>Pr
>> ependerClient.java:190)
>>        at 
>> 
>>com.datatorrent.stram.stream.BufferServerPublisher.put(BufferServerPublis
>>he
>> r.java:125)
>>        at 
>> com.datatorrent.api.DefaultOutputPort.emit(DefaultOutputPort.java:48)
>>        at 
>> 
>>com.capitalone.vault8.citadel.operators.h20.apex_demo.FeatureGenerator$1.
>>pr
>> ocess(FeatureGenerator.java:102)
>>        at 
>> 
>>com.capitalone.vault8.citadel.operators.h20.apex_demo.FeatureGenerator$1.
>>pr
>> ocess(FeatureGenerator.java:31)
>>        at 
>> com.datatorrent.api.DefaultInputPort.put(DefaultInputPort.java:67)
>>        at 
>> 
>>com.datatorrent.stram.stream.BufferServerSubscriber$BufferReservoir.sweep
>>(B
>> ufferServerSubscriber.java:244)
>>        at 
>> com.datatorrent.stram.engine.GenericNode.run(GenericNode.java:226)
>>        at 
>> 
>>com.datatorrent.stram.engine.StreamingContainer$2.run(StreamingContainer.
>>ja
>> va:1363)
>> Caused by: java.lang.UnsupportedOperationException: Client does not own
>> the socket any longer!
>> 
>> 
>> Any hints as to what¹s causing this would be great!
>> 
>> 
>> 
>> On 8/26/15, 1:13 PM, "Ganelin, Ilya" <Ilya.Ganelin@capitalone.com
>><mailto:Ilya.Ganelin@capitalone.com>> wrote:
>> 
>>> Hi all ­ I¹m seeing the following exceptions constantly in my YARN logs
>>> all seeming to do with network connectivity:
>>> 
>>> I¹m wondering if you could shed some light on what is happening here.
>>> Thanks!
>>> 
>>> 2015-08-26 16:07:23,403 ERROR com.datatorrent.stram.FSEventRecorder:
>>> Caught Exception
>>> java.lang.RuntimeException: java.util.concurrent.TimeoutException: No
>>> response received after 500
>>>       at 
>>> 
>>>com.datatorrent.netlet.util.DTThrowable.wrapIfChecked(DTThrowable.java:5
>>>0)
>>>       at 
>>> com.datatorrent.netlet.util.DTThrowable.rethrow(DTThrowable.java:31)
>>>       at 
>>> 
>>>com.datatorrent.common.util.PubSubWebSocketClient.assertUsable(PubSubWeb
>>>So
>>> cketClient.java:291)
>>>       at 
>>> 
>>>com.datatorrent.common.util.PubSubWebSocketClient.publish(PubSubWebSocke
>>>tC
>>> lient.java:303)
>>>       at 
>>> 
>>>com.datatorrent.stram.util.SharedPubSubWebSocketClient.publish(SharedPub
>>>Su
>>> bWebSocketClient.java:113)
>>>       at 
>>> 
>>>com.datatorrent.stram.FSEventRecorder$EventRecorderThread.run(FSEventRec
>>>or
>>> der.java:73)
>>> Caused by: java.util.concurrent.TimeoutException: No response received
>>> after 500
>>>       at 
>>> 
>>>com.ning.http.client.providers.netty.NettyResponseFuture.get(NettyRespon
>>>se
>>> Future.java:244)
>>>       at 
>>> 
>>>com.datatorrent.common.util.PubSubWebSocketClient.openConnection(PubSubW
>>>eb
>>> SocketClient.java:188)
>>>       at 
>>> 
>>>com.datatorrent.stram.util.SharedPubSubWebSocketClient.setup(SharedPubSu
>>>bW
>>> ebSocketClient.java:66)
>>>       at 
>>> 
>>>com.datatorrent.stram.StreamingContainerManager.setupWsClient(StreamingC
>>>on
>>> tainerManager.java:476)
>>>       at 
>>> 
>>>com.datatorrent.stram.StreamingContainerManager.<init>(StreamingContaine
>>>rM
>>> anager.java:331)
>>>       at 
>>> 
>>>com.datatorrent.stram.StreamingContainerManager.getInstance(StreamingCon
>>>ta
>>> inerManager.java:2828)
>>>       at 
>>> 
>>>com.datatorrent.stram.StreamingAppMasterService.serviceInit(StreamingApp
>>>Ma
>>> sterService.java:516)
>>>       at 
>>> 
>>>org.apache.hadoop.service.AbstractService.init(AbstractService.java:163)
>>>       at 
>>> 
>>>com.datatorrent.stram.StreamingAppMaster.main(StreamingAppMaster.java:98
>>>)
>>> 
>>> 015-08-26 16:08:24,721 ERROR com.datatorrent.stram.FSEventRecorder:
>>> Caught Exception
>>> java.net.ConnectException: connection timed out to
>>> ws://10.0.2.15:9090/pubsub
>>>       at 
>>> 
>>>com.ning.http.client.providers.netty.NettyConnectListener.operationCompl
>>>et
>>> e(NettyConnectListener.java:102)
>>>       at 
>>> 
>>>org.jboss.netty.channel.DefaultChannelFuture.notifyListener(DefaultChann
>>>el
>>> Future.java:427)
>>>       at 
>>> 
>>>org.jboss.netty.channel.DefaultChannelFuture.notifyListeners(DefaultChan
>>>ne
>>> lFuture.java:418)
>>>       at 
>>> 
>>>org.jboss.netty.channel.DefaultChannelFuture.setFailure(DefaultChannelFu
>>>tu
>>> re.java:380)
>>>       at 
>>> 
>>>org.jboss.netty.channel.socket.nio.NioClientBoss.processConnectTimeout(N
>>>io
>>> ClientBoss.java:139)
>>>       at 
>>> 
>>>org.jboss.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.j
>>>av
>>> a:82)
>>>       at 
>>> 
>>>org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSe
>>>le
>>> ctor.java:312)
>>>       at 
>>> 
>>>org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:
>>>41
>>> )
>>>       at 
>>> 
>>>org.jboss.netty.util.ThreadRenamingRunnable.run(ThreadRenamingRunnable.j
>>>av
>>> a:108)
>>>       at 
>>> 
>>>org.jboss.netty.util.internal.DeadLockProofWorker$1.run(DeadLockProofWor
>>>ke
>>> r.java:42)
>>>       at 
>>> 
>>>java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.jav
>>>a:
>>> 1145)
>>>       at 
>>> 
>>>java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.ja
>>>va
>>> :615)
>>>       at java.lang.Thread.run(Thread.java:745)
>>> Caused by: java.net.ConnectException: connection timed out
>>>       at 
>>> 
>>>org.jboss.netty.channel.socket.nio.NioClientBoss.processConnectTimeout(N
>>>io
>>> ClientBoss.java:136)
>>>       at 
>>> 
>>>org.jboss.netty.channel.socket.nio.NioClientBoss.process(NioClientBoss.j
>>>av
>>> a:82)
>>>       at 
>>> 
>>>org.jboss.netty.channel.socket.nio.AbstractNioSelector.run(AbstractNioSe
>>>le
>>> ctor.java:312)
>>>       at 
>>> 
>>>org.jboss.netty.channel.socket.nio.NioClientBoss.run(NioClientBoss.java:
>>>41
>>> )
>>>       ... 3 more
>>> 
>>> 2015-08-26 16:08:14,560 ERROR com.datatorrent.stram.FSEventRecorder:
>>> Caught Exception
>>> java.io.IOException: Connection is not open
>>>       at 
>>> 
>>>com.datatorrent.common.util.PubSubWebSocketClient.assertUsable(PubSubWeb
>>>So
>>> cketClient.java:280)
>>>       at 
>>> 
>>>com.datatorrent.common.util.PubSubWebSocketClient.publish(PubSubWebSocke
>>>tC
>>> lient.java:303)
>>>       at 
>>> 
>>>com.datatorrent.stram.util.SharedPubSubWebSocketClient.publish(SharedPub
>>>Su
>>> bWebSocketClient.java:113)
>>>       at 
>>> 
>>>com.datatorrent.stram.FSEventRecorder$EventRecorderThread.run(FSEventRec
>>>or
>>> der.java:73)
>>> ________________________________________________________
>>> 
>>> The information contained in this e-mail is confidential and/or
>>> proprietary to Capital One and/or its affiliates and may only be used
>>> solely in performance of work or services for Capital One. The
>>> information transmitted herewith is intended only for use by the
>>> individual or entity to which it is addressed. If the reader of this
>>> message is not the intended recipient, you are hereby notified that any
>>> review, retransmission, dissemination, distribution, copying or other
>>>use
>>> of, or taking of any action in reliance upon this information is
>>>strictly
>>> prohibited. If you have received this communication in error, please
>>> contact the sender and delete the material from your computer.
>> 
>> ________________________________________________________
>> 
>> The information contained in this e-mail is confidential and/or
>>proprietary to Capital One and/or its affiliates and may only be used
>>solely in performance of work or services for Capital One. The
>>information transmitted herewith is intended only for use by the
>>individual or entity to which it is addressed. If the reader of this
>>message is not the intended recipient, you are hereby notified that any
>>review, retransmission, dissemination, distribution, copying or other
>>use of, or taking of any action in reliance upon this information is
>>strictly prohibited. If you have received this communication in error,
>>please contact the sender and delete the material from your computer.
>

________________________________________________________

The information contained in this e-mail is confidential and/or proprietary to Capital One
and/or its affiliates and may only be used solely in performance of work or services for Capital
One. The information transmitted herewith is intended only for use by the individual or entity
to which it is addressed. If the reader of this message is not the intended recipient, you
are hereby notified that any review, retransmission, dissemination, distribution, copying
or other use of, or taking of any action in reliance upon this information is strictly prohibited.
If you have received this communication in error, please contact the sender and delete the
material from your computer.
Mime
View raw message