stratos-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Martin Eppel (meppel)" <mep...@cisco.com>
Subject RE: [Error] Too many files open error when running Stratos
Date Tue, 14 Apr 2015 22:52:24 GMT
I have been out for a few days so I am not sure on the status of this issue but I rerun the
test (leaving stratos running idle) for a few hours with the latest commits from today, April
14
(commit 7b09bc1534c5708a47a7b61ae0a83f96957e7eeb, Author: Chamila de Alwis chamila@apache.org<mailto:chamila@apache.org>,
Date:   Tue Apr 14 21:11:22 2015 +0530)
and I still see the same issue happening,

Thanks

Martin

Exceptions are (after letting it run for a few hours) :

TID: [0] [STRATOS] [2015-04-14 15:37:21,331]  WARN {org.apache.thrift.server.TThreadPoolServer}
-  Transport error occurred during acceptance of message.
org.apache.thrift.transport.TTransportException: java.net.SocketException: Too many open files
        at org.apache.thrift.transport.TServerSocket.acceptImpl(TServerSocket.java:118)
        at org.apache.thrift.transport.TServerSocket.acceptImpl(TServerSocket.java:35)
        at org.apache.thrift.transport.TServerTransport.accept(TServerTransport.java:31)
        at org.apache.thrift.server.TThreadPoolServer.serve(TThreadPoolServer.java:106)
        at org.wso2.carbon.databridge.receiver.thrift.internal.ThriftDataReceiver$ServerThread.run(ThriftDataReceiver.java:199)
        at java.lang.Thread.run(Thread.java:745)
Caused by: java.net.SocketException: Too many open files
        at java.net.PlainSocketImpl.socketAccept(Native Method)
        at java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398)
        at java.net.ServerSocket.implAccept(ServerSocket.java:530)
        at java.net.ServerSocket.accept(ServerSocket.java:498)
        at org.apache.thrift.transport.TServerSocket.acceptImpl(TServerSocket.java:113)

Btw, this is easily reproducible by just starting up stratos and let it run idle for a few
hours,

Regards

Martin


From: Martin Eppel (meppel)
Sent: Thursday, April 02, 2015 5:45 PM
To: dev@stratos.apache.org<mailto:dev@stratos.apache.org>
Cc: Udara Liyanage; Isuru Haththotuwa
Subject: RE: [Error] Too many files open error when running Stratos

Hi Mariangela,

Here is an update – seems this is a different issue as STRATOS-723.
After letting the system run for a few hours with the below mentioned commit (8985d96eb811aa8e9ce2c114f1856b4c4e20517b)
included.
I still see the same issue as you reported (with exception as below). The stratos process
is still running, no OOM yet but logging has stopped.

I think this is an issue which needs further investigation,

Regards

Martin

WARN {org.apache.thrift.server.TThreadPoolServer} -  Transport error occurred during acceptance
of message.
org.apache.thrift.transport.TTransportException: java.net.SocketException: Too many open files
        at org.apache.thrift.transport.TServerSocket.acceptImpl(TServerSocket.java:118)
        at org.apache.thrift.transport.TServerSocket.acceptImpl(TServerSocket.java:35)
        at org.apache.thrift.transport.TServerTransport.accept(TServerTransport.java:31)
        at org.apache.thrift.server.TThreadPoolServer.serve(TThreadPoolServer.java:106)
        at org.wso2.carbon.databridge.receiver.thrift.internal.ThriftDataReceiver$ServerThread.run(ThriftDataReceiver.java:199)

From: Martin Eppel (meppel)
Sent: Thursday, April 02, 2015 1:36 PM
To: dev@stratos.apache.org<mailto:dev@stratos.apache.org>
Cc: Udara Liyanage; Isuru Haththotuwa
Subject: RE: [Error] Too many files open error when running Stratos

Hi Mariangela,

As an FYI, this was possibly caused by https://issues.apache.org/jira/browse/STRATOS-723.

A fix was just checked in for this issue to 4.1.0, see commit 8985d96eb811aa8e9ce2c114f1856b4c4e20517b.

Regards

Martin



From: Mariangela Hills [mailto:mariangela@wso2.com]
Sent: Thursday, April 02, 2015 3:22 AM
To: dev
Cc: Udara Liyanage; Isuru Haththotuwa
Subject: Re: [Error] Too many files open error when running Stratos

I have created a JIRA [1] to track this issue!

[1] https://issues.apache.org/jira/browse/STRATOS-1308

Regards,
Mariangela
--
Mariangela Hills
Senior Technical Writer
WSO2, Inc.
lean.enterprise.middleware.
m: +94 773 500185
w: http://wso2.com<http://wso2.com/>
[http://c.content.wso2.com/signatures/general.png]<http://wso2.com/events/>

On Thu, Apr 2, 2015 at 3:43 PM, Mariangela Hills <mariangela@wso2.com<mailto:mariangela@wso2.com>>
wrote:
Thanks, Udara and Isuru for your quick response!

@Udara -

My file limit was 4000 and even after increasing it to 8000 I face the same issue:

ulimit -Sn 8000

Regards,
Mariangela
--
Mariangela Hills
Senior Technical Writer
WSO2, Inc.
lean.enterprise.middleware.


On Thu, Apr 2, 2015 at 3:29 PM, Isuru Haththotuwa <isuruh@apache.org<mailto:isuruh@apache.org>>
wrote:
Hi Mari,

Good catch. Yes, the reconnection logic seems to be missing the connection cleaning up part,
in case an error occurs in connecting to the broker. Will create a Jira to track this.

On Thu, Apr 2, 2015 at 3:13 PM, Mariangela Hills <mariangela@wso2.com<mailto:mariangela@wso2.com>>
wrote:
Hi Devs,

I got a message saying "could not connect to the message broker" even when the message broker
connection was already in place (started). There seems to be some reconnection logic, which
finally leads to a too many files opened error (part of the log can be seen in [1]). When
this happens the web browser on my machine (Chrome) gets stuck and I have to restart Stratos
to solve the issue. Any idea how I can over come this issue? I have a feeling that this issue
crops up when Stratos has been started and left idle for a considerable duration (i.e., 1
hr 30min).

--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------

[1]
[2015-04-02 13:40:16,319]  INFO {org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector}
-  Topic publisher will try to reconnect in 5 seconds: [topic-name] tenant.CompleteTenantEvent
[2015-04-02 13:40:16,319]  WARN {org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector}
-  Connection to the message broker failed
[2015-04-02 13:40:16,319]  INFO {org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector}
-  Topic publisher will try to reconnect in 2 seconds: [topic-name] tenant.CompleteTenantEvent
Exception in thread "ActiveMQ Transport: tcp://localhost/127.0.0.1:61616@50628" java.lang.OutOfMemoryError:
unable to create new native thread
at java.lang.Thread.start0(Native Method)
at java.lang.Thread.start(Thread.java:673)
at java.util.concurrent.ThreadPoolExecutor.addThread(ThreadPoolExecutor.java:681)
at java.util.concurrent.ThreadPoolExecutor.addIfUnderCorePoolSize(ThreadPoolExecutor.java:706)
at java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:650)
at org.apache.activemq.ActiveMQConnection.onAsyncException(ActiveMQConnection.java:2002)
at org.apache.activemq.ActiveMQConnection.onException(ActiveMQConnection.java:2017)[2015-04-02
13:40:16,423] ERROR {org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector}
-  Could not connect to message broker
javax.jms.JMSException: Cannot send, channel has already failed: tcp://127.0.0.1:61616<http://127.0.0.1:61616>
at org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:72)
at org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1435)
at org.apache.activemq.ActiveMQConnection.ensureConnectionInfoSent(ActiveMQConnection.java:1522)
at org.apache.activemq.ActiveMQConnection.start(ActiveMQConnection.java:527)
at org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector.connect(AmqpTopicConnector.java:78)
at org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicPublisher.reconnect(AmqpTopicPublisher.java:115)
at org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector$1.onException(AmqpTopicConnector.java:75)
at org.apache.activemq.ActiveMQConnection$5.run(ActiveMQConnection.java:2005)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
at java.lang.Thread.run(Thread.java:695)
Caused by: org.apache.activemq.transport.InactivityIOException: Cannot send, channel has already
failed: tcp://127.0.0.1:61616<http://127.0.0.1:61616>
at org.apache.activemq.transport.AbstractInactivityMonitor.doOnewaySend(AbstractInactivityMonitor.java:297)
at org.apache.activemq.transport.AbstractInactivityMonitor.oneway(AbstractInactivityMonitor.java:286)
at org.apache.activemq.transport.TransportFilter.oneway(TransportFilter.java:85)
at org.apache.activemq.transport.WireFormatNegotiator.oneway(WireFormatNegotiator.java:104)
at org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:68)
at org.apache.activemq.transport.ResponseCorrelator.asyncRequest(ResponseCorrelator.java:81)
at org.apache.activemq.transport.ResponseCorrelator.
request(ResponseCorrelator.java:86)
at org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1406)
... 9 more
at org.apache.activemq.transport.TransportFilter.onException(TransportFilter.java:101)
at org.apache.activemq.transport.ResponseCorrelator.onException(ResponseCorrelator.java:126)
at org.apache.activemq.transport.TransportFilter.onException(TransportFilter.java:101)
at org.apache.activemq.transport.TransportFilter.onException(TransportFilter.java:101)
at org.apache.activemq.transport.WireFormatNegotiator.onException(WireFormatNegotiator.java:160)
at org.apache.activemq.transport.AbstractInactivityMonitor.onException(AbstractInactivityMonitor.java:314)
at org.apache.activemq.transport.TransportSupport.onException(TransportSupport.java:96)
at org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:200)
at java.lang.Thread.run(Thread.java:695)
[2015-04-02 13:40:16,424]  WARN {org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector}
-  Could not reconnect to message broker
org.apache.stratos.messaging.domain.exception.MessagingException: Could not connect to message
broker
at org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector.connect(AmqpTopicConnector.java:82)
at org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicPublisher.reconnect(AmqpTopicPublisher.java:115)
at org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector$1.onException(AmqpTopicConnector.java:75)
at org.apache.activemq.ActiveMQConnection$5.run(ActiveMQConnection.java:2005)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
at java.lang.Thread.run(Thread.java:695)
Caused by: javax.jms.JMSException: Cannot send, channel has already failed: tcp://127.0.0.1:61616<http://127.0.0.1:61616>
at org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:72)
at org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1435)
at org.apache.activemq.ActiveMQConnection.ensureConnectionInfoSent(ActiveMQConnection.java:1522)
at org.apache.activemq.ActiveMQConnection.start(ActiveMQConnection.java:527)
at org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector.connect(AmqpTopicConnector.java:78)
... 6 more
Caused by: org.apache.activemq.transport.InactivityIOException: Cannot send, channel has already
failed: tcp://127.0.0.1:61616<http://127.0.0.1:61616>
at org.apache.activemq.transport.AbstractInactivityMonitor.doOnewaySend(AbstractInactivityMonitor.java:297)
at org.apache.activemq.transport.AbstractInactivityMonitor.oneway(AbstractInactivityMonitor.java:286)
at org.apache.activemq.transport.TransportFilter.oneway(TransportFilter.java:85)
at org.apache.activemq.transport.WireFormatNegotiator.oneway(WireFormatNegotiator.java:104)
at org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:68)
at org.apache.activemq.transport.ResponseCorrelator.asyncRequest(ResponseCorrelator.java:81)
at org.apache.activemq.transport.ResponseCorrelator.request(ResponseCorrelator.java:86)
at org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1406)
... 9 more
[2015-04-02 13:40:16,424] ERROR {org.apache.stratos.messaging.util.MessagingUtil} -  Failed
to load properties from file: /Users/Admin16/Documents/Stratos_4.1.0/master-pack/apache-stratos-4.1.0-SNAPSHOT/repository/conf/jndi.properties
java.io.FileNotFoundException: /Users/Admin16/Documents/Stratos_4.1.0/master-pack/apache-stratos-4.1.0-SNAPSHOT/repository/conf/jndi.properties
(Too many open files in system)
at java.io.FileInputStream.open(Native Method)
at java.io.FileInputStream.<init>(FileInputStream.java:120)
at org.apache.stratos.messaging.util.MessagingUtil.getProperties(MessagingUtil.java:108)
at org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector.create(AmqpTopicConnector.java:49)
at org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicPublisher.reconnect(AmqpTopicPublisher.java:114)
at org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector$1.onException(AmqpTopicConnector.java:75)
at org.apache.activemq.ActiveMQConnection$5.run(ActiveMQConnection.java:2005)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
at java.lang.Thread.run(Thread.java:695)
[2015-04-02 13:40:16,425]  INFO {org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector}
-  Topic publisher will try to reconnect in 20 seconds: [topic-name] application.signup.CompleteApplicationSignUpsEvent
[2015-04-02 13:40:16,427] ERROR {org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector}
-  Could not create topic connector
java.lang.NullPointerException
at org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector.create(AmqpTopicConnector.java:53)
at org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicPublisher.reconnect(AmqpTopicPublisher.java:114)
at org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector$1.onException(AmqpTopicConnector.java:75)
at org.apache.activemq.ActiveMQConnection$5.run(ActiveMQConnection.java:2005)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
at java.lang.Thread.run(Thread.java:695)
[2015-04-02 13:40:16,427]  WARN {org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector}
-  Could not reconnect to message broker
org.apache.stratos.messaging.domain.exception.MessagingException: Could not create topic connector
at org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector.create(AmqpTopicConnector.java:58)
at org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicPublisher.reconnect(AmqpTopicPublisher.java:114)
at org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector$1.onException(AmqpTopicConnector.java:75)
at org.apache.activemq.ActiveMQConnection$5.run(ActiveMQConnection.java:2005)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
at java.lang.Thread.run(Thread.java:695)
Caused by: java.lang.NullPointerException
at org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector.create(AmqpTopicConnector.java:53)
... 6 more
[2015-04-02 13:40:16,428]  INFO {org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector}
-  Topic publisher will try to reconnect in 20 seconds: [topic-name] application.signup.CompleteApplicationSignUpsEvent
[2015-04-02 13:40:16,445] ERROR {org.apache.stratos.messaging.util.MessagingUtil} -  Failed
to load properties from file: /Users/Admin16/Documents/Stratos_4.1.0/master-pack/apache-stratos-4.1.0-SNAPSHOT/repository/conf/jndi.properties
java.io.FileNotFoundException: /Users/Admin16/Documents/Stratos_4.1.0/master-pack/apache-stratos-4.1.0-SNAPSHOT/repository/conf/jndi.properties
(Too many open files in system)
at java.io.FileInputStream.open(Native Method)
at java.io.FileInputStream.<init>(FileInputStream.java:120)
at org.apache.stratos.messaging.util.MessagingUtil.getProperties(MessagingUtil.java:108)
at org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector.create(AmqpTopicConnector.java:49)
at org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicPublisher.reconnect(AmqpTopicPublisher.java:114)
at org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector$1.onException(AmqpTopicConnector.java:75)
at org.apache.activemq.ActiveMQConnection$5.run(ActiveMQConnection.java:2005)
at java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
at java.lang.Thread.run(Thread.java:695)
[2015-04-02 13:40:16,448] ERROR {org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector}
-  Could not create topic connector


Regards,
Mariangela
--
Mariangela Hills
Senior Technical Writer
WSO2, Inc.
lean.enterprise.middleware.

--
Thanks and Regards,

Isuru H.
+94 716 358 048<tel:%2B94%20716%20358%20048>



Mime
View raw message