stratos-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Chamila De Alwis <chami...@wso2.com>
Subject Re: [Error] Too many files open error when running Stratos
Date Wed, 15 Apr 2015 07:34:06 GMT
Hi,

Just as a side note, do we need to read the jndi.properties file every time
the connection is made? Is there a particular requirement that needs to
load changes to the jndi.properties file while Stratos is running? If not
we can use the initial read to the properties and re-use it in subsequent
connections. This does not however solve the original issue.


Regards,
Chamila de Alwis
Software Engineer | WSO2 | +94772207163
Blog: code.chamiladealwis.com



On Wed, Apr 15, 2015 at 9:06 AM, Isuru Haththotuwa <isuruh@apache.org>
wrote:

> I will look in to this Martin.
>
> On Wed, Apr 15, 2015 at 4:22 AM, Martin Eppel (meppel) <meppel@cisco.com>
> wrote:
>
>>  I have been out for a few days so I am not sure on the status of this
>> issue but I rerun the test (leaving stratos running idle) for a few hours
>> with the latest commits from today, April 14
>> (commit 7b09bc1534c5708a47a7b61ae0a83f96957e7eeb, Author: Chamila de
>> Alwis chamila@apache.org, Date:   Tue Apr 14 21:11:22 2015 +0530)
>>
>> and I still see the same issue happening,
>>
>>
>>
>> Thanks
>>
>>
>>
>> Martin
>>
>>
>>
>> Exceptions are (after letting it run for a few hours) :
>>
>>
>>
>> TID: [0] [STRATOS] [2015-04-14 15:37:21,331]  WARN
>> {org.apache.thrift.server.TThreadPoolServer} -  Transport error occurred
>> during acceptance of message.
>>
>> org.apache.thrift.transport.TTransportException:
>> java.net.SocketException: Too many open files
>>
>>         at
>> org.apache.thrift.transport.TServerSocket.acceptImpl(TServerSocket.java:118)
>>
>>         at
>> org.apache.thrift.transport.TServerSocket.acceptImpl(TServerSocket.java:35)
>>
>>         at
>> org.apache.thrift.transport.TServerTransport.accept(TServerTransport.java:31)
>>
>>         at
>> org.apache.thrift.server.TThreadPoolServer.serve(TThreadPoolServer.java:106)
>>
>>         at
>> org.wso2.carbon.databridge.receiver.thrift.internal.ThriftDataReceiver$ServerThread.run(ThriftDataReceiver.java:199)
>>
>>         at java.lang.Thread.run(Thread.java:745)
>>
>> Caused by: java.net.SocketException: Too many open files
>>
>>         at java.net.PlainSocketImpl.socketAccept(Native Method)
>>
>>         at
>> java.net.AbstractPlainSocketImpl.accept(AbstractPlainSocketImpl.java:398)
>>
>>         at java.net.ServerSocket.implAccept(ServerSocket.java:530)
>>
>>         at java.net.ServerSocket.accept(ServerSocket.java:498)
>>
>>         at
>> org.apache.thrift.transport.TServerSocket.acceptImpl(TServerSocket.java:113)
>>
>>
>>
>> Btw, this is easily reproducible by just starting up stratos and let it
>> run idle for a few hours,
>>
>>
>>
>> Regards
>>
>>
>>
>> Martin
>>
>>
>>
>>
>>
>> *From:* Martin Eppel (meppel)
>> *Sent:* Thursday, April 02, 2015 5:45 PM
>> *To:* dev@stratos.apache.org
>> *Cc:* Udara Liyanage; Isuru Haththotuwa
>> *Subject:* RE: [Error] Too many files open error when running Stratos
>>
>>
>>
>> Hi Mariangela,
>>
>>
>>
>> Here is an update – seems this is a different issue as STRATOS-723.
>>
>> After letting the system run for a few hours with the below mentioned
>> commit (8985d96eb811aa8e9ce2c114f1856b4c4e20517b) included.
>>
>> I still see the same issue as you reported (with exception as below). The
>> stratos process is still running, no OOM yet but logging has stopped.
>>
>>
>>
>> I think this is an issue which needs further investigation,
>>
>>
>>
>> Regards
>>
>>
>>
>> Martin
>>
>>
>>
>> WARN {org.apache.thrift.server.TThreadPoolServer} -  Transport error
>> occurred during acceptance of message.
>>
>> org.apache.thrift.transport.TTransportException:
>> java.net.SocketException: Too many open files
>>
>>         at
>> org.apache.thrift.transport.TServerSocket.acceptImpl(TServerSocket.java:118)
>>
>>         at
>> org.apache.thrift.transport.TServerSocket.acceptImpl(TServerSocket.java:35)
>>
>>         at
>> org.apache.thrift.transport.TServerTransport.accept(TServerTransport.java:31)
>>
>>         at
>> org.apache.thrift.server.TThreadPoolServer.serve(TThreadPoolServer.java:106)
>>
>>         at
>> org.wso2.carbon.databridge.receiver.thrift.internal.ThriftDataReceiver$ServerThread.run(ThriftDataReceiver.java:199)
>>
>>
>>
>> *From:* Martin Eppel (meppel)
>> *Sent:* Thursday, April 02, 2015 1:36 PM
>> *To:* dev@stratos.apache.org
>> *Cc:* Udara Liyanage; Isuru Haththotuwa
>> *Subject:* RE: [Error] Too many files open error when running Stratos
>>
>>
>>
>> Hi Mariangela,
>>
>>
>>
>> As an FYI, this was possibly caused by
>> https://issues.apache.org/jira/browse/STRATOS-723.
>>
>>
>>
>> A fix was just checked in for this issue to 4.1.0, see commit
>> 8985d96eb811aa8e9ce2c114f1856b4c4e20517b.
>>
>>
>>
>> Regards
>>
>>
>>
>> Martin
>>
>>
>>
>>
>>
>>
>>
>> *From:* Mariangela Hills [mailto:mariangela@wso2.com
>> <mariangela@wso2.com>]
>> *Sent:* Thursday, April 02, 2015 3:22 AM
>> *To:* dev
>> *Cc:* Udara Liyanage; Isuru Haththotuwa
>> *Subject:* Re: [Error] Too many files open error when running Stratos
>>
>>
>>
>> I have created a JIRA [1] to track this issue!
>>
>>
>>
>> [1] https://issues.apache.org/jira/browse/STRATOS-1308
>>
>>
>>    Regards,
>> Mariangela
>>
>> *--*
>>
>> Mariangela Hills
>>
>> Senior Technical Writer
>>
>>
>> *WSO2, Inc. *lean.enterprise.middleware.
>>
>> m: +94 773 500185
>> w: http://wso2.com
>>
>> [image: http://c.content.wso2.com/signatures/general.png]
>> <http://wso2.com/events/>
>>
>>
>>
>> On Thu, Apr 2, 2015 at 3:43 PM, Mariangela Hills <mariangela@wso2.com>
>> wrote:
>>
>> Thanks, Udara and Isuru for your quick response!
>>
>>
>>
>> @Udara -
>>
>>
>>
>> My file limit was 4000 and even after increasing it to 8000 I face the
>> same issue:
>>
>>
>>
>> ulimit -Sn 8000
>>
>>
>>    Regards,
>> Mariangela
>>
>> *--*
>>
>> Mariangela Hills
>>
>> Senior Technical Writer
>>
>>
>> *WSO2, Inc. *lean.enterprise.middleware.
>>
>>
>>
>>
>>
>> On Thu, Apr 2, 2015 at 3:29 PM, Isuru Haththotuwa <isuruh@apache.org>
>> wrote:
>>
>> Hi Mari,
>>
>> Good catch. Yes, the reconnection logic seems to be missing the
>> connection cleaning up part, in case an error occurs in connecting to the
>> broker. Will create a Jira to track this.
>>
>>
>>
>> On Thu, Apr 2, 2015 at 3:13 PM, Mariangela Hills <mariangela@wso2.com>
>> wrote:
>>
>> Hi Devs,
>>
>>
>>
>> I got a message saying "could not connect to the message broker" even
>> when the message broker connection was already in place (started). There
>> seems to be some reconnection logic, which finally leads to a too many
>> files opened error (part of the log can be seen in [1]). When this happens
>> the web browser on my machine (Chrome) gets stuck and I have to restart
>> Stratos to solve the issue. Any idea how I can over come this issue? I have
>> a feeling that this issue crops up when Stratos has been started and left
>> idle for a considerable duration (i.e., 1 hr 30min).
>>
>>
>>
>>
>> --------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
>>
>>
>>
>> [1]
>>
>> [2015-04-02 13:40:16,319]  INFO
>> {org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector} -
>>  Topic publisher will try to reconnect in 5 seconds: [topic-name]
>> tenant.CompleteTenantEvent
>>
>> [2015-04-02 13:40:16,319]  WARN
>> {org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector} -
>>  Connection to the message broker failed
>>
>> [2015-04-02 13:40:16,319]  INFO
>> {org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector} -
>>  Topic publisher will try to reconnect in 2 seconds: [topic-name]
>> tenant.CompleteTenantEvent
>>
>> Exception in thread "ActiveMQ Transport: tcp://localhost/127.0.0.1:61616@50628"
>> java.lang.OutOfMemoryError: unable to create new native thread
>>
>> at java.lang.Thread.start0(Native Method)
>>
>> at java.lang.Thread.start(Thread.java:673)
>>
>> at
>> java.util.concurrent.ThreadPoolExecutor.addThread(ThreadPoolExecutor.java:681)
>>
>> at
>> java.util.concurrent.ThreadPoolExecutor.addIfUnderCorePoolSize(ThreadPoolExecutor.java:706)
>>
>> at
>> java.util.concurrent.ThreadPoolExecutor.execute(ThreadPoolExecutor.java:650)
>>
>> at
>> org.apache.activemq.ActiveMQConnection.onAsyncException(ActiveMQConnection.java:2002)
>>
>> at
>> org.apache.activemq.ActiveMQConnection.onException(ActiveMQConnection.java:2017)[2015-04-02
>> 13:40:16,423] ERROR
>> {org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector} -
>>  Could not connect to message broker
>>
>> javax.jms.JMSException: Cannot send, channel has already failed: tcp://
>> 127.0.0.1:61616
>>
>> at
>> org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:72)
>>
>> at
>> org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1435)
>>
>> at
>> org.apache.activemq.ActiveMQConnection.ensureConnectionInfoSent(ActiveMQConnection.java:1522)
>>
>> at
>> org.apache.activemq.ActiveMQConnection.start(ActiveMQConnection.java:527)
>>
>> at
>> org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector.connect(AmqpTopicConnector.java:78)
>>
>> at
>> org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicPublisher.reconnect(AmqpTopicPublisher.java:115)
>>
>> at
>> org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector$1.onException(AmqpTopicConnector.java:75)
>>
>> at
>> org.apache.activemq.ActiveMQConnection$5.run(ActiveMQConnection.java:2005)
>>
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
>>
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
>>
>> at java.lang.Thread.run(Thread.java:695)
>>
>> Caused by: org.apache.activemq.transport.InactivityIOException: Cannot
>> send, channel has already failed: tcp://127.0.0.1:61616
>>
>> at
>> org.apache.activemq.transport.AbstractInactivityMonitor.doOnewaySend(AbstractInactivityMonitor.java:297)
>>
>> at
>> org.apache.activemq.transport.AbstractInactivityMonitor.oneway(AbstractInactivityMonitor.java:286)
>>
>> at
>> org.apache.activemq.transport.TransportFilter.oneway(TransportFilter.java:85)
>>
>> at
>> org.apache.activemq.transport.WireFormatNegotiator.oneway(WireFormatNegotiator.java:104)
>>
>> at
>> org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:68)
>>
>> at
>> org.apache.activemq.transport.ResponseCorrelator.asyncRequest(ResponseCorrelator.java:81)
>>
>> at org.apache.activemq.transport.ResponseCorrelator.
>>
>> request(ResponseCorrelator.java:86)
>>
>> at
>> org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1406)
>>
>> ... 9 more
>>
>> at
>> org.apache.activemq.transport.TransportFilter.onException(TransportFilter.java:101)
>>
>> at
>> org.apache.activemq.transport.ResponseCorrelator.onException(ResponseCorrelator.java:126)
>>
>> at
>> org.apache.activemq.transport.TransportFilter.onException(TransportFilter.java:101)
>>
>> at
>> org.apache.activemq.transport.TransportFilter.onException(TransportFilter.java:101)
>>
>> at
>> org.apache.activemq.transport.WireFormatNegotiator.onException(WireFormatNegotiator.java:160)
>>
>> at
>> org.apache.activemq.transport.AbstractInactivityMonitor.onException(AbstractInactivityMonitor.java:314)
>>
>> at
>> org.apache.activemq.transport.TransportSupport.onException(TransportSupport.java:96)
>>
>> at
>> org.apache.activemq.transport.tcp.TcpTransport.run(TcpTransport.java:200)
>>
>> at java.lang.Thread.run(Thread.java:695)
>>
>> [2015-04-02 13:40:16,424]  WARN
>> {org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector} -
>>  Could not reconnect to message broker
>>
>> org.apache.stratos.messaging.domain.exception.MessagingException: Could
>> not connect to message broker
>>
>> at
>> org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector.connect(AmqpTopicConnector.java:82)
>>
>> at
>> org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicPublisher.reconnect(AmqpTopicPublisher.java:115)
>>
>> at
>> org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector$1.onException(AmqpTopicConnector.java:75)
>>
>> at
>> org.apache.activemq.ActiveMQConnection$5.run(ActiveMQConnection.java:2005)
>>
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
>>
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
>>
>> at java.lang.Thread.run(Thread.java:695)
>>
>> Caused by: javax.jms.JMSException: Cannot send, channel has already
>> failed: tcp://127.0.0.1:61616
>>
>> at
>> org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:72)
>>
>> at
>> org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1435)
>>
>> at
>> org.apache.activemq.ActiveMQConnection.ensureConnectionInfoSent(ActiveMQConnection.java:1522)
>>
>> at
>> org.apache.activemq.ActiveMQConnection.start(ActiveMQConnection.java:527)
>>
>> at
>> org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector.connect(AmqpTopicConnector.java:78)
>>
>> ... 6 more
>>
>> Caused by: org.apache.activemq.transport.InactivityIOException: Cannot
>> send, channel has already failed: tcp://127.0.0.1:61616
>>
>> at
>> org.apache.activemq.transport.AbstractInactivityMonitor.doOnewaySend(AbstractInactivityMonitor.java:297)
>>
>> at
>> org.apache.activemq.transport.AbstractInactivityMonitor.oneway(AbstractInactivityMonitor.java:286)
>>
>> at
>> org.apache.activemq.transport.TransportFilter.oneway(TransportFilter.java:85)
>>
>> at
>> org.apache.activemq.transport.WireFormatNegotiator.oneway(WireFormatNegotiator.java:104)
>>
>> at
>> org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:68)
>>
>> at
>> org.apache.activemq.transport.ResponseCorrelator.asyncRequest(ResponseCorrelator.java:81)
>>
>> at
>> org.apache.activemq.transport.ResponseCorrelator.request(ResponseCorrelator.java:86)
>>
>> at
>> org.apache.activemq.ActiveMQConnection.syncSendPacket(ActiveMQConnection.java:1406)
>>
>> ... 9 more
>>
>> [2015-04-02 13:40:16,424] ERROR
>> {org.apache.stratos.messaging.util.MessagingUtil} -  Failed to load
>> properties from file:
>> /Users/Admin16/Documents/Stratos_4.1.0/master-pack/apache-stratos-4.1.0-SNAPSHOT/repository/conf/jndi.properties
>>
>> java.io.FileNotFoundException:
>> /Users/Admin16/Documents/Stratos_4.1.0/master-pack/apache-stratos-4.1.0-SNAPSHOT/repository/conf/jndi.properties
>> (Too many open files in system)
>>
>> at java.io.FileInputStream.open(Native Method)
>>
>> at java.io.FileInputStream.<init>(FileInputStream.java:120)
>>
>> at
>> org.apache.stratos.messaging.util.MessagingUtil.getProperties(MessagingUtil.java:108)
>>
>> at
>> org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector.create(AmqpTopicConnector.java:49)
>>
>> at
>> org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicPublisher.reconnect(AmqpTopicPublisher.java:114)
>>
>> at
>> org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector$1.onException(AmqpTopicConnector.java:75)
>>
>> at
>> org.apache.activemq.ActiveMQConnection$5.run(ActiveMQConnection.java:2005)
>>
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
>>
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
>>
>> at java.lang.Thread.run(Thread.java:695)
>>
>> [2015-04-02 13:40:16,425]  INFO
>> {org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector} -
>>  Topic publisher will try to reconnect in 20 seconds: [topic-name]
>> application.signup.CompleteApplicationSignUpsEvent
>>
>> [2015-04-02 13:40:16,427] ERROR
>> {org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector} -
>>  Could not create topic connector
>>
>> java.lang.NullPointerException
>>
>> at
>> org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector.create(AmqpTopicConnector.java:53)
>>
>> at
>> org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicPublisher.reconnect(AmqpTopicPublisher.java:114)
>>
>> at
>> org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector$1.onException(AmqpTopicConnector.java:75)
>>
>> at
>> org.apache.activemq.ActiveMQConnection$5.run(ActiveMQConnection.java:2005)
>>
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
>>
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
>>
>> at java.lang.Thread.run(Thread.java:695)
>>
>> [2015-04-02 13:40:16,427]  WARN
>> {org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector} -
>>  Could not reconnect to message broker
>>
>> org.apache.stratos.messaging.domain.exception.MessagingException: Could
>> not create topic connector
>>
>> at
>> org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector.create(AmqpTopicConnector.java:58)
>>
>> at
>> org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicPublisher.reconnect(AmqpTopicPublisher.java:114)
>>
>> at
>> org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector$1.onException(AmqpTopicConnector.java:75)
>>
>> at
>> org.apache.activemq.ActiveMQConnection$5.run(ActiveMQConnection.java:2005)
>>
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
>>
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
>>
>> at java.lang.Thread.run(Thread.java:695)
>>
>> Caused by: java.lang.NullPointerException
>>
>> at
>> org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector.create(AmqpTopicConnector.java:53)
>>
>> ... 6 more
>>
>> [2015-04-02 13:40:16,428]  INFO
>> {org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector} -
>>  Topic publisher will try to reconnect in 20 seconds: [topic-name]
>> application.signup.CompleteApplicationSignUpsEvent
>>
>> [2015-04-02 13:40:16,445] ERROR
>> {org.apache.stratos.messaging.util.MessagingUtil} -  Failed to load
>> properties from file:
>> /Users/Admin16/Documents/Stratos_4.1.0/master-pack/apache-stratos-4.1.0-SNAPSHOT/repository/conf/jndi.properties
>>
>> java.io.FileNotFoundException:
>> /Users/Admin16/Documents/Stratos_4.1.0/master-pack/apache-stratos-4.1.0-SNAPSHOT/repository/conf/jndi.properties
>> (Too many open files in system)
>>
>> at java.io.FileInputStream.open(Native Method)
>>
>> at java.io.FileInputStream.<init>(FileInputStream.java:120)
>>
>> at
>> org.apache.stratos.messaging.util.MessagingUtil.getProperties(MessagingUtil.java:108)
>>
>> at
>> org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector.create(AmqpTopicConnector.java:49)
>>
>> at
>> org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicPublisher.reconnect(AmqpTopicPublisher.java:114)
>>
>> at
>> org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector$1.onException(AmqpTopicConnector.java:75)
>>
>> at
>> org.apache.activemq.ActiveMQConnection$5.run(ActiveMQConnection.java:2005)
>>
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:895)
>>
>> at
>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:918)
>>
>> at java.lang.Thread.run(Thread.java:695)
>>
>> [2015-04-02 13:40:16,448] ERROR
>> {org.apache.stratos.messaging.broker.connect.amqp.AmqpTopicConnector} -
>>  Could not create topic connector
>>
>>
>>
>>
>>     Regards,
>> Mariangela
>>
>> *--*
>>
>> Mariangela Hills
>>
>> Senior Technical Writer
>>
>>
>> *WSO2, Inc. *lean.enterprise.middleware.
>>
>> --
>>
>> Thanks and Regards,
>>
>> Isuru H.
>>
>> +94 716 358 048
>>
>> --
>> <%2B94%20716%20358%20048>
>> Thanks and Regards,
>>
>> Isuru H.
>> <%2B94%20716%20358%20048>
>> +94 716 358 048* <http://wso2.com/>*
>>
>>
>> * <http://wso2.com/>*
>>
>>
>>

Mime
View raw message