cloudstack-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrija Panic <andrija.pa...@gmail.com>
Subject [HELP needed] haproxy vs mysql client timeout
Date Mon, 08 Jun 2015 12:42:49 GMT
Hi,

I'm running into intermittent problems with ACS 4.5.1 mgmt server - it
seems it's loosing connection to the haproxy/mysql, since we are using
haproxy VIP to in db.properties, and this haproxy is using 1 galera node in
the backend section...(2 others are backup etc).


haproxy timeout is 100sec.
mysql timeouts are all defaults, except innodb_lock_wait_timeout.

+-----------------------------+----------+
| Variable_name               | Value    |
+-----------------------------+----------+
| connect_timeout             | 10       |
| delayed_insert_timeout      | 300      |
| have_statement_timeout      | YES      |
| innodb_flush_log_at_timeout | 1        |
| innodb_lock_wait_timeout    | 600      |
| innodb_rollback_on_timeout  | ON       |
| interactive_timeout         | 28800    |
| net_read_timeout            | 30       |
| net_write_timeout           | 60       |
| thread_pool_idle_timeout    | 60       |
| wait_timeout                | 28800    |
+-----------------------------+----------+

To make things more interesting, all 3 servers (acs mgmt, haproxy,
galera-node1) are on the same physical host, so I rule out network
issues... and I dont see anything interesting in the haproxy logs, also no
backend downtimes etc...


The errors are like folowing, from mgmt logs, but after that mgmt server
continues to work fine...

Any suggestions on tuning timeouts ?


Caused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException:
Communications link failure

The last packet successfully received from the server was 137,744
milliseconds ago.  The last packet sent successfully to the server was 1
milliseconds ago.
        at sun.reflect.GeneratedConstructorAccessor133.newInstance(Unknown
Source)
        at
sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
        at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
        at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
        at
com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1129)
        at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3720)
        at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3609)
        at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4160)
        at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2617)
        at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2778)
        at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2825)
        at
com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:2156)
        at
com.mysql.jdbc.PreparedStatement.executeQuery(PreparedStatement.java:2313)
        at
org.apache.commons.dbcp.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:96)
        at
org.apache.commons.dbcp.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:96)
        at
com.cloud.utils.db.GenericDaoBase.findById(GenericDaoBase.java:1009)
        ... 61 more
Caused by: java.io.EOFException: Can not read response from server.
Expected to read 4 bytes, read 0 bytes before connection was unexpectedly
lost.
        ... 76 more

Thanks,

-- 

Andrija Panić

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message