cloudstack-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrija Panic <andrija.pa...@gmail.com>
Subject Re: [HELP needed] haproxy vs mysql client timeout
Date Mon, 08 Jun 2015 13:21:06 GMT
Actually, no...

On haproxy I have allowed all traffic with input interface matching the
internal interface (eth0), rest of nodes no firewall whatsoever (which will
be changed)..

On 8 June 2015 at 15:17, Rafael Fonseca <rsafonseca@gmail.com> wrote:

> Do you have a firewall in between that might be closing long or idle
> sessions?
>
> On Mon, Jun 8, 2015 at 3:14 PM, Andrija Panic <andrija.panic@gmail.com>
> wrote:
>
> > Will try  Simon.
> >
> > I expect no latency issues, since these 3 VMs are all on same
> > bridge/physical host. Other 2 Galera nodes are remote, but still low
> > latency - and zero seconds backend downtime...
> >
> > Not sure where to dig further...
> >
> > Thx
> >
> >
> >
> > On 8 June 2015 at 15:05, Simon Weller <sweller@ena.com> wrote:
> >
> > > Can you point the mgmt server directly at a single Galera node
> (removing
> > > haproxy), and see if the problem persists?
> > >
> > > Is your Galera cluster healthy? Could you be seeing latency on commit?
> > >
> > >
> > > ________________________________________
> > > From: Andrija Panic <andrija.panic@gmail.com>
> > > Sent: Monday, June 8, 2015 7:42 AM
> > > To: users@cloudstack.apache.org; dev@cloudstack.apache.org
> > > Subject: [HELP needed] haproxy vs mysql client timeout
> > >
> > > Hi,
> > >
> > > I'm running into intermittent problems with ACS 4.5.1 mgmt server - it
> > > seems it's loosing connection to the haproxy/mysql, since we are using
> > > haproxy VIP to in db.properties, and this haproxy is using 1 galera
> node
> > in
> > > the backend section...(2 others are backup etc).
> > >
> > >
> > > haproxy timeout is 100sec.
> > > mysql timeouts are all defaults, except innodb_lock_wait_timeout.
> > >
> > > +-----------------------------+----------+
> > > | Variable_name               | Value    |
> > > +-----------------------------+----------+
> > > | connect_timeout             | 10       |
> > > | delayed_insert_timeout      | 300      |
> > > | have_statement_timeout      | YES      |
> > > | innodb_flush_log_at_timeout | 1        |
> > > | innodb_lock_wait_timeout    | 600      |
> > > | innodb_rollback_on_timeout  | ON       |
> > > | interactive_timeout         | 28800    |
> > > | net_read_timeout            | 30       |
> > > | net_write_timeout           | 60       |
> > > | thread_pool_idle_timeout    | 60       |
> > > | wait_timeout                | 28800    |
> > > +-----------------------------+----------+
> > >
> > > To make things more interesting, all 3 servers (acs mgmt, haproxy,
> > > galera-node1) are on the same physical host, so I rule out network
> > > issues... and I dont see anything interesting in the haproxy logs, also
> > no
> > > backend downtimes etc...
> > >
> > >
> > > The errors are like folowing, from mgmt logs, but after that mgmt
> server
> > > continues to work fine...
> > >
> > > Any suggestions on tuning timeouts ?
> > >
> > >
> > > Caused by: com.mysql.jdbc.exceptions.jdbc4.CommunicationsException:
> > > Communications link failure
> > >
> > > The last packet successfully received from the server was 137,744
> > > milliseconds ago.  The last packet sent successfully to the server was
> 1
> > > milliseconds ago.
> > >         at
> > sun.reflect.GeneratedConstructorAccessor133.newInstance(Unknown
> > > Source)
> > >         at
> > >
> > >
> >
> sun.reflect.DelegatingConstructorAccessorImpl.newInstance(DelegatingConstructorAccessorImpl.java:45)
> > >         at
> > java.lang.reflect.Constructor.newInstance(Constructor.java:526)
> > >         at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
> > >         at
> > >
> com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:1129)
> > >         at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3720)
> > >         at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3609)
> > >         at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:4160)
> > >         at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2617)
> > >         at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2778)
> > >         at
> > com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2825)
> > >         at
> > >
> > >
> >
> com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.java:2156)
> > >         at
> > >
> >
> com.mysql.jdbc.PreparedStatement.executeQuery(PreparedStatement.java:2313)
> > >         at
> > >
> > >
> >
> org.apache.commons.dbcp.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:96)
> > >         at
> > >
> > >
> >
> org.apache.commons.dbcp.DelegatingPreparedStatement.executeQuery(DelegatingPreparedStatement.java:96)
> > >         at
> > > com.cloud.utils.db.GenericDaoBase.findById(GenericDaoBase.java:1009)
> > >         ... 61 more
> > > Caused by: java.io.EOFException: Can not read response from server.
> > > Expected to read 4 bytes, read 0 bytes before connection was
> unexpectedly
> > > lost.
> > >         ... 76 more
> > >
> > > Thanks,
> > >
> > > --
> > >
> > > Andrija Panić
> > >
> >
> >
> >
> > --
> >
> > Andrija Panić
> >
>



-- 

Andrija Panić

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message