cloudstack-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Rafael Weingärtner <rafaelweingart...@gmail.com>
Subject Re: MySQL HA
Date Mon, 18 Dec 2017 11:23:14 GMT
So, this fixed the problem?
Can you keep this running for a while longer? Just to make sure. Then, I
can open a PR to fix it in master.

On Mon, Dec 18, 2017 at 9:02 AM, Alireza Eskandari <astro.alireza@gmail.com>
wrote:

> Thank you Rafael,
> I test your fix and it seems that I have got the expected result. You
> can see the exception raised for database failover.
> I should notice I replace the file for cloudstack-mnagement and
> cloudstack-usage:
> /usr/share/cloudstack-usage/lib/cloud-framework-cluster-4.9.3.0.jar
> /usr/share/cloudstack-management/webapps/client/WEB-
> INF/lib/cloud-framework-cluster-4.9.3.0.jar
>
>
> Logs:
>
> WARN  [c.c.c.d.ManagementServerHostDaoImpl]
> (Cluster-Heartbeat-1:ctx-073cca55) (logid:e652d00b) Unexpected
> exception,
> com.cloud.utils.exception.CloudRuntimeException: Unable to commit or
> close the connection.
>         at com.cloud.utils.db.TransactionLegacy.commit(
> TransactionLegacy.java:740)
>         at com.cloud.cluster.dao.ManagementServerHostDaoImpl.update(
> ManagementServerHostDaoImpl.java:140)
>         at sun.reflect.GeneratedMethodAccessor103.invoke(Unknown Source)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(
> DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:606)
>         at org.springframework.aop.support.AopUtils.
> invokeJoinpointUsingReflection(AopUtils.java:317)
>         at org.springframework.aop.framework.ReflectiveMethodInvocation.
> invokeJoinpoint(ReflectiveMethodInvocation.java:183)
>         at org.springframework.aop.framework.ReflectiveMethodInvocation.
> proceed(ReflectiveMethodInvocation.java:150)
>         at com.cloud.utils.db.TransactionContextInterceptor.invoke(
> TransactionContextInterceptor.java:34)
>         at org.springframework.aop.framework.ReflectiveMethodInvocation.
> proceed(ReflectiveMethodInvocation.java:161)
>         at org.springframework.aop.interceptor.
> ExposeInvocationInterceptor.invoke(ExposeInvocationInterceptor.java:91)
>         at org.springframework.aop.framework.ReflectiveMethodInvocation.
> proceed(ReflectiveMethodInvocation.java:172)
>         at org.springframework.aop.framework.JdkDynamicAopProxy.
> invoke(JdkDynamicAopProxy.java:204)
>         at com.sun.proxy.$Proxy203.update(Unknown Source)
>         at com.cloud.cluster.ClusterManagerImpl$4.runInContext(
> ClusterManagerImpl.java:555)
>         at org.apache.cloudstack.managed.context.
> ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>         at org.apache.cloudstack.managed.context.impl.
> DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>         at org.apache.cloudstack.managed.context.impl.
> DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>         at org.apache.cloudstack.managed.context.impl.
> DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>         at org.apache.cloudstack.managed.context.
> ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>         at java.util.concurrent.Executors$RunnableAdapter.
> call(Executors.java:473)
>         at java.util.concurrent.FutureTask.runAndReset(
> FutureTask.java:304)
>         at java.util.concurrent.ScheduledThreadPoolExecutor$
> ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
>         at java.util.concurrent.ScheduledThreadPoolExecutor$
> ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1152)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:622)
>         at java.lang.Thread.run(Thread.java:748)
> Caused by: com.mysql.jdbc.exceptions.jdbc4.MySQLTransactionRollbackExcept
> ion:
> Deadlock found when trying to get lock; try restarting transaction
>         at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
>         at sun.reflect.NativeConstructorAccessorImpl.newInstance(
> NativeConstructorAccessorImpl.java:57)
>         at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(
> DelegatingConstructorAccessorImpl.java:45)
>         ... 46 more
> INFO  [o.a.c.f.j.i.AsyncJobManagerImpl]
> (AsyncJobMgr-Heartbeat-1:ctx-5ef0f4d1) (logid:4bfa48b2) Begin cleanup
> expired async-jobs
> INFO  [o.a.c.f.j.i.AsyncJobManagerImpl]
> (AsyncJobMgr-Heartbeat-1:ctx-5ef0f4d1) (logid:4bfa48b2) End cleanup
> expired async-jobs
> ERROR [c.c.u.d.ConnectionConcierge]
> (ConnectionConcierge-1:ctx-d3460aeb) (logid:b8c62262) Unable to keep
> the db connection for LockMaster1
> java.sql.SQLException: Connection was killed
>         at com.mysql.jdbc.SQLError.createSQLException(SQLError.java:1073)
>         at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3597)
>         at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3529)
>         at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1990)
>         at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2151)
>         at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2625)
>         at com.mysql.jdbc.LoadBalancedMySQLConnection.execSQL(
> LoadBalancedMySQLConnection.java:155)
>         at com.mysql.jdbc.PreparedStatement.executeInternal(
> PreparedStatement.java:2119)
>         at com.mysql.jdbc.PreparedStatement.executeQuery(
> PreparedStatement.java:2283)
>         at sun.reflect.GeneratedMethodAccessor75.invoke(Unknown Source)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(
> DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:606)
>         at com.mysql.jdbc.LoadBalancingConnectionProxy$
> ConnectionErrorFiringInvocationHandler.invoke(
> LoadBalancingConnectionProxy.java:103)
>         at com.mysql.jdbc.FailoverConnectionProxy$
> FailoverInvocationHandler.invoke(FailoverConnectionProxy.java:51)
>         at com.sun.proxy.$Proxy257.executeQuery(Unknown Source)
>         at org.apache.commons.dbcp.DelegatingPreparedStatement.
> executeQuery(DelegatingPreparedStatement.java:96)
>         at org.apache.commons.dbcp.DelegatingPreparedStatement.
> executeQuery(DelegatingPreparedStatement.java:96)
>         at com.cloud.utils.db.ConnectionConcierge$
> ConnectionConciergeManager.testValidity(ConnectionConcierge.java:148)
>         at com.cloud.utils.db.ConnectionConcierge$
> ConnectionConciergeManager$1.runInContext(ConnectionConcierge.java:203)
>         at org.apache.cloudstack.managed.context.
> ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>         at org.apache.cloudstack.managed.context.impl.
> DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>         at org.apache.cloudstack.managed.context.impl.
> DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>         at org.apache.cloudstack.managed.context.impl.
> DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>         at org.apache.cloudstack.managed.context.
> ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>         at java.util.concurrent.Executors$RunnableAdapter.
> call(Executors.java:473)
>         at java.util.concurrent.FutureTask.runAndReset(
> FutureTask.java:304)
>         at java.util.concurrent.ScheduledThreadPoolExecutor$
> ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
>         at java.util.concurrent.ScheduledThreadPoolExecutor$
> ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1152)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:622)
>         at java.lang.Thread.run(Thread.java:748)
> INFO  [o.a.c.f.j.i.AsyncJobManagerImpl]
> (AsyncJobMgr-Heartbeat-1:ctx-22399f19) (logid:0deff5fe) Begin cleanup
> expired async-jobs
> INFO  [o.a.c.f.j.i.AsyncJobManagerImpl]
> (AsyncJobMgr-Heartbeat-1:ctx-22399f19) (logid:0deff5fe) End cleanup
> expired async-jobs
> INFO  [o.a.c.f.j.i.AsyncJobManagerImpl]
> (AsyncJobMgr-Heartbeat-1:ctx-6004a86b) (logid:64e4a3b3) Begin cleanup
> expired async-jobs
> INFO  [o.a.c.f.j.i.AsyncJobManagerImpl]
> (AsyncJobMgr-Heartbeat-1:ctx-6004a86b) (logid:64e4a3b3) End cleanup
> expired async-jobs
> INFO  [c.c.h.v.m.HostMO] (DirectAgent-19:ctx-351cbbac
> host01.cloud.local, cmd: GetVmStatsCommand) (logid:56c8e250) VM
> i-2-124-VM not found in host cache
> INFO  [c.c.h.v.m.HostMO] (DirectAgent-1:ctx-9c0f5042
> host02.cloud.local, cmd: GetVmStatsCommand) (logid:56c8e250) VM
> i-2-125-VM not found in host cache
> ERROR [c.c.u.d.ConnectionConcierge]
> (ConnectionConcierge-1:ctx-544154d7) (logid:bd0f585a) Unable to keep
> the db connection for LockMaster1
> com.mysql.jdbc.exceptions.jdbc4.CommunicationsException:
> Communications link failure
>
> The last packet successfully received from the server was 19,479
> milliseconds ago.  The last packet sent successfully to the server was
> 19,479 milliseconds ago.
>         at sun.reflect.NativeConstructorAccessorImpl.newInstance0(Native
> Method)
>         at sun.reflect.NativeConstructorAccessorImpl.newInstance(
> NativeConstructorAccessorImpl.java:57)
>         at sun.reflect.DelegatingConstructorAccessorImpl.newInstance(
> DelegatingConstructorAccessorImpl.java:45)
>         at java.lang.reflect.Constructor.newInstance(Constructor.java:526)
>         at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
>         at com.mysql.jdbc.SQLError.createCommunicationsException(
> SQLError.java:1116)
>         at com.mysql.jdbc.MysqlIO.send(MysqlIO.java:3352)
>         at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:1971)
>         at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2151)
>         at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2625)
>         at com.mysql.jdbc.LoadBalancedMySQLConnection.execSQL(
> LoadBalancedMySQLConnection.java:155)
>         at com.mysql.jdbc.PreparedStatement.executeInternal(
> PreparedStatement.java:2119)
>         at com.mysql.jdbc.PreparedStatement.executeQuery(
> PreparedStatement.java:2283)
>         at sun.reflect.GeneratedMethodAccessor75.invoke(Unknown Source)
>         at sun.reflect.DelegatingMethodAccessorImpl.invoke(
> DelegatingMethodAccessorImpl.java:43)
>         at java.lang.reflect.Method.invoke(Method.java:606)
>         at com.mysql.jdbc.LoadBalancingConnectionProxy$
> ConnectionErrorFiringInvocationHandler.invoke(
> LoadBalancingConnectionProxy.java:103)
>         at com.mysql.jdbc.FailoverConnectionProxy$
> FailoverInvocationHandler.invoke(FailoverConnectionProxy.java:51)
>         at com.sun.proxy.$Proxy257.executeQuery(Unknown Source)
>         at org.apache.commons.dbcp.DelegatingPreparedStatement.
> executeQuery(DelegatingPreparedStatement.java:96)
>         at org.apache.commons.dbcp.DelegatingPreparedStatement.
> executeQuery(DelegatingPreparedStatement.java:96)
>         at com.cloud.utils.db.ConnectionConcierge$
> ConnectionConciergeManager.testValidity(ConnectionConcierge.java:148)
>         at com.cloud.utils.db.ConnectionConcierge$
> ConnectionConciergeManager$1.runInContext(ConnectionConcierge.java:203)
>         at org.apache.cloudstack.managed.context.
> ManagedContextRunnable$1.run(ManagedContextRunnable.java:49)
>         at org.apache.cloudstack.managed.context.impl.
> DefaultManagedContext$1.call(DefaultManagedContext.java:56)
>         at org.apache.cloudstack.managed.context.impl.
> DefaultManagedContext.callWithContext(DefaultManagedContext.java:103)
>         at org.apache.cloudstack.managed.context.impl.
> DefaultManagedContext.runWithContext(DefaultManagedContext.java:53)
>         at org.apache.cloudstack.managed.context.
> ManagedContextRunnable.run(ManagedContextRunnable.java:46)
>         at java.util.concurrent.Executors$RunnableAdapter.
> call(Executors.java:473)
>         at java.util.concurrent.FutureTask.runAndReset(
> FutureTask.java:304)
>         at java.util.concurrent.ScheduledThreadPoolExecutor$
> ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:178)
>         at java.util.concurrent.ScheduledThreadPoolExecutor$
> ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:293)
>         at java.util.concurrent.ThreadPoolExecutor.runWorker(
> ThreadPoolExecutor.java:1152)
>         at java.util.concurrent.ThreadPoolExecutor$Worker.run(
> ThreadPoolExecutor.java:622)
>         at java.lang.Thread.run(Thread.java:748)
> Caused by: java.net.SocketException: Broken pipe (Write failed)
>         at java.net.SocketOutputStream.socketWrite0(Native Method)
>         at java.net.SocketOutputStream.socketWrite(
> SocketOutputStream.java:115)
>         at java.net.SocketOutputStream.write(SocketOutputStream.java:161)
>         ... 36 more
>
> On Mon, Dec 18, 2017 at 2:03 PM, Rafael Weingärtner
> <rafaelweingartner@gmail.com> wrote:
> > Here is a fix:
> > https://www.dropbox.com/s/kgakhs3v05uz88x/cloud-
> framework-cluster-4.9.3.0.jar?dl=1
> > You need to replace this jar file in CloudStack installation. You should
> > also backup the original jar and restore it as soon as you finish
> testing.
> > To replace the JARs, you need to stop ACS, and just then start it.
> >
> > If everything works fine, I will open a PR against master, and with a bit
> > of luck we can push it into 4.11
> >
> > On Sat, Dec 16, 2017 at 8:03 AM, Alireza Eskandari <
> astro.alireza@gmail.com>
> > wrote:
> >
> >> I'm using CS 4.9.3.0-shapeblue0
> >>
> >> On Sat, Dec 16, 2017 at 12:49 PM, Rafael Weingärtner
> >> <rafaelweingartner@gmail.com> wrote:
> >> > Awesome!
> >> > I found one method that might seem the cause of the problem.
> >> > What is the version of ACS that you are using?
> >> >
> >> > On Sat, Dec 16, 2017 at 4:10 AM, Alireza Eskandari <
> >> astro.alireza@gmail.com>
> >> > wrote:
> >> >
> >> >> Hi
> >> >>
> >> >> Gabriel,
> >> >> My configuration is same as your suggestion, but I get the errors.
> >> >>
> >> >> Rafael,
> >> >> You are right. I confirm that CS works normally but I get those
> >> warnings.
> >> >> I would make me happy to help you for this fix :)
> >> >>
> >> >>
> >> >> On Tue, Dec 12, 2017 at 3:30 PM, Rafael Weingärtner
> >> >> <rafaelweingartner@gmail.com> wrote:
> >> >> > Alireza,
> >> >> > This is a warning and should not cause you much trouble. I have
> been
> >> >> trying
> >> >> > to pin point this problem for quite some time now.
> >> >> > If I generate a fix, would you be willing to test it?
> >> >> >
> >> >> > On Tue, Dec 12, 2017 at 8:56 AM, Gabriel Beims Bräscher <
> >> >> > gabrascher@gmail.com> wrote:
> >> >> >
> >> >> >> Hi Alireza,
> >> >> >>
> >> >> >> I have production environments with Master to Master replication
> and
> >> >> >> we have no problems. We may need more details of your
> configuration.
> >> >> >> Have you configured the slave database? Are you sure that
you
> >> configured
> >> >> >> correctly the ha heuristic?
> >> >> >>
> >> >> >> Considering that you already configured replication and "my.cnf",
> I
> >> will
> >> >> >> focus on the CloudSack db.properties file.
> >> >> >>
> >> >> >> When configuring Master-Master replication, you should have
at
> >> >> >> /etc/cloudstack/management/db.properties something like:
> >> >> >> -----------------------------
> >> >> >> db.cloud.autoReconnectForPools=true
> >> >> >>
> >> >> >> #High Availability And Cluster Properties
> >> >> >> db.ha.enabled=true
> >> >> >>
> >> >> >> db.cloud.queriesBeforeRetryMaster=5000
> >> >> >> db.usage.failOverReadOnly=false
> >> >> >> db.cloud.slaves=acs-db-02
> >> >> >>
> >> >> >> cluster.node.IP=<cluster node IP>
> >> >> >>
> >> >> >> db.usage.autoReconnect=true
> >> >> >>
> >> >> >> db.cloud.host=acs-db-01
> >> >> >> db.usage.host=acs-db-01
> >> >> >>
> >> >> >> #db.ha.loadBalanceStrategy=com.mysql.jdbc.
> SequentialBalanceStrategy
> >> >> >> db.ha.loadBalanceStrategy=com.cloud.utils.db.StaticStrategy
> >> >> >>
> >> >> >> db.cloud.failOverReadOnly=false
> >> >> >> db.usage.slaves=acs-db-02
> >> >> >> -----------------------------
> >> >> >>
> >> >> >> "db.ha.loadBalanceStrategy" is confiugured with the heuristic
> >> >> >> "com.cloud.utils.db.StaticStrategy"
> >> >> >>
> >> >> >> "db.ha.enabled" need to be “true”
> >> >> >>
> >> >> >> The primary database is configured with the variable
> “db.cloud.host”.
> >> >> The
> >> >> >> secondary database(s) is(are) configured with the variable
> >> >> >> “db.usage.slaves”. One variable that is different from
both Apache
> >> >> >> CloudStack servers is “cluster.node.IP”, being the ACS
mgt IP.
> >> >> >> Additionally, you will need to create a folder
> >> >> >> “/usr/share/cloudstack-mysql-ha/lib/” and move the jar
file
> >> >> >> “cloud-plugin-database-mysqlha-4.9.3.0.jar” into the new
folder.
> >> >> >>
> >> >> >> -----------------------------
> >> >> >> mkdir -p /usr/share/cloudstack-mysql-ha/lib/
> >> >> >> cp
> >> >> >> /usr/share/cloudstack-management/webapps/client/WEB-
> >> >> >> INF/lib/cloud-plugin-database-mysqlha-4.9.3.0.jar
> >> >> >> /usr/share/cloudstack-mysql-ha/lib/
> >> >> >> -----------------------------
> >> >> >>
> >> >> >> Cheers,
> >> >> >> Gabriel.
> >> >> >>
> >> >> >> 2017-12-12 6:30 GMT-02:00 Alireza Eskandari <
> astro.alireza@gmail.com
> >> >:
> >> >> >>
> >> >> >> > I have opened a new jira ticket about this problem:
> >> >> >> > https://issues.apache.org/jira/browse/CLOUDSTACK-10186
> >> >> >> >
> >> >> >>
> >> >> >
> >> >> >
> >> >> >
> >> >> > --
> >> >> > Rafael Weingärtner
> >> >>
> >> >
> >> >
> >> >
> >> > --
> >> > Rafael Weingärtner
> >>
> >
> >
> >
> > --
> > Rafael Weingärtner
>



-- 
Rafael Weingärtner

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message