cloudstack-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Kelven Yang <kelven.y...@citrix.com>
Subject Re: database connection resilience
Date Thu, 11 Jul 2013 00:14:34 GMT


On 7/8/13 1:40 PM, "Marcus Sorensen" <shadowsor@gmail.com> wrote:

>So to the original question, is it your opinion that a single
>management server (non-clustered) should also fence itself, or wait
>for the database connection to be restored?

Yes, my opinion is that a single management server should also fence
itself (if you like, use a monitoring script to bring it up automatically)

If connection failure event has been revealed to CloudStack via runtime
exception, then it is too late as current CloudStack code may not be able
to gracefully handle these out-of-flow-context unexpected exceptions in
every scenario (at least we haven't really tested it). We need the DB
connection layer to cover the multi-master failover case and make it
transparent.

Kelven 



>
>On Mon, Jul 8, 2013 at 2:03 PM, Kelven Yang <kelven.yang@citrix.com>
>wrote:
>>
>>
>> On 7/7/13 3:36 PM, "Marcus Sorensen" <shadowsor@gmail.com> wrote:
>>
>>>I think there are two separate issues here.
>>>
>>>1) The management server uses the database to determine cluster
>>>membership, and if no database connection can be made, the management
>>>server fences itself (shuts down). This is good, but in the case where
>>>there's only one management server (no cluster intended), it seems
>>>like an issue. However, it may be better to shut down, I'm not sure
>>>how the management server will react after a temporary database
>>>outage. Some opinions would be appreciated, my preference would be
>>>that a single-management server would just be able to pick back up
>>>where it left off rather than dying.
>>
>> In a management server cluster setup with multiple management servers,
>>to
>> avoid split-brian situation we will actively perform management server
>> self-fence as soon as the detection of inconsistent view of the cluster
>> from individual management servers.
>>
>> As the clustering logic relies on DB heavily, lost of DB connectivity is
>> considered as a fatal event to trigger self-fence in addition to the
>> inconsistent view detection. For a multi-master DB setup, it only works
>>if
>> the switch of database instance is transparent to CloudStack. Means,
>> database automatic fail-over should be completely handled at DB
>> connectivity layer and CloudStack should not be aware of it. Most of
>> current CloudStack logic is built upon such assumption, it may be
>>possible
>> to relax this requirement, but we need to investigate the impact and
>>test
>> out how resilient CloudStack would be to unexpected DB connectivity
>> exceptions in the middle of various orchestration work flows
>>
>>>
>>>2) There is no support for JDBC's built-in loadbalancing features. I
>>>have a patch that fixes this, however I noticed a few things that I'd
>>>like some feedback on. Namely, the awsapi database connection doesn't
>>>have its own settings, rather it uses the same host connection
>>>settings as the cloud db and the autoReconnect setting from the usage
>>>database settings. Was this a shortcut, or is there a reason for it?
>>>My current version of the patch just keeps the same methodology, but
>>>it seems that while I'm at adding properties to db.properties we could
>>>allow true db.awsapi.host and db.awsapi.port.
>>>
>>>On Sun, Jul 7, 2013 at 1:02 AM, Marcus Sorensen <shadowsor@gmail.com>
>>>wrote:
>>>> Oh, and I should correct myself, it doesn't crash, it seems that the
>>>> management server fences itself because it can't talk to the database.
>>>>
>>>> On Sun, Jul 7, 2013 at 12:59 AM, Marcus Sorensen <shadowsor@gmail.com>
>>>>wrote:
>>>>> Ok. After a cursory look, I've seen that the autoReconnect is kind of
>>>>> a bad option for jdbc. I've also found this, which seems kind of
>>>>>hairy
>>>>> for what I want to do:
>>>>>
>>>>>
>>>>>http://dev.mysql.com/doc/refman/5.0/en/connector-j-usagenotes-j2ee-con
>>>>>ce
>>>>>pts-managing-load-balanced-connections.html
>>>>>
>>>>> I don't necessarily want to hand off the loadbalancing management to
>>>>> the java code, I just want cloudstack to automatically reinitialize
>>>>> the database connection when this 'communications link failure'
>>>>> occurs, maybe with a db.cloud.connection.retry.count property or
>>>>> similar.
>>>>>
>>>>> On Sun, Jul 7, 2013 at 12:54 AM, Wido den Hollander <wido@widodh.nl>
>>>>>wrote:
>>>>>> Hi,
>>>>>>
>>>>>>
>>>>>> On 07/07/2013 08:45 AM, Marcus Sorensen wrote:
>>>>>>>
>>>>>>> I see that my db.properties has db.cloud.autoReconnect=true,
which
>>>>>>> translates to setting autoReconnect in the jdbc driver connection
>>>>>>>in
>>>>>>> utils/src/com/cloud/utils/db/Transaction.java. I also see that
if I
>>>>>>> manually trigger the issue I get:
>>>>>>>
>>>>>>
>>>>>> Just to confirm, I see the same issues. I haven't looked into this
>>>>>>yet, but
>>>>>> this is also one of the things I want to have fixed.
>>>>>>
>>>>>> Maybe create an issue for it?
>>>>>>
>>>>>> Wido
>>>>>>
>>>>>>
>>>>>>> 013-07-07 00:42:50,502 ERROR [cloud.cluster.ClusterManagerImpl]
>>>>>>> (Cluster-Heartbeat-1:null) Runtime DB exception
>>>>>>> com.mysql.jdbc.exceptions.jdbc4.CommunicationsException:
>>>>>>> Communications link failure
>>>>>>>
>>>>>>> The last packet successfully received from the server was 1,503
>>>>>>> milliseconds ago.  The last packet sent successfully to the server
>>>>>>>was
>>>>>>> 0 milliseconds ago.
>>>>>>> at sun.reflect.GeneratedConstructorAccessor159.newInstance(Unknown
>>>>>>>Source)
>>>>>>> at
>>>>>>>
>>>>>>>sun.reflect.DelegatingConstructorAccessorImpl.newInstance(Delegating
>>>>>>>Co
>>>>>>>nstructorAccessorImpl.java:45)
>>>>>>> at java.lang.reflect.Constructor.newInstance(Constructor.java:532)
>>>>>>> at com.mysql.jdbc.Util.handleNewInstance(Util.java:411)
>>>>>>> at
>>>>>>>
>>>>>>>com.mysql.jdbc.SQLError.createCommunicationsException(SQLError.java:
>>>>>>>11
>>>>>>>17)
>>>>>>> at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3567)
>>>>>>> at com.mysql.jdbc.MysqlIO.reuseAndReadPacket(MysqlIO.java:3456)
>>>>>>> at com.mysql.jdbc.MysqlIO.checkErrorPacket(MysqlIO.java:3997)
>>>>>>> at com.mysql.jdbc.MysqlIO.sendCommand(MysqlIO.java:2468)
>>>>>>> at com.mysql.jdbc.MysqlIO.sqlQueryDirect(MysqlIO.java:2629)
>>>>>>> at com.mysql.jdbc.ConnectionImpl.execSQL(ConnectionImpl.java:2719)
>>>>>>> at
>>>>>>>
>>>>>>>com.mysql.jdbc.PreparedStatement.executeInternal(PreparedStatement.j
>>>>>>>av
>>>>>>>a:2155)
>>>>>>> at
>>>>>>>
>>>>>>>com.mysql.jdbc.PreparedStatement.executeQuery(PreparedStatement.java
>>>>>>>:2
>>>>>>>318)
>>>>>>> at
>>>>>>>
>>>>>>>org.apache.commons.dbcp.DelegatingPreparedStatement.executeQuery(Del
>>>>>>>eg
>>>>>>>atingPreparedStatement.java:96)
>>>>>>> at
>>>>>>>
>>>>>>>org.apache.commons.dbcp.DelegatingPreparedStatement.executeQuery(Del
>>>>>>>eg
>>>>>>>atingPreparedStatement.java:96)
>>>>>>> at
>>>>>>>
>>>>>>>com.cloud.utils.db.GenericDaoBase.searchIncludingRemoved(GenericDaoB
>>>>>>>as
>>>>>>>e.java:409)
>>>>>>> at
>>>>>>>
>>>>>>>com.cloud.utils.component.ComponentInstantiationPostProcessor$Interc
>>>>>>>ep
>>>>>>>torDispatcher.intercept(ComponentInstantiationPostProcessor.java:125
>>>>>>>)
>>>>>>> at
>>>>>>>
>>>>>>>com.cloud.utils.db.GenericDaoBase.searchIncludingRemoved(GenericDaoB
>>>>>>>as
>>>>>>>e.java:350)
>>>>>>> at
>>>>>>>
>>>>>>>com.cloud.utils.component.ComponentInstantiationPostProcessor$Interc
>>>>>>>ep
>>>>>>>torDispatcher.intercept(ComponentInstantiationPostProcessor.java:125
>>>>>>>)
>>>>>>> at
>>>>>>>
>>>>>>>com.cloud.utils.db.GenericDaoBase.listIncludingRemovedBy(GenericDaoB
>>>>>>>as
>>>>>>>e.java:907)
>>>>>>> at
>>>>>>>
>>>>>>>com.cloud.utils.component.ComponentInstantiationPostProcessor$Interc
>>>>>>>ep
>>>>>>>torDispatcher.intercept(ComponentInstantiationPostProcessor.java:125
>>>>>>>)
>>>>>>> at
>>>>>>>
>>>>>>>com.cloud.utils.db.GenericDaoBase.listIncludingRemovedBy(GenericDaoB
>>>>>>>as
>>>>>>>e.java:912)
>>>>>>> at
>>>>>>>
>>>>>>>com.cloud.utils.component.ComponentInstantiationPostProcessor$Interc
>>>>>>>ep
>>>>>>>torDispatcher.intercept(ComponentInstantiationPostProcessor.java:125
>>>>>>>)
>>>>>>> at
>>>>>>>
>>>>>>>com.cloud.cluster.dao.ManagementServerHostDaoImpl.getActiveList(Mana
>>>>>>>ge
>>>>>>>mentServerHostDaoImpl.java:158)
>>>>>>> at
>>>>>>>
>>>>>>>com.cloud.utils.component.ComponentInstantiationPostProcessor$Interc
>>>>>>>ep
>>>>>>>torDispatcher.intercept(ComponentInstantiationPostProcessor.java:125
>>>>>>>)
>>>>>>> at
>>>>>>>
>>>>>>>com.cloud.cluster.ClusterManagerImpl.peerScan(ClusterManagerImpl.jav
>>>>>>>a:
>>>>>>>1057)
>>>>>>> at
>>>>>>>
>>>>>>>com.cloud.cluster.ClusterManagerImpl.access$1200(ClusterManagerImpl.
>>>>>>>ja
>>>>>>>va:95)
>>>>>>> at
>>>>>>>com.cloud.cluster.ClusterManagerImpl$4.run(ClusterManagerImpl.java:7
>>>>>>>89
>>>>>>>)
>>>>>>> at
>>>>>>>java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:4
>>>>>>>71
>>>>>>>)
>>>>>>> at
>>>>>>>
>>>>>>>java.util.concurrent.FutureTask$Sync.innerRunAndReset(FutureTask.jav
>>>>>>>a:
>>>>>>>351)
>>>>>>> at java.util.concurrent.FutureTask.runAndReset(FutureTask.java:178)
>>>>>>> at
>>>>>>>
>>>>>>>java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask
>>>>>>>.a
>>>>>>>ccess$201(ScheduledThreadPoolExecutor.java:165)
>>>>>>> at
>>>>>>>
>>>>>>>java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask
>>>>>>>.r
>>>>>>>un(ScheduledThreadPoolExecutor.java:267)
>>>>>>> at
>>>>>>>
>>>>>>>java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor
>>>>>>>.j
>>>>>>>ava:1146)
>>>>>>> at
>>>>>>>
>>>>>>>java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecuto
>>>>>>>r.
>>>>>>>java:615)
>>>>>>> at java.lang.Thread.run(Thread.java:679)
>>>>>>> Caused by: java.io.EOFException: Can not read response from server.
>>>>>>> Expected to read 4 bytes, read 0 bytes before connection was
>>>>>>> unexpectedly lost.
>>>>>>> ... 55 more
>>>>>>> 2013-07-07 00:42:50,505 ERROR [cloud.cluster.ClusterManagerImpl]
>>>>>>> (Cluster-Heartbeat-1:null) DB communication problem detected,
fence
>>>>>>>it
>>>>>>>
>>>>>>> And I have only to restart cloudstack-management so it can connect
>>>>>>>to
>>>>>>> another member in the loadbalanced multimaster database to get
>>>>>>>things
>>>>>>> running again.
>>>>>>>
>>>>>>>
>>>>>>> On Sun, Jul 7, 2013 at 12:35 AM, Marcus Sorensen
>>>>>>><shadowsor@gmail.com>
>>>>>>> wrote:
>>>>>>>>
>>>>>>>> I've noticed that the cloudstack management server creates
>>>>>>>>persistent
>>>>>>>> connections to the database, and crashes if the database
>>>>>>>>connection
>>>>>>>>is
>>>>>>>> lost. I haven't looked at the code yet, but I was wondering
if
>>>>>>>>anyone
>>>>>>>> knew about what was going on here, if it's simply not set
up to
>>>>>>>> gracefully handle reconnect, or something else.  We have
a
>>>>>>>> multi-master database setup, but cloudstack doesn't take
advantage
>>>>>>>>of
>>>>>>>> it since it doesn't attempt graceful reconnect, if the particular
>>>>>>>>node
>>>>>>>> it connected to on startup goes down, it simply crashes.
>>


Mime
View raw message