Mailing-List: contact users-help@qpid.apache.org; run by ezmlm
Precedence: bulk
Reply-To: users@qpid.apache.org
From: Adel Boutros <Adelboutros@live.com>
To: "users@qpid.apache.org" <users@qpid.apache.org>
Subject: Re: Testing failover on dispatcher/java-broker cluster
Thread-Topic: Testing failover on dispatcher/java-broker cluster
Thread-Index: AQHSFkaEPSCX1rdMpkuzK4vGdmvBj6CQkOaAgAAA0wCAAAF2LQ==
Date: Thu, 29 Sep 2016 14:47:47 +0000
Message-ID: <VI1PR0901MB089397866AA9B0512E0194EECDCE0@VI1PR0901MB0893.eurprd09.prod.outlook.com>
References: <CA+g_sc1KuRFmPYQCn8jeokmXeNpm0JhC5u_QY3Q+Yd5khbTK_g@mail.gmail.com>
 <bee4cc71-2131-00e5-eaf7-9a2b05bbe33a@redhat.com>
 <DUB131-W69AB0E7CB519D94998BD05CDF40@phx.gbl>
 <534e0042-81da-cf2f-eeb5-8adfac34d64a@redhat.com>
 <DUB131-W75349D5AD9EE33814BE818CDF70@phx.gbl>
 <DUB131-W86FB92D5CADE52AFE2A332CDC80@phx.gbl>
 <8a4e6473-749b-08d2-7b6a-e415523487f0@redhat.com>
 <DUB131-W8E5D80B65000D19B380C6CDCB0@phx.gbl>
 <a3db9de5-0402-ae10-0585-beaec2b507a2@redhat.com>,<eacf2169-27d6-2d45-4083-31a09176ee80@redhat.com>
In-Reply-To: <eacf2169-27d6-2d45-4083-31a09176ee80@redhat.com>
Accept-Language: en-US
Content-Language: en-US
received-spf: SoftFail (protection.outlook.com: domain of transitioning
 live.com discourages use of 10.152.16.55 as permitted sender)
spamdiagnosticoutput: 1:99
spamdiagnosticmetadata: NSPM
Content-Type: multipart/alternative;
	boundary="_000_VI1PR0901MB089397866AA9B0512E0194EECDCE0VI1PR0901MB0893_"
MIME-Version: 1.0
X-MS-Exchange-CrossTenant-originalarrivaltime: 29 Sep 2016 14:47:47.4811
 (UTC)
X-MS-Exchange-CrossTenant-fromentityheader: Internet
X-MS-Exchange-CrossTenant-id: 84df9e7f-e9f6-40af-b435-aaaaaaaaaaaa
X-MS-Exchange-Transport-CrossTenantHeadersStamped: AM5EUR03HT247
X-OriginalArrivalTime: 29 Sep 2016 14:47:59.0480 (UTC) FILETIME=[78E7B780:01D21A60]
archived-at: Thu, 29 Sep 2016 14:48:14 -0000

--_000_VI1PR0901MB089397866AA9B0512E0194EECDCE0VI1PR0901MB0893_
Content-Type: text/plain; charset="Windows-1252"
Content-Transfer-Encoding: quoted-printable

They seem fair enough and quite related.


As a side note, I have a bug with the dispatch router 0.6.1 but I haven't s=
ubmitted it yet because I haven't reduced the test case yet.

In resume, when I connect 2 dispatchers (inter-router) and then delete the =
connector/listener of "inter-router". If I delete and recreate a mobile add=
ress which has received a message on one of the dispatchers, the stats of t=
he "in" and "out" do not reset to 0 when doing "qdstat -a" but they remain =
at the old values. However they reset correctly on the other router.


Have you encountered something similar? Once I have a reduced test case, I =
will post it in a different thread of course.


Regards,

Adel

________________________________
From: Ted Ross <tross@redhat.com>
Sent: Thursday, September 29, 2016 4:38:26 PM
To: users@qpid.apache.org
Subject: Re: Testing failover on dispatcher/java-broker cluster

Sorry, those Jira numbers and descriptions are mismatched.  Here's the
correct list:

    - DISPATCH-496 - Activation of an autolink does not result in issuing
                     credit to a blocked sender
    - DISPATCH-505 - Eventual loss of credit on inter-router control
                     links when the topology changes
    - DISPATCH-523 - Topology changes can cause in-flight deliveries to
                     be stuck in the ingress router


On 09/29/2016 10:35 AM, Ted Ross wrote:
>
> On 09/24/2016 05:32 AM, Adel Boutros wrote:
>> We are indeed in favor of a minor release as long as the latest
>> version is still 0.6.x and we are willing to re-launch our tests and
>> give feedback on the release candidate once provided (It shouldn't
>> take us more than a day to compile and test).
>> Do you have a list of fixes in mind?
>
> I've identified three fixes that look like good candidates for 0.6.2:
>
>   - DISPATCH-496 - Topology changes can cause in-flight deliveries to
>                    be stuck in the ingress router
>   - DISPATCH-505 - Eventual loss of credit on inter-router control
>                    links when the topology changes
>   - DISPATCH-523 - Activation of an autolink does not result in issuing
>                    credit to a blocked sender
>
> These are all stability-related issues.
>
> Thoughts?
>
> -Ted
>
>> Regards,Adel
>>
>>> Subject: Re: Testing failover on dispatcher/java-broker cluster
>>> To: users@qpid.apache.org
>>> From: tross@redhat.com
>>> Date: Fri, 23 Sep 2016 17:23:57 -0400
>>>
>>> Hi Adel,
>>>
>>> A minor release is always possible.  It's up to us, the community, to
>>> decide whether and when to produce one.  I'm in favor of releasing an
>>> 0.6.2 with some small backports to fix bugs for users that want to stay
>>> on Proton 0.12.
>>>
>>> -Ted
>>>
>>> On 09/23/2016 09:44 AM, Adel Boutros wrote:
>>>> Hello Ted,
>>>> Did you happen to have the time to check if a minor release is
>>>> possible?
>>>> Regards,Adel
>>>>
>>>>> From: adelboutros@live.com
>>>>> To: users@qpid.apache.org
>>>>> Subject: RE: Testing failover on dispatcher/java-broker cluster
>>>>> Date: Tue, 20 Sep 2016 15:13:03 +0200
>>>>>
>>>>> Hello Ted,
>>>>>
>>>>> I confirm the fix solved the issue.
>>>>>
>>>>> Would it be possible to do a 0.6.2 release? We cannot compile newer
>>>>> versions of Proton (We currently use 0.12.2) due to lack of
>>>>> resources from our side and we really need this fix for our tests.
>>>>>
>>>>> Regards,
>>>>> Adel
>>>>>
>>>>>> Subject: Re: Testing failover on dispatcher/java-broker cluster
>>>>>> To: users@qpid.apache.org
>>>>>> From: tross@redhat.com
>>>>>> Date: Mon, 19 Sep 2016 12:18:23 -0400
>>>>>>
>>>>>> Hi Adel,
>>>>>>
>>>>>> It's a one-liner and it applies cleanly to the 0.6.x branch.
>>>>>>
>>>>>> https://git-wip-us.apache.org/repos/asf?p=3Dqpid-dispatch.git;h=3D41=
b7407
>>>>>>
>>>>>> -Ted
>>>>>>
>>>>>>
>>>>>> On 09/19/2016 11:41 AM, Adel Boutros wrote:
>>>>>>> Hello Ted,
>>>>>>>
>>>>>>> Antoine is on vacation so I will be taking over this task.
>>>>>>>
>>>>>>> Does this fix have any dependencies? We would like to apply it on
>>>>>>> 0.6.1 without other fixes because it seems the master branch
>>>>>>> requires proton 0.13.0 minimum whereas we have currently 0.12.2
>>>>>>> and we cannot upgrade at the time being.
>>>>>>>
>>>>>>> Regards,
>>>>>>> Adel
>>>>>>>
>>>>>>>> Subject: Re: Testing failover on dispatcher/java-broker cluster
>>>>>>>> To: users@qpid.apache.org
>>>>>>>> From: tross@redhat.com
>>>>>>>> Date: Fri, 16 Sep 2016 16:53:05 -0400
>>>>>>>>
>>>>>>>> Antoine,
>>>>>>>>
>>>>>>>> I think I know what that problem is.  I belileve you've stumbled
>>>>>>>> upon
>>>>>>>> this issue:
>>>>>>>>
>>>>>>>> https://issues.apache.org/jira/browse/DISPATCH-496
>>>>>>>>
>>>>>>>> Your second delivery, the one resulting in a timeout, is causing
>>>>>>>> the
>>>>>>>> inbound link to be blocked (i.e. it has undelivered messages).
>>>>>>>> When the
>>>>>>>> broker reattaches, the blocked links are supposed to become
>>>>>>>> unblocked
>>>>>>>> but they don't in the case of auto-links.
>>>>>>>>
>>>>>>>> This has been fixed on the master branch if you'd like to try
>>>>>>>> applying
>>>>>>>> the patch.
>>>>>>>>
>>>>>>>> -Ted
>>>>>>>>
>>>>>>>> On 09/15/2016 04:56 AM, Antoine Chevin wrote:
>>>>>>>>> Hi Ted,
>>>>>>>>>
>>>>>>>>> You=92re right, the connection close looked strange before
>>>>>>>>> stopping of the
>>>>>>>>> broker. I manually added the annotation (# stopping the broker)
>>>>>>>>> and was
>>>>>>>>> wrong about the position of this one. I replayed the test and the
>>>>>>>>> connection close happens *after* the broker stop. I assume it
>>>>>>>>> is the broker
>>>>>>>>> that initiates it.
>>>>>>>>>
>>>>>>>>> I found something interesting. In my test, I always sent a
>>>>>>>>> message when the
>>>>>>>>> broker is down, expecting to get a JmsSendTimedOutException
>>>>>>>>> (waiting for
>>>>>>>>> the disposition frame). I assumed this was harmless. But it
>>>>>>>>> turns out this
>>>>>>>>> is not. When I don=92t do that, I can send a message after the
>>>>>>>>> broker
>>>>>>>>> restart. So to sum up the experiment I did:
>>>>>>>>>
>>>>>>>>> * I use Wireshark between the JMS client and the dispatcher. *
>>>>>>>>>
>>>>>>>>> 1)      Using JMS I establish a connection to the dispatcher
>>>>>>>>> and create a
>>>>>>>>> message producer (Wireshark: connection open -> attach)
>>>>>>>>> 2)      I=92m able to send a message to the broker through the
>>>>>>>>> dispatcher (
>>>>>>>>> Wireshark: transfer -> disposition)
>>>>>>>>> 3)      I stop the broker
>>>>>>>>> 4)      With the same link, I send a message and I get a
>>>>>>>>> JmsSendTimedOutException (waiting for the disposition frame)
>>>>>>>>> (Wireshark:
>>>>>>>>> transfer)
>>>>>>>>> 5)      I restart the broker
>>>>>>>>> 6)      With the same link, I try to send a message and I get a
>>>>>>>>> JmsSendTimedOutException for the same reason (waiting for the
>>>>>>>>> disposition
>>>>>>>>> frame) (Wireshark: transfer)
>>>>>>>>>
>>>>>>>>> If I skip step (4), I cannot reproduce step (6) and my messages
>>>>>>>>> arrive
>>>>>>>>> (Wireshark: transfer -> disposition) to the restarted broker.
>>>>>>>>>
>>>>>>>>> I hope it makes it clearer for you. Sorry for my rookie
>>>>>>>>> mistakes :-).
>>>>>>>>>
>>>>>>>>> Note: My colleague and I ran a small experiment to identify if
>>>>>>>>> the problem
>>>>>>>>> comes from JMS or the AMQP protocol. He changed the code of the
>>>>>>>>> java broker
>>>>>>>>> to not send the disposition frame one time out of two.
>>>>>>>>>
>>>>>>>>> We got these results:
>>>>>>>>>
>>>>>>>>> * I use Wireshark between the JMS client and the patched broker. =
*
>>>>>>>>>
>>>>>>>>> 1) Using JMS I establish a connection to the patched broker and
>>>>>>>>> create a
>>>>>>>>> message producer (Wireshark: connection open -> attach)
>>>>>>>>> 2)  I send a message to the broker and it replies with the
>>>>>>>>> disposition
>>>>>>>>> frame (Wireshark: transfer -> disposition)
>>>>>>>>> 3) I send a message to the broker which drops the disposition
>>>>>>>>> frame. I get
>>>>>>>>> a send timeout in JMS (Wireshark: transfer)
>>>>>>>>> 2)  I send a message to the broker and it replies with the
>>>>>>>>> disposition frame
>>>>>>>>> (Wireshark: transfer -> disposition). It works fine.
>>>>>>>>>
>>>>>>>>> We assume that there is something going on in the dispatcher.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> Thanks,
>>>>>>>>> Antoine
>>>>>>>>>
>>>>>>>>
>>>>>>>> ------------------------------------------------------------------=
---
>>>>>>>>
>>>>>>>> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
>>>>>>>> For additional commands, e-mail: users-help@qpid.apache.org
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>> --------------------------------------------------------------------=
-
>>>>>> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
>>>>>> For additional commands, e-mail: users-help@qpid.apache.org
>>>>>>
>>>>>
>>>>
>>>>
>>>
>>> ---------------------------------------------------------------------
>>> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
>>> For additional commands, e-mail: users-help@qpid.apache.org
>>>
>>
>>
>
> ---------------------------------------------------------------------
> To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
> For additional commands, e-mail: users-help@qpid.apache.org
>

---------------------------------------------------------------------
To unsubscribe, e-mail: users-unsubscribe@qpid.apache.org
For additional commands, e-mail: users-help@qpid.apache.org


--_000_VI1PR0901MB089397866AA9B0512E0194EECDCE0VI1PR0901MB0893_--