Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id 604B1200B85 for ; Thu, 15 Sep 2016 16:25:07 +0200 (CEST) Received: by cust-asf.ponee.io (Postfix) id 5EC5A160AB5; Thu, 15 Sep 2016 14:25:07 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id A42D5160AB7 for ; Thu, 15 Sep 2016 16:25:06 +0200 (CEST) Received: (qmail 71911 invoked by uid 500); 15 Sep 2016 14:25:05 -0000 Mailing-List: contact users-help@activemq.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: users@activemq.apache.org Delivered-To: mailing list users@activemq.apache.org Received: (qmail 71867 invoked by uid 99); 15 Sep 2016 14:25:05 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd4-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 15 Sep 2016 14:25:05 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd4-us-west.apache.org (ASF Mail Server at spamd4-us-west.apache.org) with ESMTP id 079E3C05C3 for ; Thu, 15 Sep 2016 14:25:05 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd4-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 1.55 X-Spam-Level: * X-Spam-Status: No, score=1.55 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, FREEMAIL_ENVFROM_END_DIGIT=0.25, HEADER_FROM_DIFFERENT_DOMAINS=0.001, HTML_MESSAGE=2, RCVD_IN_DNSWL_LOW=-0.7, SPF_PASS=-0.001] autolearn=disabled Authentication-Results: spamd4-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx2-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd4-us-west.apache.org [10.40.0.11]) (amavisd-new, port 10024) with ESMTP id U5S11cLwX_dI for ; Thu, 15 Sep 2016 14:25:01 +0000 (UTC) Received: from mail-vk0-f41.google.com (mail-vk0-f41.google.com [209.85.213.41]) by mx2-lw-eu.apache.org (ASF Mail Server at mx2-lw-eu.apache.org) with ESMTPS id D0A025FE36 for ; Thu, 15 Sep 2016 14:25:00 +0000 (UTC) Received: by mail-vk0-f41.google.com with SMTP id b81so29141424vkd.0 for ; Thu, 15 Sep 2016 07:25:00 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:sender:in-reply-to:references:from:date:message-id :subject:to; bh=SbHSNLUsc7GowCo8lks5Nagt+ML07m9F8FTT3iJ6yt4=; b=LZ/TcU3egEYnufOac2HEz/zv8ndV2Z30WuYp+k0sVJd5IPxKtP4zmfUIHc5tGJGaxb CLtVOnf7en9AOvi2gaMxCaeAESVJ9VWe+zwn1h9/1ruUBruHfPx0+7tFR8xaBHtTwIPL NpyZRfpAog6jjPKlmcoEM00tEzHXJdxuEEBPdWYP9ur0+UDJznRFTJf5rWqGyHIS79gF 6pnw6pFgZtfDa2QigvoXMe8SNpsXWzUjhwa3gSSQQ3WDArw9cMMSYUffI35/sUxB9FW/ IQg5nTm+eJeSu97gVKtWC4j/mi9TJHa8LoFIHcTrYSNxJypTwNzt6JTtUrZicmbi+9t/ Nqiw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:sender:in-reply-to:references:from :date:message-id:subject:to; bh=SbHSNLUsc7GowCo8lks5Nagt+ML07m9F8FTT3iJ6yt4=; b=lQoOM+AUurLOZwlT7KMqsGkhCFzll+hVmVMY/U4AW5U696W6dqwhfTth3maRn8vFVV FkadHuH+IkkNC1cV/UM6ez1N16hlylxA0H62VKicy9uTma00yPYedrxWrWVjcQEJJF0f X1O4NvVkuX5q22e+DGQWvSra80iZE3+6rEa7aloc1ScT6EFHrTEzrzdxb/GaJ3OsVcBW DR3yL+ZBYmRUecMmPf+Jw76XP5HaDM2Ue8mKt3fLbIELd3jGuckSGCEKHjeJmvzCb3vx G7AC+vPDlLulY6x0UPhOF/YB8wVub0vcAc4OgF6VzQ1+5vzjkpK3cT8mQy/e63uW7Wcu VwEw== X-Gm-Message-State: AE9vXwPk8WBPbBqYxb90tsN5w2KYqy2KzmIp5EOiJh6qapotjHqJLjp3wq03OsJJosyu+tTBPNUyNs/PxSgexQ== X-Received: by 10.31.167.149 with SMTP id q143mr3029687vke.57.1473949487640; Thu, 15 Sep 2016 07:24:47 -0700 (PDT) MIME-Version: 1.0 Sender: tbain98@gmail.com Received: by 10.103.110.65 with HTTP; Thu, 15 Sep 2016 07:24:46 -0700 (PDT) Received: by 10.103.110.65 with HTTP; Thu, 15 Sep 2016 07:24:46 -0700 (PDT) In-Reply-To: <7e190268-f0f3-c2a9-f54a-957abd5a7ee2@gmail.com> References: <7e190268-f0f3-c2a9-f54a-957abd5a7ee2@gmail.com> From: Tim Bain Date: Thu, 15 Sep 2016 08:24:46 -0600 X-Google-Sender-Auth: YDPPzmy2-A6vkd0PMlwm0qdWS7o Message-ID: Subject: Re: Problem With Network of Brokers 5.14.0 To: ActiveMQ Users Content-Type: multipart/alternative; boundary=001a1142608ca0b86e053c8c9b6d archived-at: Thu, 15 Sep 2016 14:25:07 -0000 --001a1142608ca0b86e053c8c9b6d Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable If that's true, the decreaseNetworkConsumerPriority property (see http://activemq.apache.org/networks-of-brokers.html for details) would prevent the problem. Though it could result in less even load balancing across consumers connected to different brokers; it depends on how the producers and consumers are distributed. On Sep 15, 2016 8:04 AM, "Matt Pavlovich" wrote: > How are your consumers setup? This looks like a mesh config.. I'm > wondering if the max message hop ttl was reached and the messages are > parked in that one broker. That scenarios is helped by splitting the > consumerTTL and the messageTTL and not using networkTTL. > > Do you have a specific reason for not setting conduitSubscriptions=3Dtrue= ? > > uri=3D"static:(tcp://c01-meq1:61616)" > networkTTL=3D"3" > conduitSubscriptions=3D"false" /> > uri=3D"static:(tcp://c01-meq2:61616)" > networkTTL=3D"3" > conduitSubscriptions=3D"false" /> > > > On 9/14/16 10:01 PM, Geoffrey Mina wrote: > >> Greetings, >> I am pretty new to ActiveMQ as we just deployed into our stack. >> >> We had a network of brokers have a significant problem tonight due to >> (what appeared to be) a single broker. 2 of the 3 were processing messa= ges >> OK and the third was queuing up and not processing messages quickly. Th= is >> is for a real-time application and seconds (even milliseconds) count. >> >> >> There was absolutely nothing in the log file that was interesting. I >> even restarted the =E2=80=9Cbad=E2=80=9D broker and when it came back on= line it was >> behaving identically. We have persistence completely disabled and there= is >> not even a kahadb on the file system. >> >> >> I have attached my config here for one of the brokers (all configs >> identical except for the the network config). If anyone sees anything >> glaringly obvious, please let me know. We just deployed this cluster of >> servers last Friday. It has processed hundreds of millions of messages = in >> the last few days =E2=80=93 before it began to misbehave tonight. >> >> >> ActiveMQ 5.14.0 >> >> >> java version "1.8.0_102" >> >> Java(TM) SE Runtime Environment (build 1.8.0_102-b14) >> >> Java HotSpot(TM) 64-Bit Server VM (build 25.102-b14, mixed mode) >> >> >> Amazon EC2 Host >> >> CentOS 7 >> >> 3.10.0-327.10.1.el7.x86_64 >> >> 4 x Intel(R) Xeon(R) CPU E5-2666 v3 @ 2.90GHz >> >> 8G RAM >> >> >> >> Thanks in advance! >> >> Geoff >> >> > --001a1142608ca0b86e053c8c9b6d--