Return-Path: X-Original-To: archive-asf-public-internal@cust-asf2.ponee.io Delivered-To: archive-asf-public-internal@cust-asf2.ponee.io Received: from cust-asf.ponee.io (cust-asf.ponee.io [163.172.22.183]) by cust-asf2.ponee.io (Postfix) with ESMTP id CE6D8200D44 for ; Mon, 20 Nov 2017 14:45:34 +0100 (CET) Received: by cust-asf.ponee.io (Postfix) id CAEBC160BF9; Mon, 20 Nov 2017 13:45:34 +0000 (UTC) Delivered-To: archive-asf-public@cust-asf.ponee.io Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by cust-asf.ponee.io (Postfix) with SMTP id EAE19160BEC for ; Mon, 20 Nov 2017 14:45:33 +0100 (CET) Received: (qmail 91540 invoked by uid 500); 20 Nov 2017 13:45:32 -0000 Mailing-List: contact users-help@activemq.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: users@activemq.apache.org Delivered-To: mailing list users@activemq.apache.org Received: (qmail 91527 invoked by uid 99); 20 Nov 2017 13:45:32 -0000 Received: from pnap-us-west-generic-nat.apache.org (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Mon, 20 Nov 2017 13:45:32 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id BDC4AC0644 for ; Mon, 20 Nov 2017 13:45:31 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 3.763 X-Spam-Level: *** X-Spam-Status: No, score=3.763 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, FREEMAIL_ENVFROM_END_DIGIT=0.25, HEADER_FROM_DIFFERENT_DOMAINS=0.001, HTML_MESSAGE=2, KB_WAM_FROM_NAME_SINGLEWORD=0.2, RCVD_IN_DNSWL_NONE=-0.0001, RCVD_IN_MSPIKE_H2=-0.001, SPF_PASS=-0.001, URI_HEX=1.313, URI_TRY_3LD=0.001] autolearn=disabled Authentication-Results: spamd1-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=gmail.com Received: from mx1-lw-eu.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id LfvjOCGgeKQ2 for ; Mon, 20 Nov 2017 13:45:26 +0000 (UTC) Received: from mail-wr0-f177.google.com (mail-wr0-f177.google.com [209.85.128.177]) by mx1-lw-eu.apache.org (ASF Mail Server at mx1-lw-eu.apache.org) with ESMTPS id 8DAC45F366 for ; Mon, 20 Nov 2017 13:45:25 +0000 (UTC) Received: by mail-wr0-f177.google.com with SMTP id z14so8114244wrb.8 for ; Mon, 20 Nov 2017 05:45:25 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=mime-version:sender:in-reply-to:references:from:date:message-id :subject:to; bh=HYOo7HdYuW0ZUiyJRDsf2Rxll3a1009Ulsoq1nOtvSw=; b=V8YedhjU2m4sx0fMu5mPenqIU3klboDZis0Vj4lKbtj9ATlq/hki4smdI2+CYw765w 9jjNej+d9phZZ9eLaK33KzF3oAC+b8QLTybww5Z/Epn5F+aSkzaG/px09R+gxpQ9xmmk c+4vFScbZWC1GcLyr6gxrbPPSvu88ebYSUPb67gHBhRV3G4tX75ae8woUGFTS4sWbUyv pXpAvLAI7zBE2EM68SOy7RJpjgcReNgc/BSdxUNsyR3zQ2idxfohVigE7+o8mzE5LpGj 2C70/4ErdKxvxEV37MQga68L7VMF9HuuUrOH+k5sFu4s4HS6cmT/WQCNjqME/sgTVGo9 IYxw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:sender:in-reply-to:references:from :date:message-id:subject:to; bh=HYOo7HdYuW0ZUiyJRDsf2Rxll3a1009Ulsoq1nOtvSw=; b=fg2vY2Ld+WL1Nh+TwWhhC8t3GfOgMhtbDHhQYh84CbKWim6VhDmugLDLnhvg991I/G Bd8wUEOnYddIpnyUuEXvdpAWoJ+y3sAEU4iRApZz7SSUBqbG189vwuLRcbYl0UpyHnHY vXR3XX2dRpt68efuP7Ownf8LSdrsjjDyUKddDoMrWeDEhBy8umfK9+Yy/P3PfEJvVdMx mqQVn5pDFrkXcYKYuI/7nKXCVOAjYAlO7ARgmYYdNwj7U8z9jJGSprozoSrcUjQP4iht yXXVSayQkosTgXj4uFsjO1TF3YMpNHnPKePPJPb27qK+EIwRjXqviXzKz6f4984hjidw Fdqg== X-Gm-Message-State: AJaThX6R1em/u8Np6MjnjxgPvz2ZWbwA3ZiTGooKQy9NvXXS/5bQQ5AY g8aXK1uoHM4rkci8mDioA69BmfC2vZ5tXTZiz+8= X-Google-Smtp-Source: AGs4zMaOAmOBUP3ClLFySgik5TQ09v8ay/XRSloJcXB93EJJg5L8gEhq9VevNn5qEd1/apjMN5nsSzmhLRa8DBCVzgc= X-Received: by 10.223.196.247 with SMTP id o52mr11308386wrf.119.1511185524807; Mon, 20 Nov 2017 05:45:24 -0800 (PST) MIME-Version: 1.0 Sender: tbain98@gmail.com Received: by 10.223.142.113 with HTTP; Mon, 20 Nov 2017 05:45:22 -0800 (PST) Received: by 10.223.142.113 with HTTP; Mon, 20 Nov 2017 05:45:22 -0800 (PST) In-Reply-To: References: <1505518942569-0.post@n4.nabble.com> <1510917058886-0.post@n4.nabble.com> <1511176072574-0.post@n4.nabble.com> From: Tim Bain Date: Mon, 20 Nov 2017 06:45:22 -0700 X-Google-Sender-Auth: zxwP17X8mZvZqMfQGspRnGcOPXU Message-ID: Subject: Re: ActiveMQ DLQ issues To: ActiveMQ Users Content-Type: multipart/alternative; boundary="f403045f5526655ee0055e6a4cc7" archived-at: Mon, 20 Nov 2017 13:45:35 -0000 --f403045f5526655ee0055e6a4cc7 Content-Type: text/plain; charset="UTF-8" Thanks Gary. I was convinced that the default configuration did not contain any of the message-discarding options from the slow consumers page I linked to; I'm not sure where I got that idea. But that explanation makes more sense than my wild theory about the no-consumer behavior of statically included virtual topics. Tim On Nov 20, 2017 4:54 AM, "Gary Tully" wrote: > dlqDeliveryFailureCause java.lang.Throwable: TopicSubDiscard. > ID:localhost->fmiuseastp01-63022-1486246061090-100948:1:1:2 > > The cause indicates the a subscriber is full and has been configured to > discard messages[1]. See policyEntry from the default config [2] > > https://github.com/apache/activemq/blob/41a100766c19655816d575841ba559 > d33c63313d/activemq-broker/src/main/java/org/apache/ > activemq/broker/region/TopicSubscription.java#L718 > > https://github.com/apache/activemq/blob/41a100766c19655816d575841ba559 > d33c63313d/assembly/src/release/conf/activemq.xml#L55 > > On Mon, 20 Nov 2017 at 11:08 augustl wrote: > > > Thanks for the detailed response! > > > > Unfortunately I don't know when the out of memory error happened. My logs > > were full of them. Messages started appearing > > in the DLQ about 21:00, and the logs with the OOM start at around 05:00 > the > > day after. > > > > On the producer end, I connect to ActiveMQ using a > > ActiveMQConnectionFactory > > with the URL set to `failover:(tcp://localhost:61616)`. I create the > > connection by calling `createConnection` on this factory. I call > > `setClientId` on the connection before I call `start` on it to connect. > To > > create the session, I call `createSession(false, > > javax.jms.Session.AUTO_ACKNOWLEDGE)` on the connection object. I then > call > > `createProducer` on the session, with the destination being `new > > ActiveMQTopic("VirtualTopic.MyTopicNameHere")`. The message I send is > > created by calling `createTextMessage` on the session, and I pass a > string > > to it, containing some XML. Then I call `send` on the producer, with the > > message I just created. This producer uses > > "org.apache.activemq:activemq-core:5.7.0". > > > > The ActiveMQ broker that I connect to runs on version 5.14.4. > > > > I don't override any defaults or set any properties on the connection or > > the > > session or the producer or the message, so pretty much everything seems > to > > be running in default configuration as far as I can tell. > > > > The topic I posted messages to, "VirtualTopic.MyTopicNameHere" (which > isn't > > the real name, obviously), where the messages ended up in DLQ, is a topic > > that is automatically created by the producer. I don't have any extra > > config > > for it in the activemq config files. > > > > Your theory about the consumer for that topic on the remote broker is > > interesting. Things went a bit haywire so we restarted some things here > and > > there before we were able to fully collect information about what > happened. > > The broker that we configured as a network connector and that had > > VirtualTopic.MyTopicNameHere statically included, seemed to have problems > > as > > well. I noticed, for example, that this remote broker didn't accept new > > connections, until we restarted it. So it's possible that the root cause > is > > that the remote broker failed, and this caused things to go haywire on > the > > local broker. > > > > I was a bit surprised that messages to a topic ended up on a DLQ, since > if > > nobody consumes a message on a topic, the message is just discarded and > the > > world moves on. But you're saying that if a topic is statically included > as > > a destination in a network connector, and the broker cannot access that > > network connector, those messages end up in the DLQ of the local broker? > > That makes sense to me, at least. In that case, I suppose the only input > I > > have here is that it would be nice if the broker that puts the message on > > the DLQ has a more informative reason in the dlqDeliveryFailureCause > > attribute :) > > > > > > > > -- > > Sent from: > > http://activemq.2283324.n4.nabble.com/ActiveMQ-User-f2341805.html > > > --f403045f5526655ee0055e6a4cc7--