activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Adrian Co <>
Subject Re: Log Aggregator
Date Mon, 30 Oct 2006 23:14:12 GMT

Kevin Kessler wrote:
> We are trying to build a system that could be thought of as a sort of log
> aggregator, with a small handful of listeners, and a much larger number of
> publishers (~150).  The messages are going to be 2-5K in size, and we may
> burst up to 3000 messages per second.  My main concern is that no Subscriber
> or JMS issue affects the publishers.  That means no flow-control, no-broker
> problem causing a publisher to hang, etc.  If something happens I just want
> to lose messages, and the publishers continue their real jobs unimpeded.  
Publish to a topic with no durable subscriber and use non-persistent 
delivery and you should be fine I think. Though you might run into the 
slow consumer problem, you can use the eviction strategy of ActiveMQ to 
handle it.

> Most of the publishers will be JBoss, although some are WebLogic.  I’ve
> played around with the embedded broker in JBoss, using log4j’s JMS appender
> to publisher messages, and I was very happy with the performance.  I think I
> don’t want to use a traditional hub and spoke sort of configuration, where
> the JMS clients connect to a centralized cluster of brokers, because I don’t
> want to make a remote network call in-line with the publishing system’s
> actual function call just for logging, and I don’t need durable
> subscriptions or delivery guarantees that the hub and spoke design lends
> itself to.  I figured a more peer-to-peer approach of embedded brokers would
> match the requirements better.  My initial idea was to use unreliable
> multicast to transmit the messages between the brokers, but, reading your
> web site, it doesn’t look like that is a design that you favor for a
> production environment.  I’m concerned that making a TCP connection to every
> other broker in the environment is going to be a big problem, especially
> since the vast majority of the brokers are publishers, and have no interest
> in what the other brokers are doing.
I would suggest using an embedded broker for each application that is 
doing the logging, this decouples your publisher from any external 
broker. Then creating a separate embedded broker with your consumer that 
would network with the publishers broker to receive messages. There 
could be a better way, but thats my two cents for now. :)
> Do you think ActiveMQ is a match for this application, and what transport
> protocol would you suggest?
I think ActiveMQ can handle it. VM transport with a TCP transport for 
the publisher and consumer I think.
> Thanks,
> Kevin Kessler

View raw message