Mailing-List: contact users-help@activemq.apache.org; run by ezmlm
Precedence: bulk
Reply-To: users@activemq.apache.org
Received-SPF: pass (athena.apache.org: local policy)
Message-ID: <19113841.125261210236974250.JavaMail.root@elysia.void.it>
MIME-Version: 1.0
Date: Thu, 08 May 2008 10:56:06 +0200
From: Stefano Bagnara <apache@bago.org>
User-Agent: Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US;
 rv:1.9a1) Gecko/20051206 Thunderbird/1.6a1 Mnenhy/0.7.3.0
To: users@activemq.apache.org
Subject: SMTP Server (Apache James) spooling hints
Content-Type: text/plain; charset=ISO-8859-15; format=flowed
Content-Transfer-Encoding: 7bit

Hi all,

I'm an Apache JAMES committer and I'm "almost" new to ActiveMQ.

I'm starting analysis on how to replace our default spool with ActiveMQ 
and I hope you can give me some hints :-)
It would be better to use ActiveMQ via JMS (more flexibility) but if 
there is any better solution to our problems by using specific ActiveMQ 
APIs then why not!!

Our scenario is an SMTP Server so we have something like this:

1) SMTP Server receives messages and put them to the spool. The spool 
have to be persistent because once the message has been posted via SMTP 
we cannot loose it. Most time the message will be consumed very fast, so 
in past I looked at using Kaha directly for this, but maybe the 5.0  AMQ 
Message Store already handle this one in a performant way?

2) Our current spooling have this architecture:
we have a single "spool" that contains messages with a "state". We read 
a random message from the spool, look at its state and then start the 
processing depending on the state itself at the end of the processing we 
can alter the state and leave the message in the spool, or we can remove 
it from the spool. In the processing we could even push more messages 
into the spool (e.g: to split the message to 2 different paths). ATM the 
re is no transaction management.
The processing from a state to another (or to delete) is a sequence of 
micro-processings (named matchers/mailets in james), so the actual 
status depends also on what matchers/mailets have been processed so far, 
but we currently keep this in memory and never store this. So if 
something goes wrong (given that we don't have transactions) we simply 
start from the beginning of that "state processor" (I'd like to improve 
this issue, too, with the new ActiveMQ based spool).
Some times the message is simply moved from one state to another a few 
times and then it is removed from the spool because of 2 causes:
a) it has been moved to the "outgoing spool" (the spool for the messages 
to be sent to other smtp servers)
b) it has been posted to an user inbox.
Other times the message is altered in its content.
So you see in James we currently have a single "message store" and we 
can "lock on a message" (so no other thread will take it) "retrieve it", 
"update and unlock it" (alter its state or state+content) or "remove 
it". How would you manage this with ActiveMQ?

3) Outgoing spool:
The outgoing spool in JAMES is a spool like the main spool, with the 
difference that a message delivery could fail and there is a retry 
schedule. So we try to send a message, on failure we try again 10 
minutes later, then 30 minutes later, then 2 hours later (it is 
configurable) and so on. ATM we store the "next-attempt-date" and then 
each "deliverer" simply take the message with the minor 
next-attempt-date and if it is due for delivery it starts its work, 
otherwise it will simply wait the needed time (one deliverer is noticed 
when a *new* message enter this spool / They all "wait" on the spool and 
the spool is noticed one at each store).
The most common case is:
a) the message we received at #1 entered the spool #2 and is processed 
very fast and it ends in the outgoing spool #3 where it is delivered on 
the first attempt. In this case it would be cool if the message was in 
memory and simply written once for safety because the processing should 
be fast and it would be slow to read it again from the disk.
b) we fail our first attempt, then it does not make sense to keep it in 
memory because we know we won't need it in the next X minutes/hours.
Any suggestions on how to do this with ActiveMQ?

As a last point we have to take care of 2 different use-cases:
I) most traffic is done by fastmoving small messages but
II) many messages are 1-10MB in size, and a few message could be even 
100MB or even more: how should we handle this messages in ActiveMQ given 
that we can't take them in memory but we simply want to stream then in 
and out from the server?

I understand this is a lot of questions, but I would really appreciate 
any hint, even partial. I'm collecting ideas :-)

Stefano

PS: we are also evaluating using JCR for inboxes if you was wondering, 
but this is another story, for another list ;-)