activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "James Strachan" <james.strac...@gmail.com>
Subject Re: FileMessage: we would like to contribute
Date Tue, 20 Feb 2007 08:47:57 GMT
On 2/14/07, Aleksi Kallio <aleksi.kallio@csc.fi> wrote:
>
> There has been discussion regarding FileMessage or something similar: a
> message for transferring large amounts of data.
>
> For background, see:
> http://issues.apache.org/activemq/browse/AMQ-1075
>
> We need something that offers better service than streaming, of course
> by building on top of it. That is why we would be interested in this and
> ready to contribute our effort into making it happen.
>
> We have to be capable of moving BIG files over JMS, in the order of
> couple of gigabytes. Normally they are a lot smaller, but also the big
> ones must be handled.
>
> Features/fixes we are looking for are:
>
> 1) streaming requires dedicated destination, which makes things complicated
> 2) resuming transfers
> 3) easy monitoring of progress
>
>
>  From core developers we would appreciate feedback on how to proceed
> with developing such a feature. From rest of the communinity we would
> gladly get feedback on how you would like this to be done and what kind
> of API would be the best.
>
>
> In the spirit of making ActiveMQ even better,

Sounds great! Welcome!

The comments on the JIRA issue describe the kind of API I was thinking of...
http://issues.apache.org/activemq/browse/AMQ-1075


There are a few different possible implementations...

(i) FileMessage ends up being a wrapper on top of the existing JMS streams
http://activemq.apache.org/jms-streams.html

(ii) FileMessage uses some out of band transfer mechanism. e.g. the
FileMessage essentially just carries around a URL; then the file is
transferred using some FTP/HTTP site. e.g. the producer sends to the
remote file/web server, then sends the message with a reference to it.
i.e. using an out-of-band transfer mechanism. The benefit of this is
that existing file/web servers can be used for the actual file
transfer piece, then the message broker can be used for reliable
message load balancing and failover. i.e. making sure exactly one
consumer processes the message properly, in a transactional way etc.

(iii) direct connection. This option is similar to (ii) but rather
than putting the file on some remote file server, the file stays on
the producer until the consumer has received it; also the producer
listens on a socket for the consumer. Then the FileMessage includes a
URL which when opened on the consumer will essentially connect
directly to the producer of the file. So the FileMessage is a normal
JMS message sent around the JMS network (over clusters & network store
& forward etc). Then when its finally received by a consumer, the
consumer opens the URL which in effect causes the consumer to connect
directly to the producer, to then stream the file.

All of these options have different strengths and weaknesses. I think
for many use cases (iii) is good but I could see folks liking (i) and
(ii) as well.

The nice thing is I think all of these approaches can be handled
nicely by the single FileMessage API; then it can be a
configuration/policy issue as to exactly which implementation is used.

In terms of getting started, the simplest route is probably to add the
API in first (to start with assuming just a URL to the file which is a
no brainer) then we should be able to start adding different providers
to suit.

BTW we might also want to support senders providing a File rather than
an InputStream if you want to be able to monitor progress (as with a
stream you've no idea how long its gonna be)

-- 

James
-------
http://radio.weblogs.com/0112098/

Mime
View raw message