activemq-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "James Strachan" <james.strac...@gmail.com>
Subject Re: BlobMessage support for sending out-of-band BLOBs (was Re: FileMessage: we would like to contribute
Date Sun, 25 Feb 2007 20:36:57 GMT
I've renamed the thread - for those new to it there's some brief docs
http://cwiki.apache.org/ACTIVEMQ/blob-messages.html


On 2/22/07, Aleksi Kallio <aleksi.kallio@csc.fi> wrote:
>
> I'll move this discussion over to dev@activemq.apache.org as it belongs
> there...

cool

> Below I go on sketching how FileMessage might work. As you can see it is
> still quite sketchy in some parts so all comments and good ideas are
> welcome.
>
> Btw. is FileMessage a good name? In some cases we are not transferring
> files, but data in memory. Basically we are transferring a finite byte
> sequence. Is ByteSequenceMessage too awkward? I think it is maybe...

Yeah good point. It'd be good to not confuse things with ByteMessage
and StreamMessage from JMS.

So lets go with BlobMessage for now (as in Binary Large Object, like
JDBC and databases do?)



> > There are a few different possible implementations...
> >
> > (i) FileMessage ends up being a wrapper on top of the existing JMS streams
>
> This would be the preferred method in our case.
>
> Current streaming API uses a designated destination for the stream. If
> that destination is used for other purposes (other streams or messages)
> there has to be a way of separating current stream from the rest of the
> traffic.

Yeah - we should be able to use selectors on the types of messages and so forth.


> It would be great if FileMessage would behave just like other types of
> messages. In the streaming case I guess it is not possible to achieve?

Yeah - we could maybe put some facade on top of JMS Streams to do that

>

> > (ii) FileMessage uses some out of band transfer mechanism.
> > (iii) direct connection. This option is similar to (ii) but rather
> > than putting the file on some remote file server, the file stays on
> > the producer until the consumer has received it;
> >
> > The nice thing is I think all of these approaches can be handled
> > nicely by the single FileMessage API; then it can be a
> > configuration/policy issue as to exactly which implementation is used.
>
> If we look at the three implementations and what they would require from
> the two endpoints:
>
> 1. producer must inform what selector is to be used to receive data
> through JMS stream, consumers must acknowledge when they are ready to
> receive data
> 2. producer must place file available to external server and place URL
> to the message
> 3. producer must open a port and place URL to the message
>
> .. we see that streaming is maybe the trickiest one. If producer just
> starts streaming there is no guarantee (in the general case) that
> consumer will receive the whole stream from the beginning. Is that correct?
>
> In case 2, there is the question about pruning the files. Are they left
> to external server for ever? Or should consumer be capable of confirming
> the transfer, and after confirmation producer would remove the file? I
> don't think we should assume the consumer has a write access to file server.
>
> Case 3 is actually pretty straightforward. Do we want allow also a push
> option where consumer opens the port and producer delivers the file?
>
> > In terms of getting started, the simplest route is probably to add the
> > API in first (to start with assuming just a URL to the file which is a
> > no brainer) then we should be able to start adding different providers
> > to suit.
>
> Yes, I think that's the best way to go forward. I'll write something
> based on that JIRA issue and send it to dev@activemq.apache.org for
> comments. Does that sound good?

Sounds great

To get some of the lower level stuff sorted (basically changing
OpenWire to support BlobMessage) I've gone a head and hacked in a
basic solution of 2 and 3 from a JMS client / OpenWire perspective.

Grab trunk and look at the new methods on ActiveMQSession....

BlobMessage message = createBlobMessage(URL | File | InputStream)

Then you just send it like normal (and ditto on the receiver side)

The idea being that if you use the URL version, the URL is some
external URL that sticks around for a while (such as an out-of-band
upload to FTP or some external web URL etc). The File/InputStream
version would upload the blob to some server then make the URL from
that.

I've included a little strategy for doing that (which currently has no
implementation yet!) - I wondered if you fancied trying to do that
using the URL class?


So things to do are...

* implement the code in the DefaultBlobUploadStrategy class to use the
URL class to upload the file/stream and test it works

* add an ability to the broker to delete the URLs on BlobMessages
after they are no longer required when being used with queues or
durable topics etc

* write lots of tests!!

* document on the wiki - I've made a little start here:
http://cwiki.apache.org/ACTIVEMQ/blob-messages.html

* wrap up a server side uploader implementation; e.g. a little
embedded Jetty which can support GET  PUT/POST and DELETE operations
on a web URL for doing RESTful acess to local files which can then be
embedded either in producers (for cases where consumers connect
directly to producers) or inside brokers, which is a more common
scenario.

I've updated the main JIRA so far (we could spin off child/sub/related
JIRAs as required)
https://issues.apache.org/activemq/browse/AMQ-1075

Do you fancy diving in and trying to tackle some of the above?

-- 

James
-------
http://radio.weblogs.com/0112098/

Mime
View raw message