activemq-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Hiram Chirino" <hi...@hiramchirino.com>
Subject Re: BlobMessage support for sending out-of-band BLOBs
Date Wed, 14 Mar 2007 16:02:33 GMT
On 3/14/07, Hiram Chirino <hiram@hiramchirino.com> wrote:
> On 3/14/07, James Strachan <james.strachan@gmail.com> wrote:
> > On 3/14/07, Aleksi Kallio <aleksi.kallio@csc.fi> wrote:
> > > >> I had a bit of a look at the Jetty code and it doesn't seem to have
a
> > > >> simple 'file servlet' for supporting GET/PUT/DELETE of files in a
> > > >> directory; so I was thinking we could contribute one to the Jetty6
> > > >> tree and reuse that. e.g. maybe use the DefaultServlet in Jetty (which
> > > >> does GET fine) then add PUT/DELETE support
> > >
> > > I have this working (upload, download, remove) and have tested it with
> > > files in the order of a couple of gigabytes.
> >
> > Awesome!
> >
> >
> > > I'd have two questions, all input is appreciated...
> > >
> > >
> > > 1) I'm currently using standard Java URL / HttpURLConnection stuff to
> > > connect to the web server. Unfortunately there is a bug or design issue
> > > that makes HttpURLConnection to load everything into memory:
> > >
> > > http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=5026745
> > > http://bugs.sun.com/bugdatabase/view_bug.do?bug_id=4212479
> > >
> > > That is definitely not appropriate for our purposes. Currently you can
> > > also use chunked transfer mode
> > > (HttpUrlConnection.setChunkedStreamingMode()) and it works well with
> > > Jetty. However not all HTTP servers support this mode, I don't know if
> > > it is an issue. The bigger issue is that the method only exists in Java
> > > 1.5. In other words, it is not usable in the ActiveMQ context?
> > >
> > > I think the only solution would be to use some other HTTP package,
> > > probably commons-http. Or do you see some other workarounds? Personally
> > > I would have preferred if the thing would have not required anything but
> > > standard JRE classes. Unfortunately it seems impossible, unless one
> > > writes the HTTP code from scratch, which does not sound like a clever
> > > thing to do.
> >
> > Good point. Trunk of ActiveMQ is currently Java 5 specific anyway, so
> > using chunked streaming mode sounds a fine workaround for now.
> >
> > We could always switch to using either the Jetty client stuff or
> > commons-httpclient if its on the classpath. It'd be nice to keep the
> > dependency list as small as possible; but work better if
> > commons-httpclient or jetty-client is on the classpath etc.
> >
> >
> > > 2) While storing files to an intermediary file server they are named
> > > after their message ID's. Is that wise?
> >
> > Well the Message IDs are guarrenteed to be system wide unique strings,
> > so they're quite good for the names of the files - but...
> >
> > > Message ID's look a bit unfit
> > > for filenames. Do you think in some environments/FS's these kind of
> > > names might cause problems?
> >
> > Agreed. We should definitely make the message IDs both URI and file
> > name friendly. So how about we search & replace the message ID string
> > and switch some of the dodgy characters and replace them with
> > something.
> >
> > e.g.
> >
> > switch : to a / maybe?
> >
> > Then each producer will be in a directory which will be inside a
> > directory named after the connection ID etc.
> >
> > Thinking about it - it might make more sense to change ActiveMQ itself
> > to create MessageId strings which use / rather than : maybe...
> >
>
> The character restrictions will vary depending on the final store,
> different files systems have different restrictions.  So encoding the
> message id to a safe file name should be left up to the servlet
> managing that file system.
>

I could image someone may use S3 buckets to store this stuff :)

>
> > --
> >
> > James
> > -------
> > http://radio.weblogs.com/0112098/
> >
>
>
> --
> Regards,
> Hiram
>
> Blog: http://hiramchirino.com
>


-- 
Regards,
Hiram

Blog: http://hiramchirino.com

Mime
View raw message