Mailing-List: contact users-help@activemq.apache.org; run by ezmlm
Precedence: bulk
Reply-To: users@activemq.apache.org
Received-SPF: pass (athena.apache.org: domain of marc.weil@gmail.com
 designates 74.125.92.148 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=mime-version:in-reply-to:references:from:date:message-id:subject:to
         :content-type;
        b=fxJAg+kQ61/MwkBWcKhGr+auUBj4/AO632IWzS0WfTbK3+kUhZQa4rP6GlRNSvpJ4K
         CsvFmfIaIJieTrFP1NFeoay1PU7/Mp0rlkvIPoFSXxPclsrJ7pUroNII23zlFBE6sPV1
         eH8uJBdhAaCSEm+zq+dK19W6rlMy0ov90Fqj4=
MIME-Version: 1.0
In-Reply-To: <69b2d60e1003031426o63c8df20tb8002df970dd2467@mail.gmail.com>
References: <27766358.post@talk.nabble.com>
 <0669EB61C3BF9143AF3D7BE84DF29B52C7658061@rocjexm01.endava.net>
	<27766785.post@talk.nabble.com>
 <0669EB61C3BF9143AF3D7BE84DF29B52C7658062@rocjexm01.endava.net>
	<22a6436b1003031006v745ef5ebt8cc648a3bf288b40@mail.gmail.com>
	<69b2d60e1003031426o63c8df20tb8002df970dd2467@mail.gmail.com>
From: Marc Weil <marc.weil@gmail.com>
Date: Thu, 4 Mar 2010 17:52:28 -0500
Message-ID: <22a6436b1003041452h735852favbc73bb2e42ecef9@mail.gmail.com>
Subject: Re: best approach to transfer large File, modify content on receive
To: users@activemq.apache.org
Content-Type: multipart/alternative; boundary=0016362845663358180481017538

--0016362845663358180481017538
Content-Type: text/plain; charset=windows-1252
Content-Transfer-Encoding: quoted-printable

Right now we are just using the built-in Jetty HTTP server that comes with
ActiveMQ. However, this fails after files get any larger than ~1GB, due to
some internal problem with Jetty's handling of large file transfers over
HTTP. Our project is still in initial stages of development so we haven't
had to really face this problem head-on quite yet. Right now what we are
doing is sending a URL pointing to the >1GB file on the remote server as a
BlobMessage, and ActiveMQ pretends like it was actually responsible for
transferring the file. In the future we will be using SFTP or SCP as our
out-of-band transfer method. We haven't really figured out the logistics fo=
r
this yet though.

The good thing about BlobMessages is that you are not restricted by the hea=
p
space of the JVM when transferring large files as you might be if you used =
a
stream (since you'd have to read the stream into memory before writing it
out unless you wrote your own chunk-and-flush method, which is what the
BlobMessage does for you anyway). They were a breeze to set up and use and
have been perfect for our needs thus far.

Right now we haven't had to do any performance tuning of AMQ besides what
the default configuration came with. But again, we aren't production-ready
yet so we haven't hit any configuration-related bottlenecks yet. We also
aren't sending a large quantity of these large messages. The levels of
traffic anticipated for this project are high-read (small AMQ messages for
queries), low-write (potentially very large AMQ messages writing big pieces
of data), so we haven't run into any producer/consumer flow problems yet.

Hope that helped a little.

- Marc

On Wed, Mar 3, 2010 at 5:26 PM, Fred Moore <fred.moore77@gmail.com> wrote:

> Hi Marc,
>
> > We are sending files >1GB around using the ActiveMQ's BlobMessage.
>
> ...mmh you have an interesting real-life use case: can you give us a few
> more details?
>
> Things like the "side protocol" you use (FTP? WebDAV?), daily volumes, an=
y
> special other ActiveMQ tuning you performed etc
>
> Cheers,
> F.
>
>
>
>
> On Wed, Mar 3, 2010 at 7:06 PM, Marc Weil <marc.weil@gmail.com> wrote:
>
> > 100 MB isn't really that large. We are sending files >1GB around using
> the
> > ActiveMQ's BlobMessage. Why don't you try using that instead of doing
> > chunking and serialization/deserialization manually?
> >
> > http://activemq.apache.org/blob-messages.html
> >
> > Marc
> >
> > On Wed, Mar 3, 2010 at 6:17 AM, Cristian Botiza
> > <Cristian.Botiza@endava.com>wrote:
> >
> > > I would look for some streaming options in ActiveMQ if you want to se=
nd
> > the
> > > full file. But 100 MB is quite huge...anyway see class BytesMessage.
> You
> > may
> > > try using a BufferedStream around the byte array, or read it in chunc=
ks
> > > using readBytes(byte[]).
> > >
> > > ________________________________________
> > > From: dara kok [mrpc.cambodia@gmail.com]
> > > Sent: 03 March 2010 13:05
> > > To: users@activemq.apache.org
> > > Subject: RE: best approach to transfer large File, modify content on
> > > receive
> > >
> > > So you think sending the file in full size is not performance
> efficient?
> > I
> > > wonder what happens when the receiving app get the file, will the ful=
l
> > > content of the file load into memory or will just part of it will get
> > > loaded
> > > and the rest will load on demand?
> > >
> > > Thanks,
> > >
> > >
> > > Can you chunk the file? In this case you might split it between
> multiple
> > > producers and consumers
> > > and play with the number of producers/consumers and chunck size. You
> may
> > > send and process chuncks
> > > concurrently and also limit the message size.
> > >
> > > Just one option...
> > > --
> > > View this message in context:
> > >
> >
> http://old.nabble.com/best-approach-to-transfer-large-File%2C-modify-cont=
ent-on-receive-tp27766358p27766785.html
> > > Sent from the ActiveMQ - User mailing list archive at Nabble.com.
> > >
> > > The information in this email is confidential and may be legally
> > > privileged. It is intended solely for the addressee. Any opinions
> > expressed
> > > are mine and do not necessarily represent the opinions of the Company=
.
> > > Emails are susceptible to interference. If you are not the intended
> > > recipient, any disclosure, copying, distribution or any action taken =
or
> > > omitted to be taken in reliance on it, is strictly prohibited and may
> be
> > > unlawful. If you have received this message in error, do not open any
> > > attachments but please notify the EndavaIT Service Desk on (+44 (0)87=
0
> > 423
> > > 0187), and delete this message from your system. The sender accepts n=
o
> > > responsibility for information, errors or omissions in this email, or
> for
> > > its use or misuse, or for any act committed or omitted in connection
> with
> > > this communication. If in doubt, please verify the authenticity of th=
e
> > > contents with the sender. Please rely on your own virus checkers as n=
o
> > > responsibility is taken by the sender for any damage rising out of an=
y
> > bug
> > > or virus infection.
> > >
> > > Endava Limited is a company registered in England under company numbe=
r
> > > 5722669 whose registered office is at 125 Old Broad Street, London,
> EC2N
> > > 1AR, United Kingdom. Endava Limited is the Endava group holding compa=
ny
> > and
> > > does not provide any services to clients. Each of Endava Limited and
> its
> > > subsidiaries is a separate legal entity and has no liability for
> another
> > > such entity's acts or omissions. Please refer to the =93Legal=94 sect=
ion on
> > our
> > > website for a list of legal entities.
> > >
> >
>

--0016362845663358180481017538--