activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From kevinle92618 <kevinle92...@yahoo.com>
Subject Re: Large File Transfer
Date Thu, 10 Jun 2010 14:07:18 GMT

Hi Dejan,

Currently we have a hard requirement to support large file transfer resume. 
We're very keen on using ActiveMQ, along with Camel, thus we must implement
this "resume" feature.  I'm in the investigation stage but I was thinking of
implement this the follow way:

Following Lou's suggestion, the producer will break down the large file to
be sent into chunks and send them to the broker (ActiveMQ) one at a time
until the entire file is sent.  In the event of a file transfer
interruption, the producer will detect this event and when connection is
restore, it will only send the remainder to the broker (with a resume flag
or something like that).  The broker will then continue to receive the rest
of the file knowing that it will have to concatenate the files together
before putting it on the queue for consumption.

I'm not sure if this is the best solution or not.  If you have a better
suggestion please feel free to let me know.  My personal email address is
kevinle92618@yahoo.com



Dejan Bosanac wrote:
> 
> Hi,
> 
> just to confirm that ActiveMQ comes with HTTP and FTP strategies for
> upload
> and download, that doesn't support resume. This can be implemented as a
> separate strategy. All contributions are welcomed.
> 
> Cheers
> --
> Dejan Bosanac - http://twitter.com/dejanb
> 
> Open Source Integration - http://fusesource.com/
> ActiveMQ in Action - http://www.manning.com/snyder/
> Blog - http://www.nighttale.net
> 
> 
> On Thu, Jun 10, 2010 at 12:28 AM, kevinle92618
> <kevinle92618@yahoo.com>wrote:
> 
>>
>> Thanks Lou.  I was afraid of this.  If I understand you correctly,
>> ActiveMQ
>> does NOT support resume transfer.  However, you were able to get around
>> that
>> problem by implementing the snippet codes you mentioned in your previous
>> post right?  Do you mind posting the codes for the TransferStatus class
>> as
>> well.  Many thanks.
>>
>>
>> Lou Parisi wrote:
>> >
>> >
>> > kevinle92618 wrote:
>> >>
>> >> I don't have answers to your questions.  I'm interested in knowing if
>> >> ActiveMQ has the "resume" transfer capability for large file.  Please
>> >> reply if you have the answer.  Thanks.
>> >>
>> >
>> > It has been a while since I worked on this but I did get the "resume"
>> > working.  As I recall, I kept track of the number of bytes transferred
>> and
>> > if there was a failure I just had to resume and read past that many
>> bytes
>> > and restart transfer.  Here is a snippet from some sample code I
>> created
>> > for testing.
>> >
>> >
>> >    private static void startTransfer() throws Exception {
>> >       RandomAccessFile outFile = null;
>> >       InputStream in = null;
>> >       byte[] buf = new byte[bufSize];
>> >       int len = 0;
>> >       URLConnection conn = null;
>> >
>> >       // len == -1 when end of InputStream is read
>> >       while (len != -1) {
>> >          try {
>> >             conn = url.openConnection();
>> >             conn.setConnectTimeout(connectionTimeout);
>> >             in = getInputStream(conn);
>> >             in.skip(transferStatus.getBytesDownloaded());
>> >
>> >             int count = 0;
>> >             System.out.println("downloading");
>> >             transferStatus.setStatus(TransferStatus.DOWNLOADING);
>> >             outFile = new RandomAccessFile("build/outfile", "rw");
>> >             outFile.seek(transferStatus.getBytesDownloaded());
>> >             while ((len = in.read(buf)) > 0) {
>> >                outFile.write(buf, 0, len);
>> >                transferStatus.incrementBytesDownloaded(len);
>> >                updateElapsedTime();
>> >             }
>> >          } catch (Exception e) {
>> >             System.out.println("Exception transfering data: " +
>> > e.getMessage());
>> >          } finally {
>> >             if ( in != null ) {
>> >                in.close();
>> >             }
>> >             if ( outFile != null ) {
>> >                outFile.close();
>> >             }
>> >             conn = null;
>> >          }
>> >       }
>> >       System.out.println("transfer complete");
>> >       transferStatus.setStatus(TransferStatus.COMPLETE);
>> >    }
>> >
>> >
>>
>> --
>> View this message in context:
>> http://old.nabble.com/Large-File-Transfer-tp23378044p28836565.html
>> Sent from the ActiveMQ - User mailing list archive at Nabble.com.
>>
>>
> 
> 
> -----
> Dejan Bosanac
> 
> Open Source Integration - http://fusesource.com/
> ActiveMQ in Action - http://www.manning.com/snyder/
> Blog - http://www.nighttale.net
> 

-- 
View this message in context: http://old.nabble.com/Large-File-Transfer-tp23378044p28843604.html
Sent from the ActiveMQ - User mailing list archive at Nabble.com.


Mime
View raw message