Return-Path: Delivered-To: apmail-geronimo-activemq-commits-archive@www.apache.org Received: (qmail 47476 invoked from network); 2 Feb 2006 16:37:11 -0000 Received: from hermes.apache.org (HELO mail.apache.org) (209.237.227.199) by minotaur.apache.org with SMTP; 2 Feb 2006 16:37:11 -0000 Received: (qmail 80372 invoked by uid 500); 2 Feb 2006 16:36:33 -0000 Delivered-To: apmail-geronimo-activemq-commits-archive@geronimo.apache.org Received: (qmail 79832 invoked by uid 500); 2 Feb 2006 16:36:30 -0000 Mailing-List: contact activemq-commits-help@geronimo.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: activemq-dev@geronimo.apache.org Delivered-To: mailing list activemq-commits@geronimo.apache.org Received: (qmail 79451 invoked by uid 99); 2 Feb 2006 16:36:26 -0000 Received: from asf.osuosl.org (HELO asf.osuosl.org) (140.211.166.49) by apache.org (qpsmtpd/0.29) with ESMTP; Thu, 02 Feb 2006 08:36:26 -0800 X-ASF-Spam-Status: No, hits=0.6 required=10.0 tests=NO_REAL_NAME X-Spam-Check-By: apache.org Received: from [209.237.227.194] (HELO minotaur.apache.org) (209.237.227.194) by apache.org (qpsmtpd/0.29) with SMTP; Thu, 02 Feb 2006 08:35:52 -0800 Received: (qmail 44714 invoked by uid 65534); 2 Feb 2006 16:35:30 -0000 Message-ID: <20060202163530.44711.qmail@minotaur.apache.org> Content-Type: text/plain; charset="utf-8" MIME-Version: 1.0 Content-Transfer-Encoding: 7bit Subject: svn commit: r374430 [38/45] - /incubator/activemq/site/ Date: Thu, 02 Feb 2006 16:33:52 -0000 To: activemq-commits@geronimo.apache.org From: jstrachan@apache.org X-Mailer: svnmailer-1.0.5 X-Virus-Checked: Checked by ClamAV on apache.org X-Spam-Rating: minotaur.apache.org 1.6.2 0/1000/N Added: incubator/activemq/site/Supporting+IO+Streams URL: http://svn.apache.org/viewcvs/incubator/activemq/site/Supporting%2BIO%2BStreams?rev=374430&view=auto ============================================================================== --- incubator/activemq/site/Supporting+IO+Streams (added) +++ incubator/activemq/site/Supporting+IO+Streams Thu Feb 2 08:31:10 2006 @@ -0,0 +1,268 @@ + + + + + + + + ActiveMQ - Supporting IO Streams + + + + + + + + + + + + + + + +
+ + + + + + +
+

Overview

+ +

Community

+ +

Using ActiveMQ

+ +

Features

+ +

Connectivitiy

+ +

Utilities

+ +

External Tools

+ +

Related Projects

+ +

Support

+ +

Developers

+ +

Tools we use

 + +

Feeds

+ + + + + + + + + +
+
+ 		+ Site +
+ + + News +
+ 		+ +
+ + + + + +
+ Supporting IO Streams + + +
+
+ + +
+
+

It'd be great to offer kick ass support for streaming files over ActiveMQ of any arbitrary size. The basic idea is to fragment the stream into multiple messages and send/receive those over JMS.

+

There are a few issues to consider...

+

Use casess

+

+

    +
  1. many producers writing and only 1 consumer. Each consumer ideally should completely process 1 stream at once (IO streams are often blocking IO, so the client thread can't do much with other streams while its reading 1).
    2. +
  2. if a consumer gets 1Gb through a 10Gb file and dies, we need to re-deliver the messages to another consumer.
    3. +
+

+

Goal

+

Our goal should be

+
    +
  • for each consumer to process a single stream in one go before trying to process another.
    • +
+

Possible Failure Conditions

+
    +
  • Consumer could die in the middle of reading a stream. Recovery options:
      +
    1. Trash the rest of the stream
      2. +
    2. Restart delivery of the stream to the next consumer
      3. +
    3. Continue delivery of the steam to the next consumer at the point of failure.
      4. +
    +
    • +
  • Producer could die in the middle of writing a stream. We may need to detect his failure. We could
      +
    1. Send the stream in a transaction. The stream is not sent to a consumer until it's fully received by the broker. Down side: consumer has high latency before being able to receive the message.
      2. +
    2. Consumer timeout: if a message is not receive soon enough the consumer assumes the producer is dead. (What if he's not??)
      3. +
    +
    • +
  • Consumer could start to receive in the middle of a stream. Condition could happen:
      +
    1. if another consumer assumed the broker was dead (but it was not).
      2. +
    2. if a non stream consumer acidentally removed messages, or messages were sent to the DLQ due to consumer rollbacks.
      3. +
    +
    • +
+

Implementation Issues

+
    +
  • we can use message groups to ensure the same consumer processes all messages for a given stream - but unfortunately message groups does not prevent a consumer being overloaded with more than one message group at a time - maybe thats a new feature we can add?
    • +
  • avoid the broker running out of RAM - so spool to disk (or throttle the producer) if the only consumer is working on a different stream.
      +
    • If messages sent using transactions, major chanages need may need to be made to how we do transaction management and message journaling. Currently, all messages in a in progress transaction are held in memory until it commits (Synchronization callbacks are holding on the the messages). The journal currently holds on to all transacted messages in it's journal + memory until commit, it does now allow the journal to rollover messages that are part of transaction that is in progess.
      • +
    +
    • +
  • given that the entire stream must be processed in one go, we can only ACK the entire stream of messages - so we need to disable pre-fetch?
      +
    • Prefetch disabling is not needed as the consumers send back a special kind of ack which only temporarily expands the prefetch window.
      • +
    +
    • +
+
+
+ +   +