camel-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Daniel Carleton (Updated) (JIRA)" <j...@apache.org>
Subject [jira] [Updated] (CAMEL-5113) Parallel and fault tolerant message processing for SQS endpoints.
Date Fri, 23 Mar 2012 17:45:28 GMT

     [ https://issues.apache.org/jira/browse/CAMEL-5113?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel
]

Daniel Carleton updated CAMEL-5113:
-----------------------------------

    Description: 
I'm using Camel to implement parallel processing of jobs in an SQS queue, and ran into a few
issues with the current implementation of the SQS component:

# SqsConsumer uses a blocking/synchronous processor, which prevents parallel processing of
multiple messages from a single endpoint.
# Having a maxMessagesPerPoll other than one doesn't seem to make sense, because any messages
not being actively processed should be left back in the queue for other consumers to have
a chance with.
# Rollback processing doesn't go back to SQS and set the visibility timeout to zero, which
prevents immediate retries. 

I propose the following solutions to these problems:

# Use an asynchronous processor in SqsConsumer by way of getAsyncProcessor().  (Messages in
SQS aren't guaranteed to be FIFO anyway, so there should be no issue with order of processing.)
# Replace maxMessagesPerPoll with maxInFlightMessages.  Put a semaphore in SqsConsumer to
control the maximum number of in flight messages, and when polling SQS always set the number
of available permits as the maximum number of messages to retrieve.
# In the onFailure callback for an exchange set the visibility timeout in SQS to zero via
ChangeMessageVisibility.

How does this sound?  I'm working on a patch.  This is my first work on Camel, so if you see
any problems with my approach let me know!

Thanks,

- Dan


  was:
I'm using Camel to implement parallel processing of jobs in an SQS queue, and ran into a few
issues with the current implementation of the SQS component:

# SqsConsumer uses a blocking/synchronous processor, which prevents parallel processing of
multiple messages from a single endpoint.
# Having a maxMessagesPerPoll other than one doesn't seem to make sense, because any messages
not being actively processed should be left back in the queue for other consumers to have
a chance with.
# Rollback processing doesn't go back to SQS and set the visibility timeout to zero, which
prevents immediate retries.  (I guess if deleteAfterRead is turned on this doesn't make sense,
but then doesn't turning on deleteAfterRead break fault tolerance completely?)

I propose the following solutions to these problems:

# Use an asynchronous processor in SqsConsumer by way of getAsyncProcessor().  (Messages in
SQS aren't guaranteed to be FIFO anyway, so there should be no issue with order of processing.)
# Replace maxMessagesPerPoll with maxInFlightMessages.  Put a semaphore in SqsConsumer to
control the maximum number of in flight messages, and when polling SQS always set the number
of available permits as the maximum number of messages to retrieve.
# In the onFailure callback for an exchange set the visibility timeout in SQS to zero via
ChangeMessageVisibility.

How does this sound?  I'm working on a patch.  This is my first work on Camel, so if you see
any problems with my approach let me know!

Thanks,

- Dan


    
> Parallel and fault tolerant message processing for SQS endpoints.
> -----------------------------------------------------------------
>
>                 Key: CAMEL-5113
>                 URL: https://issues.apache.org/jira/browse/CAMEL-5113
>             Project: Camel
>          Issue Type: Improvement
>          Components: camel-aws
>            Reporter: Daniel Carleton
>
> I'm using Camel to implement parallel processing of jobs in an SQS queue, and ran into
a few issues with the current implementation of the SQS component:
> # SqsConsumer uses a blocking/synchronous processor, which prevents parallel processing
of multiple messages from a single endpoint.
> # Having a maxMessagesPerPoll other than one doesn't seem to make sense, because any
messages not being actively processed should be left back in the queue for other consumers
to have a chance with.
> # Rollback processing doesn't go back to SQS and set the visibility timeout to zero,
which prevents immediate retries. 
> I propose the following solutions to these problems:
> # Use an asynchronous processor in SqsConsumer by way of getAsyncProcessor().  (Messages
in SQS aren't guaranteed to be FIFO anyway, so there should be no issue with order of processing.)
> # Replace maxMessagesPerPoll with maxInFlightMessages.  Put a semaphore in SqsConsumer
to control the maximum number of in flight messages, and when polling SQS always set the number
of available permits as the maximum number of messages to retrieve.
> # In the onFailure callback for an exchange set the visibility timeout in SQS to zero
via ChangeMessageVisibility.
> How does this sound?  I'm working on a patch.  This is my first work on Camel, so if
you see any problems with my approach let me know!
> Thanks,
> - Dan

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message