Mailing-List: contact users-help@camel.apache.org; run by ezmlm
Precedence: bulk
Reply-To: users@camel.apache.org
MIME-Version: 1.0
In-Reply-To: <1460370065371-5780899.post@n5.nabble.com>
References: <1460370065371-5780899.post@n5.nabble.com>
Date: Wed, 13 Apr 2016 09:43:48 +0100
Message-ID: 
 <CAD3A9nM26hWQCB-M1YdYP=Erz_A89396cpwPLq13LBrKG2RppA@mail.gmail.com>
Subject: Re: Guaranteed file processing - JMS and file producer
From: Andrea Tarocchi <andrea.tarocchi@gmail.com>
To: users@camel.apache.org
Content-Type: multipart/alternative; boundary=001a11407f4ac087d7053059c6a0

--001a11407f4ac087d7053059c6a0
Content-Type: text/plain; charset=UTF-8
Content-Transfer-Encoding: quoted-printable

Hi Glen,

do you have a way to make the writing of the file itself idempotent?
i.e. you might use a unique id in the file name tush writing it a second
time would alter only the timestamp.

Is that doable? In this way you can put aside the idempotent repository and
overcome your issue.

Regards,
Andrea.

On Mon, Apr 11, 2016 at 11:21 AM, glenn <glen.neal@hotmail.co.uk> wrote:

> Hi
>
> I=E2=80=99ve been looking at how to use camel-jms to consume xml from a p=
ersistent
> queue, do some minor transformation/validation of the xml and either outp=
ut
> a single file or route a message to an error queue.
>
> The ultimate goal is to ensure that ALL messages are consumed and either =
a
> file is created ONCE or an error is routed ONCE to an error queue.
>
> As not wishing to use XA transactions by default, I opted to try out an
> approach using local JMS transactions, the Camel file component producing
> the files and use of an idempotent repository to guard against the
> re-processing of redelivered, duplicate messages.
>
> I=E2=80=99d appreciate views on whether it is the case that the approach =
of using
> the Camel file producer alongside native JMS transactions and an
> idempotentRepository cannot achieve the desired goal and that to achieve
> the
> goal must I opt to go down the XA route?
>
> My preference was a) not to have to add the additional complexity/overhea=
d
> of global transactions unless absolutely necessary and b) to use the Came=
l
> file component as the file producer.
>
> Whilst I can see the original, native JMS transaction based approach
> working
> in part, when considering just the sunny day scenario of successful file
> production only the approach does not guarantee that files will only be
> produced ONCE. A JVM crash at an inopportune moment between the
> JmsTransactionManager transaction commit and file repository record
> insertion can lead to skipping file production (the risk of breaking this
> guarantee is niche but it does, nonetheless, seem to exist).
>
> My current  thinking is the way to achieve the goal is to use XA
> transactions alongside an XA compliant file producer and JTA  transaction
> manager. However, that does seem to preclude the use of the Camel file
> component (being non fully transactional =E2=80=93 =E2=80=9Cbest efforts=
=E2=80=9D based).
>
> For record, I=E2=80=99m using the Camel File component as the file produc=
er with
> local JMS transactions, an externally defined Spring transaction manager,
> use of the Camel <transacted/> element within route and use of the
> FileIdempotentRepository. The FileIdempotentRepository filters out messag=
e
> duplicates based on the JmsMessageId (it is configured with eager=3Dtrue,
> using false just shifts problem).
> The log output below,  highlights the point where a JVM crash will lead t=
o
> failure to produce a file but still consume the message.
>
> <code>
> 05 Apr 2016 09:06:27   JmsTransactionManager          DEBUG Created JMS
> transaction on Session [Cached JMS Session: ActiveMQSession
> {id=3DID:ANAME-ABCDEFGH-63012-1459843265976-1:1:1,started=3Dfalse}
> java.lang.Object@44eb7aa] from Connection [Shared JMS Connection:
> ActiveMQConnection
>
> {id=3DID:ANAME-ABCDEFGH-63012-1459843265976-1:1,clientId=3DID:ANAME-ABCDE=
FGH-63012-1459843265976-0:1,started=3Dfalse}]
> 05 Apr 2016 09:06:27   TransactionContext             DEBUG
> Begin:TX:ID:ANAME-ABCDEFGH-63012-1459843265976-1:1:4
> 05 Apr 2016 09:06:27   ultJmsMessageListenerContainer DEBUG Received
> message
> of type [class org.apache.activemq.command.ActiveMQBytesMessage] from
> consumer [ActiveMQMessageConsumer {
> value=3DID:ANAME-ABCDEFGH-63012-1459843265976-1:1:1:244, started=3Dtrue }=
] of
> transactional session [Cached JMS Session: ActiveMQSession
> {id=3DID:ANAME-ABCDEFGH-63012-1459843265976-1:1:1,started=3Dtrue}
> java.lang.Object@44eb7aa]
> 05 Apr 2016 09:06:27   EndpointMessageListener        DEBUG
> Endpoint[inputQueue://ANAME.INV.RESP.Q] consumer received JMS message:
> ActiveMQBytesMessage ...
> 05 Apr 2016 09:06:27   TransactionErrorHandler        DEBUG Transaction
> begin (0x2e900519) redelivered(false) for (MessageId:
> ID:ANAME-ABCDEFGH-64219-1459497606992-9:1:1:1:30 on ExchangeId:
> ID-ANAME-ABCDEFGH-62996-1459843263591-0-22))
> 05 Apr 2016 09:06:27   JmsTransactionManager          DEBUG Participating
> in
> existing transaction
> 05 Apr 2016 09:06:27   TransactionErrorHandler        TRACE isRunAllowed(=
)
> -> true (Run allowed if we are not stopped/stopping)
> ...
> 05 Apr 2016 09:06:27   routeUnderTest           INFO
> ID:ANAME-ABCDEFGH-64219-1459497606992-9:1:1:1:30: Queue message: <?xml
> version=3D"1.0" encoding=3D"UTF-8" standalone=3D"yes"?>
> 05 Apr 2016 09:06:27   SendProcessor                  DEBUG >>>>
> Endpoint[AcknowledgementProcessor]
>
> Exchange[ID-ANAME-ABCDEFGH-62996-1459843263591-0-22][JmsMessage[JmsMessag=
eID:
> ID:ANAME-ABCDEFGH-64219-1459497606992-9:1:1:1:30]]
> 05 Apr 2016 09:06:27   AcknowledgementProcessor       DEBUG Payload is ..=
.
> 05 Apr 2016 09:06:27   AcknowledgementProcessor       DEBUG Body is ...
> 05 Apr 2016 09:06:27   routeUnderTest           INFO
> queue_ANAME.INV.RESP.Q_ID_ANAME-ABCDEFGH-64219-1459497606992-9_1_1_1_30:
> Processed message: <?xml version=3D"1.0" encoding=3D"UTF-8" standalone=3D=
"yes"?>
> 05 Apr 2016 09:06:27   DefaultStreamCachingStrategy   DEBUG Should spool
> cache 1023 -> false
> ...
> 05 Apr 2016 09:06:27   routeUnderTest           INFO
> queue_ANAME.INV.RESP.Q_ID_ANAME-ABCDEFGH-64219-1459497606992-9_1_1_1_30:
> Validated message: <?xml version=3D"1.0" encoding=3D"UTF-8" standalone=3D=
"yes"?>
> 05 Apr 2016 09:06:27   FileIdempotentRepository       DEBUG Appending
> ID:ANAME-ABCDEFGH-64219-1459497606992-9:1:1:1:30 to idempotent filestore:
> C:\Files\filestore\filesOut\.filestore.dat
>
> *** If JVM CRASH occurs here BEFORE JmsTransactionManager commit then
> FileIdempotentRepository entry will remain BUT no file has been created a=
nd
> no FileIdempotentRepository after processing operations will occur. Upon
> restoration, broker will redeliver and message will be consumed but
> FileIdempotentRepository will filter out duplicate message and hence NO
> file
> will be produced.  ***
>
> 05 Apr 2016 09:06:27   FilterProcessor                DEBUG Filter matche=
s:
> false for exchange:
>
> Exchange[ID-ANAME-ABCDEFGH-62996-1459843263591-0-22][JmsMessage[JmsMessag=
eID:
> ID:ANAME-ABCDEFGH-64219-1459497606992-9:1:1:1:30]]
> 05 Apr 2016 09:06:27   SendProcessor                  DEBUG >>>>
> Endpoint[file://name/someResponses]
>
> Exchange[ID-ANAME-ABCDEFGH-62996-1459843263591-0-22][JmsMessage[JmsMessag=
eID:
> ID:ANAME-ABCDEFGH-64219-1459497606992-9:1:1:1:30]]
> 05 Apr 2016 09:06:27   FileOperations                 DEBUG Using
> InputStream to write file: name\someResponses\BuyDeal.xml
> 05 Apr 2016 09:06:27   GenericFileProducer            DEBUG Wrote
> [name\someResponses\BuyDeal.xml] to [Endpoint[file://name/someResponses]]
> 05 Apr 2016 09:06:27   routeUnderTest           INFO
>
> queue_ANAME.INV.RESP.Q_ID_ANAME-ABCDEFGH-64219-1459497606992-9_1_1_1_30:B=
uyDeal.xml:
> Completed Processing
> 05 Apr 2016 09:06:27   TransactionErrorHandler        TRACE Is exchangeId=
:
> ID-ANAME-ABCDEFGH-62996-1459843263591-0-22 interrupted? false
> ...
> 05 Apr 2016 09:06:27   TransactionErrorHandler        DEBUG Transaction
> commit (0x2e900519) redelivered(false) for (MessageId:
> ID:ANAME-ABCDEFGH-64219-1459497606992-9:1:1:1:30 on ExchangeId:
> ID-ANAME-ABCDEFGH-62996-1459843263591-0-22))
> 05 Apr 2016 09:06:27   JmsTransactionManager          DEBUG Initiating
> transaction commit
> 05 Apr 2016 09:06:27   JmsTransactionManager          DEBUG Committing JM=
S
> transaction on Session [Cached JMS Session: ActiveMQSession
> {id=3DID:ANAME-ABCDEFGH-63012-1459843265976-1:1:1,started=3Dtrue}
> java.lang.Object@44eb7aa]
> 05 Apr 2016 09:06:27   ActiveMQSession                DEBUG
> ID:ANAME-ABCDEFGH-63012-1459843265976-1:1:1 Transaction Commit
> :TX:ID:ANAME-ABCDEFGH-63012-1459843265976-1:1:4
> 05 Apr 2016 09:06:27   TransactionContext             DEBUG Commit:
> TX:ID:ANAME-ABCDEFGH-63012-1459843265976-1:1:4 syncCount: 2
>
> </code>
>
> Would appreciate views.
>
> Thanks
>
> Glen
>
>
>
>
>
>
> --
> View this message in context:
> http://camel.465427.n5.nabble.com/Guaranteed-file-processing-JMS-and-file=
-producer-tp5780899.html
> Sent from the Camel - Users mailing list archive at Nabble.com.
>


--=20
"In a world without walls and fences who needs Windows and Gates?"

--001a11407f4ac087d7053059c6a0--