activemq-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steve Barham" <steve.bar...@formicary.net>
Subject Re: AW: AW: AW: Final repost: SSL transport failure
Date Wed, 29 Mar 2006 08:11:50 GMT
Hi Mike,

Thanks for your continued input. Unfortunately, the message definitely
does not get onto the queue - the client does not receive it, and after
the exception the server seems to be disconnected from the broker - it can
no longer send or consume messages.

The error did not occur with the TCP transport. I've not had a chance to
try any of the others, as the TCP and SSL transports seemed the most sane
for my application.

Steve


> hi,
>
> the whole thing sounds a bit strange. It looks like the file is transfered
> successfully into the queue and then the connection is closed. Can your
> client access the message successfully? I am not sure if this error is
> critical, it is just a warning after all. Have you tried it without the
> SSL protocol and did the error also occure?
> From my understanding of AMQ a message of about 700kb should not be a big
> problem.
> AMQ get a EOS packet and passes it through to the
> asyncchanneltoasynccommandchannel, where it throws the peer disconnected
> exception. That is then handed to the activemqconnection where the no
> exception listener warning it thrown out, if there would have been a
> exception listener it the error would have been passed further on.
> But to be honest I have no more clues beside this. My understanding of the
> internal mechanism and of activeio and how messages are transported is
> just to bad. I would say this exception is no big deal, but I have no idea
> what the EOS packet means.
>
> -----Ursprüngliche Nachricht-----
> Von: Steve Barham [mailto:steve.barham@formicary.net]
> Gesendet: Dienstag, 28. März 2006 17:48
> An: activemq-users@geronimo.apache.org
> Betreff: Re: AW: AW: Final repost: SSL transport failure
>
>
>
> Hi Mike,
>
> Thanks for the input. It's a single (albeit large) message. And I'm not
> sure as to why this would cause a failure in my 'server' application?
>
> 3163 SERVER <---> Broker : Queue <---> CLIENT 3175
>
> To me, the fact that it always occurs on a message of this size suggests
> that some internal buffer is being overflowed (hence the lines:)
>
>> Socket Connection: /127.0.0.1:61616 -> /127.0.0.1:3163, called close()
>> Socket Connection: /127.0.0.1:61616 -> /127.0.0.1:3163, called
> closeInternal(true)
>> Socket Connection: /127.0.0.1:61616 -> /127.0.0.1:3163, SEND TLSv1
> ALERT: warning, description = close_notify
>> Socket Connection: /127.0.0.1:61616 -> /127.0.0.1:3163, WRITE: TLSv1
> Alert, length = 18
>> Socket Connection: /127.0.0.1:61616 -> /127.0.0.1:3163, called close()
>> Socket Connection: /127.0.0.1:61616 -> /127.0.0.1:3163, called
> closeInternal(true)
>> Socket Connection: /127.0.0.1:61616 -> /127.0.0.1:3163, called close()
>> Socket Connection: /127.0.0.1:61616 -> /127.0.0.1:3163, called
> closeInternal(true)
>
> Surely it's the job of ActiveMQ to split large messages where the
> underlying transport requires short messages, or to maintain a connection
> where the timeout is so short? I've not seen such problems using SSL
> Sockets before now in other Java applications.
>
> Cheers,
>
> steve
>
>
>>
>> hmm I think it looks like the app 3175 isn'T replying.  When my
>> understanding is correct the timeout of SSL is about 500ms, maybe
>> something takes to long. Or could it be that the app has closed the
>> connection?
>> Have you tried to send a lower number of messages and tried to find at
>> if
>> it stops to work at a certain threshold?
>>
>> -----Ursprüngliche Nachricht-----
>> Von: Steve Barham [mailto:steve.barham@formicary.net]
>> Gesendet: Dienstag, 28. März 2006 17:07
>> An: activemq-users@geronimo.apache.org
>> Betreff: Re: AW: Final repost: SSL transport failure
>>
>>
>>
>> With SSL debug on, this is the output from the broker during the failure
>> (but no stack traces at debug log4j level):
>>
>> Socket Connection: /127.0.0.1:61616 -> /127.0.0.1:3163, READ: TLSv1
>> Application
>> Data, length = 16400
>> Socket Connection: /127.0.0.1:61616 -> /127.0.0.1:3163, READ: TLSv1
>> Application
>> Data, length = 16400
>> Socket Connection: /127.0.0.1:61616 -> /127.0.0.1:3163, READ: TLSv1
>> Application
>> Data, length = 16400
>> Socket Connection: /127.0.0.1:61616 -> /127.0.0.1:3163, READ: TLSv1
>> Application
>> Data, length = 16400
>> Socket Connection: /127.0.0.1:61616 -> /127.0.0.1:3163, READ: TLSv1
>> Application
>> Data, length = 16400
>> Socket Connection: /127.0.0.1:61616 -> /127.0.0.1:3163, READ: TLSv1
>> Application
>> Data, length = 16400
>> Socket Connection: /127.0.0.1:61616 -> /127.0.0.1:3163, READ: TLSv1
>> Application
>> Data, length = 16400
>> Socket Connection: /127.0.0.1:61616 -> /127.0.0.1:3163, READ: TLSv1
>> Application
>> Data, length = 16400
>> Socket Connection: /127.0.0.1:61616 -> /127.0.0.1:3163, READ: TLSv1
>> Application
>> Data, length = 16400
>> Socket Connection: /127.0.0.1:61616 -> /127.0.0.1:3163, READ: TLSv1
>> Application
>> Data, length = 16400
>> Socket Connection: /127.0.0.1:61616 -> /127.0.0.1:3163, READ: TLSv1
>> Application
>> Data, length = 16400
>> Socket Connection: /127.0.0.1:61616 -> /127.0.0.1:3163, READ: TLSv1
>> Application
>> Data, length = 16400
>> Socket Connection: /127.0.0.1:61616 -> /127.0.0.1:3163, READ: TLSv1
>> Application
>> Data, length = 16400
>> Socket Connection: /127.0.0.1:61616 -> /127.0.0.1:3163, READ: TLSv1
>> Application
>> Data, length = 16400
>> Socket Connection: /127.0.0.1:61616 -> /127.0.0.1:3163, READ: TLSv1
>> Application
>> Data, length = 16400
>> Socket Connection: /127.0.0.1:61616 -> /127.0.0.1:3163, READ: TLSv1
>> Application
>> Data, length = 16400
>> Socket Connection: /127.0.0.1:61616 -> /127.0.0.1:3163, READ: TLSv1
>> Application
>> Data, length = 16400
>> Socket Connection: /127.0.0.1:61616 -> /127.0.0.1:3163, READ: TLSv1
>> Application
>> Data, length = 16400
>> Socket Connection: /127.0.0.1:61616 -> /127.0.0.1:3163, READ: TLSv1
>> Application
>> Data, length = 16400
>> Socket Connection: /127.0.0.1:61616 -> /127.0.0.1:3163, READ: TLSv1
>> Application
>> Data, length = 16400
>> Socket Connection: /127.0.0.1:61616 -> /127.0.0.1:3163, READ: TLSv1
>> Application
>> Data, length = 16400
>> Socket Connection: /127.0.0.1:61616 -> /127.0.0.1:3163, READ: TLSv1
>> Application
>> Data, length = 16400
>> Socket Connection: /127.0.0.1:61616 -> /127.0.0.1:3163, READ: TLSv1
>> Application
>> Data, length = 16400
>> Socket Connection: /127.0.0.1:61616 -> /127.0.0.1:3163, READ: TLSv1
>> Application
>> Data, length = 16400
>> Socket Connection: /127.0.0.1:61616 -> /127.0.0.1:3163, READ: TLSv1
>> Application
>> Data, length = 16400
>> Socket Connection: /127.0.0.1:61616 -> /127.0.0.1:3163, READ: TLSv1
>> Application
>> Data, length = 16400
>> Socket Connection: /127.0.0.1:61616 -> /127.0.0.1:3163, READ: TLSv1
>> Application
>> Data, length = 16400
>> Socket Connection: /127.0.0.1:61616 -> /127.0.0.1:3163, READ: TLSv1
>> Application
>> Data, length = 16400
>> Socket Connection: /127.0.0.1:61616 -> /127.0.0.1:3163, READ: TLSv1
>> Application
>> Data, length = 16400
>> Socket Connection: /127.0.0.1:61616 -> /127.0.0.1:3163, READ: TLSv1
>> Application
>> Data, length = 16400
>> Socket Connection: /127.0.0.1:61616 -> /127.0.0.1:3163, READ: TLSv1
>> Application
>> Data, length = 16400
>> Socket Connection: /127.0.0.1:61616 -> /127.0.0.1:3163, READ: TLSv1
>> Application
>> Data, length = 16400
>> Socket Connection: /127.0.0.1:61616 -> /127.0.0.1:3163, READ: TLSv1
>> Application
>> Data, length = 16400
>> Socket Connection: /127.0.0.1:61616 -> /127.0.0.1:3163, READ: TLSv1
>> Application
>> Data, length = 16400
>> Socket Connection: /127.0.0.1:61616 -> /127.0.0.1:3163, READ: TLSv1
>> Application
>> Data, length = 4814
>> Socket Connection: /127.0.0.1:61616 -> /127.0.0.1:3163, WRITE: TLSv1
>> Application
>>  Data, length = 27
>> Socket Connection: /127.0.0.1:61616 -> /127.0.0.1:3175, handling
>> exception: java
>> .net.SocketTimeoutException: Read timed out
>> Socket Connection: /127.0.0.1:61616 -> /127.0.0.1:3175, handling
>> exception: java
>> .net.SocketTimeoutException: Read timed out
>> Socket Connection: /127.0.0.1:61616 -> /127.0.0.1:3175, handling
>> exception: java
>> .net.SocketTimeoutException: Read timed out
>> Socket Connection: /127.0.0.1:61616 -> /127.0.0.1:3175, handling
>> exception: java
>> .net.SocketTimeoutException: Read timed out
>> Socket Connection: /127.0.0.1:61616 -> /127.0.0.1:3175, handling
>> exception: java
>> .net.SocketTimeoutException: Read timed out
>> Socket Connection: /127.0.0.1:61616 -> /127.0.0.1:3175, handling
>> exception: java
>> .net.SocketTimeoutException: Read timed out
>> Socket Connection: /127.0.0.1:61616 -> /127.0.0.1:3175, handling
>> exception: java
>> .net.SocketTimeoutException: Read timed out
>> Socket Connection: /127.0.0.1:61616 -> /127.0.0.1:3175, handling
>> exception: java
>> .net.SocketTimeoutException: Read timed out
>> Socket Connection: /127.0.0.1:61616 -> /127.0.0.1:3175, handling
>> exception: java
>> .net.SocketTimeoutException: Read timed out
>> Socket Connection: /127.0.0.1:61616 -> /127.0.0.1:3175, handling
>> exception: java
>> .net.SocketTimeoutException: Read timed out
>> Socket Connection: /127.0.0.1:61616 -> /127.0.0.1:3163, called close()
>> Socket Connection: /127.0.0.1:61616 -> /127.0.0.1:3163, called
>> closeInternal(true)
>> Socket Connection: /127.0.0.1:61616 -> /127.0.0.1:3163, SEND TLSv1
>> ALERT:
>> warning, description = close_notify
>> Socket Connection: /127.0.0.1:61616 -> /127.0.0.1:3163, WRITE: TLSv1
>> Alert, length = 18
>> Socket Connection: /127.0.0.1:61616 -> /127.0.0.1:3163, called close()
>> Socket Connection: /127.0.0.1:61616 -> /127.0.0.1:3163, called
>> closeInternal(true)
>> Socket Connection: /127.0.0.1:61616 -> /127.0.0.1:3163, called close()
>> Socket Connection: /127.0.0.1:61616 -> /127.0.0.1:3163, called
>> closeInternal(true)
>>
>> Port 3163 in this instance is my 'server' app (one JMS client), and 3175
>> the 'client' app. I'm assuming that the socket timeout exceptions are
>> normal, and part of the Java SSL impl. as they occur at very regular
>> intervals.
>>
>> Corresponding 'server' stacktrace is:
>>
>> WARN [2006-03-28 16:00:37,484] activemq.ActiveMQConnection "Async
>> exception with no exception listener: java.io.EOFException: Peer
>> disconnected."
>> java.io.EOFException: Peer disconnected.
>> 	at
>> org.apache.activeio.command.AsyncChannelToAsyncCommandChannel$1.onPacket(AsyncChannelToAsyncCommandChannel.java:64)
>> 	at
>> org.apache.activemq.transport.activeio.PacketAggregatingAsyncChannel.onPacket(PacketAggregatingAsyncChannel.java:59)
>> 	at
>> org.apache.activeio.packet.async.FilterAsyncChannel.onPacket(FilterAsyncChannel.java:89)
>> 	at
>> org.apache.activemq.transport.activeio.ActivityMonitor.onPacket(ActivityMonitor.java:42)
>> 	at
>> org.apache.activeio.adapter.SyncToAsyncChannel.run(SyncToAsyncChannel.java:127)
>> 	at
>> edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
>> 	at
>> edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
>> 	at java.lang.Thread.run(Thread.java:595)
>>
>> Cheers,
>>
>> steve
>>
>>
>>
>>> BTW is there anything interesting appearing on the broker log whne
>>> your client dies?
>>>
>>> James
>>>
>>> On 3/28/06, Steve Barham <steve.barham@formicary.net> wrote:
>>>> Can you give an example of how to enable this debug logging?
>>>>
>>>> I've just tried with 4.0-SNAPSHOT rather than -M4, and get this
>>>> exception:
>>>>  WARN [2006-03-28 15:44:24,515] activemq.ActiveMQConnection "Async
>>>> exception with no exception listener: java.io.EOFException: Peer
>>>> disconnected."
>>>> java.io.EOFException: Peer disconnected.
>>>>         at
>>>> org.apache.activeio.command.AsyncChannelToAsyncCommandChannel$1.onPacket(AsyncChannelToAsyncCommandChannel.java:64)
>>>>         at
>>>> org.apache.activemq.transport.activeio.PacketAggregatingAsyncChannel.onPacket(PacketAggregatingAsyncChannel.java:59)
>>>>         at
>>>> org.apache.activeio.packet.async.FilterAsyncChannel.onPacket(FilterAsyncChannel.java:89)
>>>>         at
>>>> org.apache.activemq.transport.activeio.ActivityMonitor.onPacket(ActivityMonitor.java:42)
>>>>         at
>>>> org.apache.activeio.adapter.SyncToAsyncChannel.run(SyncToAsyncChannel.java:127)
>>>>         at
>>>> edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:650)
>>>>         at
>>>> edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:675)
>>>>         at java.lang.Thread.run(Thread.java:595)
>>>>
>>>> So very similar behaviour...
>>>>
>>>> steve
>>>> >
>>>> > I am not sure if I can help, but I think I had a similar error in my
>>>> > experiments with SSL.
>>>> > It looks like SSL doesn't work so fast or something, try to enable
>>>> the
>>>> SSL
>>>> > debug and see if you can get there a hint. My problem was at the
>>>> startup,
>>>> > that the logging took to long and I always got timeout errors.
>>>> >
>>>> >
>>>> > -----Ursprüngliche Nachricht-----
>>>> > Von: Steve Barham [mailto:steve.barham@formicary.net]
>>>> > Gesendet: Dienstag, 28. März 2006 16:16
>>>> > An: activemq-users@geronimo.apache.org
>>>> > Betreff: Final repost: SSL transport failure
>>>> >
>>>> >
>>>> >
>>>> > This is likely to be my last post unless someone has any bright
>>>> ideas
>>>> - I
>>>> > need SSL support for my application to be deployed, and I cannot
>>>> work
>>>> > around the following error.
>>>> >
>>>> > When I send a large (List of 8000 objects, each of about 10 Strings,
>>>> > around 10 characters each - serialized size is only 627k) message, I
>>>> get
>>>> > the following stacktrace:
>>>> >
>>>> > WARN [2006-03-28 15:13:46,140] activemq.ActiveMQConnection "Async
>>>> > exception with no exception listener: java.io.EOFException: Peer
>>>> > disconnected."
>>>> > java.io.EOFException: Peer disconnected.
>>>> >       at
>>>> > org.activeio.command.AsyncChannelToAsyncCommandChannel$1.onPacket(AsyncChannelToAsyncCommandChannel.java:65)
>>>> >       at
>>>> > org.apache.activemq.transport.activeio.PacketAggregatingAsyncChannel.onPacket(PacketAggregatingAsyncChannel.java:59)
>>>> >       at
>>>> org.activeio.FilterAsyncChannel.onPacket(FilterAsyncChannel.java:87)
>>>> >       at
>>>> > org.apache.activemq.transport.activeio.ActivityMonitor.onPacket(ActivityMonitor.java:42)
>>>> >       at
>>>> > org.activeio.adapter.SyncToAsyncChannel.run(SyncToAsyncChannel.java:150)
>>>> >       at
>>>> > edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:643)
>>>> >       at
>>>> > edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:668)
>>>> >       at java.lang.Thread.run(Thread.java:595)
>>>> >  WARN [2006-03-28 15:13:46,156] activemq.ActiveMQConnection "Cleanup
>>>> > failed"
>>>> > javax.jms.JMSException: Connection closed by remote host
>>>> >       at
>>>> > org.apache.activemq.util.JMSExceptionSupport.create(JMSExceptionSupport.java:57)
>>>> >       at
>>>> > org.apache.activemq.ActiveMQConnection.asyncSendPacket(ActiveMQConnection..java:1043)
>>>> >       at
>>>> > org.apache.activemq.ActiveMQConnection.cleanup(ActiveMQConnection.java:1191)
>>>> >       at
>>>> > org.apache.activemq.ActiveMQConnection.transportFailed(ActiveMQConnection..java:1585)
>>>> >       at
>>>> > org.apache.activemq.ActiveMQConnection.onException(ActiveMQConnection.java:1338)
>>>> >       at
>>>> > org.apache.activemq.transport.TransportFilter.onException(TransportFilter..java:102)
>>>> >       at
>>>> > org.apache.activemq.transport.TransportFilter.onException(TransportFilter..java:102)
>>>> >       at
>>>> > org.apache.activemq.transport.TransportFilter.onException(TransportFilter..java:102)
>>>> >       at
>>>> > org.apache.activemq.transport.InactivityMonitor.onException(InactivityMonitor.java:96)
>>>> >       at
>>>> > org.apache.activemq.transport.activeio.ActiveIOTransport$1.onError(ActiveIOTransport.java:71)
>>>> >       at
>>>> > org.activeio.command.AsyncChannelToAsyncCommandChannel$1.onPacket(AsyncChannelToAsyncCommandChannel.java:65)
>>>> >       at
>>>> > org.apache.activemq.transport.activeio.PacketAggregatingAsyncChannel.onPacket(PacketAggregatingAsyncChannel.java:59)
>>>> >       at
>>>> org.activeio.FilterAsyncChannel.onPacket(FilterAsyncChannel.java:87)
>>>> >       at
>>>> > org.apache.activemq.transport.activeio.ActivityMonitor.onPacket(ActivityMonitor.java:42)
>>>> >       at
>>>> > org.activeio.adapter.SyncToAsyncChannel.run(SyncToAsyncChannel.java:150)
>>>> >       at
>>>> > edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor$Worker.runTask(ThreadPoolExecutor.java:643)
>>>> >       at
>>>> > edu.emory.mathcs.backport.java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:668)
>>>> >       at java.lang.Thread.run(Thread.java:595)
>>>> > Caused by: java.net.SocketException: Connection closed by remote
>>>> host
>>>> >       at
>>>> > com.sun.net.ssl.internal.ssl.SSLSocketImpl.checkWrite(SSLSocketImpl.java:1168)
>>>> >       at
>>>> > com.sun.net.ssl.internal.ssl.AppOutputStream.write(AppOutputStream.java:43)
>>>> >       at
>>>> > org.activeio.net.SocketStreamChannel.write(SocketStreamChannel.java:183)
>>>> >       at
>>>> > org.activeio.adapter.OutputStreamChannelToOutputStream.write(OutputStreamChannelToOutputStream.java:48)
>>>> >       at
>>>> org.activeio.packet.ByteArrayPacket.writeTo(ByteArrayPacket.java:126)
>>>> >       at
>>>> org.activeio.net.SocketSyncChannel.write(SocketSyncChannel.java:113)
>>>> >       at
>>>> > org.activeio.adapter.SyncToAsyncChannel.write(SyncToAsyncChannel.java:184)
>>>> >       at
>>>> org.activeio.FilterAsyncChannel.write(FilterAsyncChannel.java:50)
>>>> >       at
>>>> > org.apache.activemq.transport.activeio.ActivityMonitor.write(ActivityMonitor.java:47)
>>>> >       at
>>>> org.activeio.FilterAsyncChannel.write(FilterAsyncChannel.java:50)
>>>> >       at
>>>> > org.activeio.command.AsyncChannelToAsyncCommandChannel.writeCommand(AsyncChannelToAsyncCommandChannel.java:41)
>>>> >       at
>>>> > org.apache.activemq.transport.activeio.ActiveIOTransport.oneway(ActiveIOTransport.java:95)
>>>> >       at
>>>> > org.apache.activemq.transport.TransportFilter.oneway(TransportFilter.java:90)
>>>> >       at
>>>> > org.apache.activemq.transport.TransportFilter.oneway(TransportFilter.java:90)
>>>> >       at
>>>> > org.apache.activemq.transport.WireFormatNegotiator.oneway(WireFormatNegotiator.java:65)
>>>> >       at
>>>> > org.apache.activemq.transport.MutexTransport.oneway(MutexTransport.java:44)
>>>> >       at
>>>> > org.apache.activemq.transport.ResponseCorrelator.oneway(ResponseCorrelatorjava:54)
>>>> >       at
>>>> > org.apache.activemq.ActiveMQConnection.asyncSendPacket(ActiveMQConnection..java:1041)
>>>> >       ... 16 more
>>>> >
>>>> >
>>>> > I will have to move to SwiftMQ if noone has any insight or
>>>> suggestions
>>>> for
>>>> > this problem, which isn't really something I want to do!
>>>> >
>>>> > Cheers,
>>>> >
>>>> > Steve
>>>> >
>>>> >
>>>> > --
>>>> > Steve Barham                       tel: +44 (0)7973 199 471
>>>> > Systems Engineer
>>>> > Formicary - delivering quality financial technology solutions
>>>> > http://www.formicary.net/
>>>> >
>>>> >
>>>> >
>>>> > This mail has originated outside your organization,
>>>> > either from an external partner or the Global Internet.
>>>> > Keep this in mind if you answer this message.
>>>> >
>>>> > This mail has originated outside your organization, either from an
>>>> > external partner or the Global Internet. Keep this in mind if you
>>>> answer
>>>> > this message.
>>>> >
>>>>
>>>>
>>>> --
>>>> Steve Barham                       tel: +44 (0)7973 199 471
>>>> Systems Engineer
>>>> Formicary - delivering quality financial technology solutions
>>>> http://www.formicary.net/
>>>>
>>>>
>>>
>>>
>>> --
>>>
>>> James
>>> -------
>>> http://radio.weblogs.com/0112098/
>>>
>>
>>
>> --
>> Steve Barham                       tel: +44 (0)7973 199 471
>> Systems Engineer
>> Formicary - delivering quality financial technology solutions
>> http://www.formicary.net/
>>
>>
>>
>> This mail has originated outside your organization,
>> either from an external partner or the Global Internet.
>> Keep this in mind if you answer this message.
>>
>> This mail has originated outside your organization, either from an
>> external partner or the Global Internet. Keep this in mind if you answer
>> this message.
>>
>
>
> --
> Steve Barham                       tel: +44 (0)7973 199 471
> Systems Engineer
> Formicary - delivering quality financial technology solutions
> http://www.formicary.net/
>
>
>
> This mail has originated outside your organization,
> either from an external partner or the Global Internet.
> Keep this in mind if you answer this message.
>
> This mail has originated outside your organization, either from an
> external partner or the Global Internet. Keep this in mind if you answer
> this message.
>


-- 
Steve Barham                       tel: +44 (0)7973 199 471
Systems Engineer
Formicary - delivering quality financial technology solutions
http://www.formicary.net/


Mime
View raw message