mina-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Emmanuel Lecharny <elecha...@gmail.com>
Subject Re: Data loss when sending large strings
Date Tue, 31 May 2011 04:39:37 GMT
On 5/31/11 12:05 AM, Burt Alexander wrote:
> Hi, I'm using MINA successfully with one exception.  Sending large Strings between a
NioSocketAcceptor&  Connector (143 megs)

Do I have to understand that you are transmitting a 143 Mb String ???

If so, here is what is going to happen, if you use the 
PrefixedStringCodecFactory : a buffer will be used to cumulate the 
incoming bytes until everything has been received. This buffer will be 
associated with the session, and resides in memory. That means you will 
store a 143 Mb buffer in memory...

Add to that the conversion to a String when all the bytes have been 
received, that means that at a point, you will suck 3 times 143 Mb of 
memory (143 Mb for the buffer, and as a char is 2 bytes long, 2 times 
143 Mb for the String).

Your JVM will be under extreme load for a moment.
>   appears to have the effect of losing much of the message.  The acceptor then waits
for the 'rest' of the message, no exception is thrown.
> I'm using the PrefixedStringCodecFactory with the encoder and decoder max length set
to Integer.MAX_VALUE.  I'd like to be able to throttle the sending of the message but I'm
unable to determine how I can go about doing it - I've investigated using an ExecutorFilter
with an IoEventQueueThrottle, a WriteRequestFilter with the same, changing read&  write
buffer sizes all to no avail.  Using tcp to monitor the connection I see output like the following
when I lose data:
> tcp        0 453224 ::ffff:    ::ffff:    ESTABLISHED
> Any help will be most appreciated.  Thanks.
Here, if you can provide the snippet of code you are using, that could 
help. Please also provide the MINA version you are using.

All in all, I'd like to understand why you have to convert the big chunk 
of data to a String. You may need to implement some other logic to 
smoothly handled such a transfer. Anyway, I need more info to be able to 
propose alternatives here.

Emmanuel L├ęcharny

View raw message