directory-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Niklas Therning <nik...@trillian.se>
Subject Re: [MINA] OutOfMemoryError on streaming large files
Date Mon, 12 Dec 2005 22:49:41 GMT
From what I understand your IoHandler on the sending side just writes an
InputStream using session.write(). Then you have this IoFilter which
reads the bytes from the stream into a buffer and forwards the buffer to
the next filter. It's a very handy filter to have, we should definitely
have something like that included in MINA.

One problem though with your approach is that it reads the entire input
in one go. I think the memory problem you're seeing could be related to
this. If you read faster than MINA can write, MINA's internal write
queue will be filled and the VM could eventually run out of memory. This
was the only thing I could think of right now. It would be great if you
could try to run the client and server in different VM:s so we could see
on which side the problem lies.

If my theory is correct the remedy would be to remove the while-loop and
instead add the InputStream you're reading from as a session attribute.
Then override messageSent() in your IoFilter and each time it gets
called you write the next 4K of data from the InputStream. This way you
write data in chunks of 4K and you only send the next 4K once the
previous 4K has been sent.

Once all the data from the InputStream has been written you forward the
messageSent() call to the IoHandler with the original InputStream as
message.

If you try this out and it works I think we could add this filter to
MINA if you are willing to share it.

/Niklas

Johannes Zillmann wrote:
> Thank you for response !
> 
> --- Niklas Therning <niklas@trillian.se> schrieb:
> 
>>Are you running both the client and server in the
>>same VM? If you are, 
>>please try to run them in separate VM:s and see if
>>both still run out of 
>>memory. 
> 
> 
> Yes both are in one vm. I can try with seperation but
> later.
> 
> 
>>I suspect that some buffer isn't released
>>properly but it seems 
>>unlikely that both ends fail at the same time unless
>>they are in the 
>>same VM.
>>
>>The maximum heap size is at 64 MB, right?
> 
> 
>  right !
> 
> 
>>Would it possible for you to post your code (the
>>relevant parts) to the 
>>list?
> 
> 
> I will try it with some semi code lines in it...
> the sending part(inside a filer):
> 
> public void filterWrite(NextFilter nextFilter,
> IoSession session, WriteRequest writeRequest) throws
> Exception {
>  if (message instanceof InputStream) {
> while (inputStream.available() > 0) {
>             ByteBuffer buffer =
> ByteBuffer.allocate(4*1024);
>             FileUtil.writeBytes(inputStream,
> buffer.asOutputStream(), 4*1024);
>             buffer.flip();
>             
>             nextFilter.filterWrite(session, new
> WriteRequest(buffer, writeRequest.getFuture()));
>             if(10MB are written)
>                 Thread.sleep(500);
>             //other wise the memory exception will
> raise earlier (but increasing sleep has no effect even
> if its 5000)
>         }
>         inputStream.close();
>  }
> }
> 
> the receiving part(inside a filer):
> 
> //get the IoSessionInputStream
> InputStream inputStream = session.get(KEY_IN);
>         File copy = new File("target/testFileCopy");
>         outputStream = new FileOutputStream(copy);
>         FileUtil.writeBytes(inputStream,
> outputStream);
> 
> whereby the FileUtil methods looks like:
> 
>  public static void writeBytes(InputStream iStream,
> OutputStream oStream, int length) throws IOException {
>         byte[] bytes = new byte[DEFAULT_BUFFER_SIZE];
>         int pieceLength = bytes.length;
>         while (length > 0) {
>             if (length < bytes.length)
>                 pieceLength = iStream.read(bytes, 0,
> length);
>             else
>                 pieceLength = iStream.read(bytes, 0,
> bytes.length);
> 
>             if (pieceLength == -1) {
>                 return;
>             }
>             
>             oStream.write(bytes, 0, pieceLength);
>             length -= pieceLength;
>         }
>     }
> 
> and:
> 
>  public static void writeBytes(InputStream iStream,
> OutputStream oStream) throws IOException {
>         int length = -1;
>         byte[] bytes = new byte[DEFAULT_BUFFER_SIZE];
>         while ((length = iStream.read(bytes)) != -1) {
>             oStream.write(bytes, 0, length);
>         }
>     }
> 
> hope this is readable and helps!
> best regards 
> johannes
> 
> 
> 
> 	
> 
> 	
> 		
> ___________________________________________________________ 
> Telefonate ohne weitere Kosten vom PC zum PC: http://messenger.yahoo.de
> 


-- 
Med vänlig hälsning

Niklas Therning
Software Architect

niklas@trillian.se
Mobil: +46 739 75 05 73

Trillian - Software Design at its best
www.trillian.se


Mime
View raw message