commons-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Bernd Eckenfels (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (IO-429) ByteArrayOutputStream can overflow
Date Wed, 12 Mar 2014 20:44:48 GMT

    [ https://issues.apache.org/jira/browse/IO-429?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13932340#comment-13932340
] 

Bernd Eckenfels commented on IO-429:
------------------------------------

The current code is nicely reusing ByteArrayOutputStream as well as largeCopy() from IOUtils.
There would be the option to implement it much more directly (using a gatherlist approach
and size limited buffers), which would have a number of advantages:

- reduced array copies (each byte at most once)
- controlled array resizing (for example 2 step linear growth)
- flatter stacks
- using some available() hints

But the code would be more complicated. Something like (pseudocode):

{code}
minsize=4k;
normsize=1M;
offset=0;
buf = new byte[minsize];
while(hasRoom(buf,offfset)
  c = in.read(buf,offset,bug.length-offset);
  read += c;
if (eof) return Arrays.copy(buf,read);
gatherlist = new ArrayList<byte[])(); // max 2047 1mb fragments
gatherlist.add(buf);
while(true) {
  buf= new byte[normsize]; 
  c = in.readFully(buf);
  read+=c;
  if read > MAX_INT
    throw();
  if (eof) return arrayFrom(gatherlist, buf);
  gatherlist.add(buf);
}
{code}

If you think it is worth to have this kind of optimized implementation I can provide a working
version.

But actually ByteBuffer would be much nicer as a return type.

> ByteArrayOutputStream can overflow
> ----------------------------------
>
>                 Key: IO-429
>                 URL: https://issues.apache.org/jira/browse/IO-429
>             Project: Commons IO
>          Issue Type: Bug
>          Components: Utilities
>            Reporter: Fabian Lange
>
> There are many places involved in the problem, and a good fix might be problematic performance
wise.
> For example:
> IOUtils.toByteArray(InputStream input) invoked with a Stream which feeds more than Integer.MAX_VALUE
bytes will either crash with NegativeArraySizeException or maybe worse overflow in such a
way that it returns fine (but only with partial data)
> The ByteArrayOutputStream will happily consume the full stream but "int count" will overflow.
At some point then toByteArray is invoked which will do like new byte[count].
> maybe "needNewBuffer" can throw the IllegalArgumentException, as it gets  the count and
could check for the overflow.



--
This message was sent by Atlassian JIRA
(v6.2#6252)

Mime
View raw message