db-derby-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Bryan Pendleton (JIRA)" <j...@apache.org>
Subject [jira] Closed: (DERBY-867) Performance issues with large object handling in DDMWriter
Date Tue, 18 Nov 2008 00:15:44 GMT

     [ https://issues.apache.org/jira/browse/DERBY-867?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel

Bryan Pendleton closed DERBY-867.

    Resolution: Won't Fix

I agree with the observation that we can close this.

> Performance issues with large object handling in DDMWriter
> ----------------------------------------------------------
>                 Key: DERBY-867
>                 URL: https://issues.apache.org/jira/browse/DERBY-867
>             Project: Derby
>          Issue Type: Improvement
>          Components: Network Server, Performance
>    Affects Versions:,
>            Reporter: Bryan Pendleton
>            Priority: Minor
> There are some performance issues related to the way that DDMWriter handles extremely
large network messages, such as those that arise in DERBY-125 and DERBY-492. The performance
problems involve both memory usage and CPU cycles.
> Among the problems that could be investigated and improved:
> 1) The DDMWriter internal byte array grows, but never shrinks (except when the DDMWriter
as a whole is collected). This means that there may be some memory wastage when the buffer
grows to a large value, then is later used for only small buffers.
> 2) When the buffer grows, it tends to double in size. When you start gettting to moderately
large buffer sizes, this may mean that the  buffer over-expands during a growth event, and
grows to a larger size than is needed, again leading to memory wastage.
> 3) I'm not sure what would happen if memory were unavailable when the buffer wanted to
grow; I think it would be a very hard error. There may be ways to handle this better, such
as: waiting a while to see if memory becomes available; writing previously chained messages
out of the buffer to free up space, etc. At the very least, it would be nice if an out-of-memory
problem were handled cleanly, for example by terminating this connection but leaving the rest
of the Network Server online and available for other users.
> 4) Perhaps we could eliminate the expansion all together by instead of expanding have
a pool of buffers available to all the DDMWriters.  If  a DDMWriter needs to write more, 
it picks up a new buffer and returns it to the pool when no longer needed.
> 5) Large DDM objects which require Layer B segmentation are written as a single object,
then later the segmentation is added after the payload data is complete. This processing is
handled by endDdm() and finalizeDssLength(). This code currently copies the bytes around in
the buffer two "extra" times: once to insert the extended length bytes at the front of the
large DDM object, and a second time to insert the DSS Continuation Headers every 32K bytes.
Given the API between DDMWriter and its callers, it may be impossible to totally remove this
memory copying, but it does seem like we could reduce from 2 full copies to 1, by combining
the insertion of the extended length bytes with the continuation header segmentation. It might
also be possible to detect a large object once it spills out from one DSS block into the second
block, and hence avoid copying for subsequent blocks, even if those first two blocks do have
to be copied once.
> There may be other opportunities in this part of the code; this issue is just a collection
of the ideas that came up while the developers were discussing bugs DERBY-125, DERBY-170,
DERBY-491, and DERBY-492. A review of the processing that occurs in those test cases should
be performed as part of working on this bug.

This message is automatically generated by JIRA.
You can reply to this email to add a comment to the issue online.

View raw message